Feeds

This week was full of major feature work and UI polishing, in addition to a lot of bug-fixing! I'm pretty sure everyone will find something to be excited about here:

Notable New Features

You can now swap the functions of drawing tablet pen buttons if you like the function performed by one of the buttons, but not which button activates it. (Joshua Goins, 6.3.0. Link)

Info Center now shows all of your GPUs, not just one of them. And they are now indicated in a fancy way! (Harald Sitter, 6.3.0. Link 1 and link 2)

Discover now shows you when apps are either packaged directly by their developer, or verified by a trusted third party. (Aleix Pol Gonzalez, 6.3.0. Link)

The Printers widget now shows each printer's print queue inline, right there in the widget! (Kai Uwe Broulik, 6.3.0. Link)

The Task Manager widget now lets you configure whether the "I'm playing audio" icons that appear in the corner of tasks can be clicked to mute the audio. In addition, the audio controls now always appear in the window preview thumbnails, and can't be accidentally disabled. (Petar Margetic, 6.3.0. Link)

When you've set up your system to reboot into the bootloader menu the next time it reboots (and not the firmware screen; that case was already handled), the logout screen how indicates this. (Nikolay Kochulin, 6.3.0. Link)

Notable UI Improvements

When you put a pie chart style System Monitor widget on a really skinny panel, the percentage circle in the center now looks nice, rather than being jagged and overly bold. (Arjen Hiemstra, 6.2.4. Link)

Made the progress indicators on Discover's Updates page more readable. (Nate Graham, 6.2.4. Link)

The Keyboard Indicator widget now indicates when modifier keys are "latched" or "locked", as they can be when using certain accessibility settings. (Nicolas Fella, 6.3.0. Link)

Removed the "Settings" launcher menu category! Now all of its contents have been moved into the "System" category. This reduces the number of categories that don't offer meaningful grouping. (Nate Graham, 6.3.0. Link)

The Printers widget now shows a little busy spinner for any printers that are currently printing, to make them easier to pick out among others when there are a lot of printers available. (Kai Uwe Broulik. 6.3.0. Link)

Widgets placed on the desktop are now very slightly translucent, just like the popups of widgets placed on the panel. (Marco Martin, 6.3.0. Link)

When for some reason the system time zone is set incorrectly or not set at all, the Digital Clock widget now tells you what's going on and offers you the chance to fix it yourself instead of just being broken. (Niccolò Venerandi, 6.3.0. Link)

Added a distinct Breeze icon for System Settings' Shortcuts page. (Joshua Goins, Frameworks 6.9. Link)

Improved the Breeze icon shown on password dialogs. (Kai Uwe Broulik, Frameworks 6.9. Link)

Notable Bug Fixes

Fixed a case where KWin would crash when plugging in external screens. (Xaver Hugl, 6.2.3. Link)

Fixed a regression caused by a version 0.22 of the power-profiles-daemon service that caused power profiles to not be registered properly by Plasma's Power and Battery widget until you manually restarted the Powerdevil service. (Méven Car, 6.2.3. Link)

Discover's auto-update feature once again works for Flatpak apps. (Harald Sitter, 6.2.4. Link)

The speaker test window on System Settings' Audio page now always fits its content, no matter what the name of the device or its audio profile may be. (Ismael Asensio, 6.2.4. Link)

Fixed a strange issue that would make GTK 4 apps look too dark when using HDR mode. Now they're a bit too light, but this is a GTK 4 bug. (Xaver Hugl, 6.2.4. Link)

Plasma panels in "fit content" mode are now compatible with flexible spacers; placing one on a panel no longer causes it to slowly grow to its full width when Plasma starts up — which was hilarious, but wrong. (Niccolò Venerandi, 6.2.4. Link)

Fixed a small visual glitch in KWin's Overview effect that caused windows dragged-and-dropped onto the desktop bar to strangely glide off screen. (Marco Martin, 6.2.4. Link)

Did a major code refactor and bug-fixing spree for desktop icon positioning, which fixes almost all of the outstanding bug reports about icons moving around, and also adds autotests to validate the fixes! You can read more about it here, too. (Akseli Lahtinen, 6.3.0. Link)

Alt key accelerators for items in the Global Menu widget now work on Wayland. (Nicolas Fella, 6.3.0. Link)

Performing a major system update using Discover no longer makes a pointless and invisible authentication prompt appear behind the logout window when you try to reboot to complete the update. (Alessandro Astone, PackageKit 1.3.1. Link)

Other bug information of note:

• 4 Very high priority Plasma bug (same as last week). Current list of bugs
• 37 15-minute Plasma bugs (down from 40 last week). Current list of bugs
• 106 KDE bugs of all kinds fixed over the last week. Full list of bugs

Notable in Performance & Technical

The feature to let you record the screen without re-approval if it's the same as one you did last time now also works when using rectangular region recordings. (David Redondo, 6.3.0. Link)

Implemented support for the Wayland system bell protocol. (David Redondo, 6.3.0. Link)

How You Can Help

KDE has become important in the world, and your time and contributions have helped us get there. As we grow, we need your support to keep KDE sustainable.

You can help KDE by becoming an active community member and getting involved somehow. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine!

You don’t have to be a programmer, either. Many other opportunities exist:

• Filter and confirm bug reports, maybe even identify their root cause
• Contribute designs for wallpapers, icons, and app interfaces
• Design and maintain websites
• Translate user interface text items into your own language
• Promote KDE in your local community
• …And a ton more things!

You can also help us by donating to our yearly fundraiser! Any monetary contribution — however small — will help us cover operational costs, salaries, travel expenses for contributors, and in general just keep KDE bringing Free Software to the world.

To get a new Plasma feature or a bugfix mentioned here, feel free to push a commit to the relevant merge request on invent.kde.org.

In this week's Beyond Blocks podcast episode, I spoke to Jess Archer - Engineering Team Lead at Laravel.

We discuss Laravel Prompts which was released at Laracon US last year and added to Drush 13, but also about PHP, Laravel, Neovim, working in the terminal and bullet journaling.

Listen to the episode now.

A few weeks ago Qt 6.8 was released, delivering many fixes and improvements for our software. Some of them were contributed by yours truly, and in this post I want to highlight some of them.

They relate to graphics tablet/stylus input on Wayland. Before we go into the fixes let’s have a quick overview of the flow of tablet input events on Wayland:

The genesis of input events is in the kernel driver for the particular tablet, which talks to the hardware (via USB, bluetooth etc). The kernel passes events to userspace via the evdev system. On the userspace side a Wayland compositor reads those events. Most compositors leverage libinput for this, a library that transforms raw evdev events into something more usable for the compositor while applying some cleaning and configuration. The compositor then delivers events to native Wayland applications using the tablet-v2 protocol. For legacy XWayland applications the compositor uses this protocol to send events to XWayland, which translates them into things X11 apps understand. The compositor can apply useful transformations to the input, like which area of the screen the tablet is mapped to, changing the pressure curve of the pen, or binding buttons to keyboard shortcuts.

The application then processes the events, potentially leveraging a UI toolkit like Qt. Qt creates QTabletEvent object for incoming Wayland events and delivers those to all UI elements. In case no UI element reacts to the event Qt will synthesize a mouse event from the tablet event and deliver that to the UI elements. That way most controls like buttons and menus don’t need special code to handle tablet input. Only when doing very low-level input handling or when needing tablet-specific interactions (like reacting to different pressure values) application developers need to explicitly handle tablet events in their code. Most UI toolkits will work somewhat similar to this. If an application doesn’t react to tablet input at all please file a bug against the application.

Qt Wayland had support for tablet input for a few years now, so what exactly needed fixing? The first thing is cursors. In Wayland the application tells the compositor which cursor to use. This can be done either by specifying a surface (i.e. an image) or a named cursor shape. Guess which one Qt used? That’s right: Neither. It just didn’t specify a cursor at all. The result depends on the compositor: KWin would display a cross-hair cursor as a fallback, which is ~okay, but not the cursor the application developer asked for. On other compositors I’ve tested there will be no cursor visible at all, which isn’t okay at all. For Qt 6.8 I implemented the missing cursor support, so now tablets get the same cursor as mouse input (unless of course the application developer wants a different cursor for tablet input).

Another thing that affected mostly non-Plasma users is client-side decorated windows. On Plasma Qt applications usually use the server-side decoration provided by KWin, but e.g. on GNOME Qt is responsible for drawing and handling window decorations. For this Qt features a plugin system so different decorations with different look-and-feels can be swapped out. Unfortunately those decorations didn’t handle tablet input at all, so it wasn’t possible to move around or close windows using a tablet stylus. I fixed this by pretending the tablet input is mouse input to the decorations, which was a simple yet effective solution for the issue. If there is ever a need for decorations to treat tablet input different to mouse input we can revisit this.

Talking about moving windows, a feature most KDE apps have (even if off-by-default these days) is dragging any empty area to move the window. This wasn’t working when using a stylus. Why? For that we need to look at how this works on a Wayland level. The xdg-shell protocol (the one responsible for most application windows) has a move request that asks the compositor to start a move interaction for the window. As part of the request the application must pass a serial, which is a number that corresponds to the last input event the application has received. To avoid applications suddenly deciding to move in the background compositors usually only allow move requests as a result of direct user input, so this serial must match the last input event. Qt wraps this move request inside the QWindow::startSystemMove function. The problem was that Qt didn’t keep track of the serial it received as part of tablet input, so when starting the move it would pass a wrong serial and the compositor (rightfully) refused the move. A few extra lines later the serial was tracked properly and moving windows with a stylus worked, just in time for Nate to disable the feature by default.

The same problem also affected drag-and-drop. When starting a drag with a stylus Qt now passes the correct serial, making drag-and-drop work (at least on the Qt side, there currently is a bug on the KWin side that prevents this from working).

The last fix for today relates to how applications react to the tablet events. Sometimes applications process clicks differently depending on what keyboard modifiers are pressed. For example pressing Ctrl while clicking on files in Dolphin allows to select multiple files. For this to work Qt conveniently delivers the active modifiers with every input event. Alas for tablet input the modifiers got lost along the way, so it wasn’t possible to select multiple files using a stylus. One small fix later it works as expected.

That’s all the Wayland tablet related fixes for today. If you find more issues in Qt/KDE apps related to tablet input on Wayland please report them on bugs.kde.org and I’ll look into them.

That’s not all there is about Wayland tablet improvements though. Fully in the spirit of the “We care about your input!” Goal there are exiting things happening on the KWin/Plasma side that I’ve been involved in. Stay tuned for more!

In my position of Software Platform Engineer at KDE I work on common building blocks for KDE software, like Qt and KDE Frameworks. This work is possible thanks to your generous donations. Check out our end-of-the-year fundraiser if you’d like to see more work like this.

Just in time for the winter holidays, the GNU Press shop is open!

Just in time for the winter holidays, the GNU Press shop is open!

Make sure you commit anything you want to end up in the KDE Gear 24.12
releases to them

Next Dates:

•   November 14, 2024: 24.12 freeze and beta (24.11.80) tagging and release
•   November 28, 2024: 24.12 RC (24.11.90) tagging and release
•   December  5, 2024: 24.12 tagging
•   December 12, 2024: 24.12 release

https://community.kde.org/Schedules/KDE_Gear_24.12_Schedule

I created a dashboard for testing multiple URLs at the same time using Axe.

Let’s go for my web review for the week 2024-45.

Voted in America? This Site Doxed You

Tags: tech, democracy

I didn’t know the voting rolls were public record in the US… this is a really bad idea.

https://www.404media.co/voted-in-america-this-site-doxed-you/

One weird trick to get the whole planet to send abuse complaints to your best friend(s)

Tags: tech, networking, security, protocols

Looks like there are people out there to get Tor relays down… and they found a smart networking trick I’d expect to not work anymore.

https://delroth.net/posts/spoofed-mass-scan-abuse/

XMPP: the forgotten gem of Instant Messaging

Tags: tech, xmpp, ux

Could XMPP make a come back if the user experience was better?

https://adele.pages.casa/md/blog/xmpp-the-forgotten-gem-of-instant-messaging.md

A conceptual model of ATProto and ActivityPub

Tags: tech, protocols, fediverse, governance, decentralized

Interesting explanation of the different visions and governance behind ActivityPub and ATProto.

https://fediversereport.com/a-conceptual-model-of-atproto-and-activitypub/

A Map of Sync

Tags: tech, syncing, data

Interesting dimensions to use when classifying syncing solutions and to see which ones will meet your constraints.

https://stack.convex.dev/a-map-of-sync

DuckDB over Pandas/Polars

Tags: tech, databases, data-science

A good reminder that I should probably evaluate DuckDB for some of my tooling.

https://pgrs.net/2024/11/01/duckdb-over-pandas-polars/

Why systemd is a problem for embedded Linux

Tags: tech, systemd, embedded

This is a good point. systemd is kind of turning into a monoculture, but what are we loosing? Indeed, right now things could be better on deeply embedded systems (I insist on the “deep embedded” here which is often more limited in resources). Unlike the author I think they could be improved and they don’t quite apply to a good chunk of recent embedded platforms though.

https://kevinboone.me/systemd_embedded.html

How the New sqlite3_rsync Utility Works

Tags: tech, databases, sqlite, tools, syncing

If you wonder how the protocol is designed and how the actual implementation works, this is a nice introduction. Clearly it’s helped by the size of that program which is fairly small.

https://nochlin.com/blog/how-the-new-sqlite3_rsync-utility-works

C++, Complexity, and Compiler Bugs

Tags: tech, c++, complexity

Good illustration of how the C++ language complexity is out of hands.

https://azeemba.com/posts/cpp-complexity-compiler-bugs.html

Having fun with modern C++ – Daniel Lemire’s blog

Tags: tech, c++

Still, C++ improved in recent years. The problem is that we tend to stick too much to the old constructs we already know and that things keep piling up but nothing is removed.

https://lemire.me/blog/2024/11/02/having-fun-with-modern-c/

smolweb HTML specification

Tags: tech, html, accessibility, low-tech, complexity

A nice subset of HTML to ensure better accessibility and reduced complexity.

https://smolweb.org/specs/index.html

HTML link, or button, that is the question

Tags: tech, web, frontend, html

Each has a use, they shouldn’t be conflated. It makes for poor user experience and accessibility otherwise.

https://marijkeluttekes.dev/blog/articles/2024/11/04/html-link-or-button-that-is-the-question/

Long Distance Relationships

Tags: tech, remote-working, management, product-management

A bit biased toward stable product teams only. Still, there are good tips which are more widely applicable here. This gives a good idea of the management of a distributed team of remote workers.

https://matt.blwt.io/post/long-distance-relationships/

Bye for now!

The topic of this posting already tells you that an old Unix guy tells stories about old techniques.

I'm a happy NIS (formerly YP) user since 30+ years. I started using it with SunOS 4.0, later using it with Solaris and with Linux since 1999.

In the past, a colleague wasn't happyly using NIS+ when he couldn't log in as root after a short time because of some well known bugs and wrong configs. NIS+ was also much slower than my NIS setup. I know organisations using NIS for more than 80.000 user accounts in 2024.

I know the security implications of NIS but I can live with them, because I manage all computers in the network that have access to the NIS maps. And NIS on Linux offers to use shadow maps, which are only accessible to the root account. My users are forced to use very long passwords.

Unfortunately NIS support for the PAM modules was removed in Debian in pam 1.4.0-13, which means Debian 12 (bookworm) is lacking NIS support in PAM, but otherwise it is still supported. This only affects changing the NIS password via passwd. You can still authenticate users and use other NIS maps.

But yppasswd is deprecated and you should not use it! If you use yppasswd it may generate a new password hash by using the old DES crypt algorithm, which is very weak and only uses the first 8 chars in your password. Do not use yppasswd any more! yppasswd only detects DES, MD5, SHA256 and SHA512 hashes, but for me and some colleagues it only creates weak DES hashes after a password change. yescrypt hashes which are the default in Debian 12 are not supported at all. The solution is to use the plain passwd program.

On the NIS master, you should setup your NIS configuration to use /etc/shadow and /etc/passwd even if your other NIS maps are in /var/yp/src or similar. Make sure to have these lines in your /var/yp/Makefile:

PASSWD = /etc/passwd SHADOW = /etc/shadow

Call make once, and it will generate the shadow and passwd map. You may want to set the variable MINUID which defines which entries are not put into the NIS maps.

On all NIS clients you still need the entries (for passwd, shadow, group,...) that point to the nis service. E.g.:

passwd: files nis systemd group: files nis systemd shadow: files nis

You can remove all occurences of "nis" in your /etc/pam.d/common-password file.

Then you can use the plain passwd program to change your password on the NIS master. But this does not call make in /var/yp for updating the NIS shadow map.

Let's use inotify(7) for that. First, create a small shell script /usr/local/sbin/shadow-change:

#! /bin/sh PATH=/usr/sbin:/usr/bin # only watch the /etc/shadow file if [ "$2" != "shadow" ]; then exit 0 fi cd /var/yp || exit 3 sleep 2 make

Then install the package incron.

# apt install incron # echo root >> /etc/incron.allow # incrontab -e

Add this line:

/etc IN_MOVED_TO /usr/local/sbin/shadow-change $@ $# $%

It's not possible to use IN_MODIFY or watch other events on /etc/shadow directly, because the passwd command creates a /etc/nshadow file, deletes /etc/shadow and then moves nshadow to shadow. inotify on a file does not work after the file was removed.

You can see the logs from incrond by using:

# journalctl _COMM=incrond e.g. Oct 01 12:21:56 kueppers incrond[6588]: starting service (version 0.5.12, built on Jan 27 2023 23:08:49) Oct 01 13:43:55 kueppers incrond[6589]: table for user root created, loading Oct 01 13:45:42 kueppers incrond[6589]: PATH (/etc) FILE (shadow) EVENT (IN_MOVED_TO) Oct 01 13:45:42 kueppers incrond[6589]: (root) CMD ( /usr/local/sbin/shadow-change /etc shadow IN_MOVED_TO)

I've disabled the execution of yppasswd using dpkg-divert

# dpkg-divert --local --rename --divert /usr/bin/yppasswd-disable /usr/bin/yppasswd chmod a-rwx /usr/bin/yppasswd-disable

Do not forget to limit the access to the shadow.byname map in ypserv.conf and general access to NIS in ypserv.securenets.

I've also discovered the package pamtester, which is a nice package for testing your pam configs.

Have you wanted the flexibility of f-strings but need safety checks in place? What if you could have deferred evaluation for logging or avoiding injection attacks? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

DrupalCamp Berlin 2024 has officially started! More than 200 Drupal enthusiasts are gathering at the atmospheric Alte Münze in the heart of Berlin to explore the latest trends and share insights. Our CEO, Grzegorz Bartman, is representing Droptica there. To mark the occasion, we've teamed up with our developers to curate a handpicked list of seven sessions from the event program that are well worth attending during this two-day gathering.

What is this pattern about?

The Pipeline design pattern (also known as Chain of Command pattern) is a flexible way to handle a sequence of actions, where each handler in the chain processes the input data and passes it to the next handler. This pattern is commonly used in scenarios involving data processing, web scraping, or middleware systems.

In this blog post, I’ll walk you through a specific example that leverages Python’s powerful functools.reduce and partial functions, along with the BeautifulSoup library for parsing HTML content. This code showcases the Pipeline pattern applied to HTML table extraction and processing.

What Does the Code Do?

The code defines a pipeline of data parsing steps for extracting and cleaning tables from an HTML file. It follows a functional programming approach to compose several processing functions into a unified flow using the Chain of Command pattern.

Key Concepts

1. Functional Composition: Combining multiple functions into one that executes in a specific order.
2. Data Parsing Pipeline: Sequential processing of HTML content into structured data (a DataFrame).
3. Error Handling: Ensuring the pipeline gracefully handles missing or malformed data.

Let’s break down the code step by step:

1. Function Composition with composefrom functools import reduce, partial from typing import Callable

The pipeline is created by composing multiple parsing functions into a single unified function. The compose function uses reduce to chain these functions together:

def compose(*functions: ParsingPipeline) -> ParsingPipeline: """Composes functions into a single function""" return reduce(lambda f, g: lambda x: g(f(x)), functions, lambda x: x)

This allows you to define an ordered flow of operations that process input data from one function to the next. Each function modifies the input data, which is then passed down the pipeline.

2. Reading HTML Content

The first step in the pipeline is to read the contents of an HTML file. This is done by read_htm_from:

def read_htm_from(filename: T, mode: T = "r", encoding: T = "utf-8") -> T: with open(filename, mode, encoding=encoding) as file: html_content = file.read() return html_content

This function opens an HTML file and returns its content as a string. It supports different file modes and encodings, making it flexible for various file formats.

Note that T is defined here as TypeVar("T"), see the typing docs.

3. Parsing the HTML Table

Next, read_table_from uses BeautifulSoup to find the HTML table within the file:

from bs4 import BeautifulSoup def read_table_from(htm_file: T, parser: str = "html.parser") -> T: soup = BeautifulSoup(htm_file, parser) table = soup.find("table") return table

This function converts the HTML content into a BeautifulSoup object and extracts the first table it finds. The parsed table is passed down the pipeline for further processing.

4. Extracting Rows and Data

Once the table is identified, the pipeline extracts the rows and applies filtering logic based on custom markers:

def extract_row_data_from( table_rows: T, start_markers: T, continue_markers: T, end_markers: T ) -> T: row_data: T = [] start_processing = False for row in table_rows: if any(marker in row.text for marker in start_markers) and not start_processing: start_processing = True continue if start_processing: if any(marker in row.text for marker in continue_markers): continue if any(marker in row.text for marker in end_markers): break row_data.append(row) return row_data[:-1]

This function inspects each row in the table, checking if the row text matches specified start, continue, or end markers. Data extraction begins after encountering the start marker and ends when the end marker is found.

5. Converting Rows to DataFrame

The next steps involve transforming the extracted row data into a structured pandas DataFrame. First, the rows are separated into individual columns using separate_columns_in:

def separate_columns_in(rows: T) -> T: data_rows: T = [] try: for row in rows: columns = row.find_all(["td", "th"]) data = [col.text for col in columns] data_rows.append(data) return data_rows except Exception as e: print(f"An error occurred: {str(e)}") return []

Then, convert_to_dataframe reshapes this data into a pandas DataFrame:

def convert_to_dataframe(data_rows: T) -> T: df = pd.DataFrame(data_rows) df = df.rename(columns=df.iloc[0]).drop(df.index[0]) df.columns = COLUMN_NAMES df.drop(columns=COLUMNS_TO_REMOVE, axis=1, inplace=True) df.set_index(df.columns[0], inplace=True, drop=True) return df

The DataFrame is cleaned up by renaming columns, removing unnecessary columns, and setting the correct index.

6. Assigning Correct Data Types

Finally, assign_correct_data_type_to ensures that the DataFrame columns have the appropriate data types:

def assign_correct_data_type_to( df: T, dict_types: dict[str, str] = COLUMN_TYPES, datetime_columns: list[str] = DATETIME_COLUMN_NAMES, ) -> T: if not isinstance(df, pd.DataFrame): raise ValueError("Input `df` must be a pandas DataFrame.") df = df.copy() for column in datetime_columns: if column in df.columns: df[column] = pd.to_datetime(df[column]) for column, col_type in dict_types.items(): if column in df.columns: try: if col_type == "numeric": df[column] = pd.to_numeric(df[column], errors="coerce") else: df[column].astype(col_type) except Exception as e: print(f"Error converting column {column} to {col_type}: {e}") return df

This function converts columns into numeric or datetime formats as needed, ensuring that the data is properly structured for analysis.

7. Putting It All Together

At the end of the code, the pipeline is composed by chaining all of the above functions together:

parse_gbx_bt: ParsingPipeline = compose( partial(read_htm_from, mode="r", encoding="utf-8"), read_table_from, read_rows_from, partial( extract_row_data_from, start_markers=["Closed Transactions:"], continue_markers=["Genbox", "balance", "Deposit"], end_markers=["Closed P/L:"], ), separate_columns_in, convert_to_dataframe, assign_correct_data_type_to, )

This creates a fully automated pipeline that:

1. Reads an HTML file.
2. Extracts table data.
3. Cleans and converts the data into a pandas DataFrame.
4. Assigns the correct data types.

Conclusion

This implementation of the Chain of Command or Pipeline pattern in Python demonstrates how to apply functional programming principles to data parsing tasks. The use of functools.reduce and partial, and BeautifulSoup provides a flexible, reusable way to process HTML content and structure it into usable data.

If you’re looking to create complex data processing pipelines that need to handle dynamic data from HTML or other sources, this approach is a clean and maintainable solution.

You can find the code in the repo: https://github.com/jjeg1979/pyBacktestAnalyzer.

And if you want to watch the code clinic where I presented the tool, feel free to check it out at https://pybites.circle.so/c/circle-coaching-calls/python-for-the-trader-code-clinic.

If you cannot access…well, what are you waiting for to become a PDM member?

Exposing External and Custom Data to Views with hook_views_data() admin Fri, 11/08/2024 - 11:12

Debian Contributions: 2024-10

Contributing to Debian is part of Freexian’s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

rebootstrap, by Helmut Grohne

After significant changes earlier this year, the state of architecture cross bootstrap is normalizing again. More and more architectures manage to complete rebootstrap testing successfully again. Here are two examples of what kind of issues the bootstrap testing identifies.

At some point, libpng1.6 would fail to cross build on musl architectures whereas it would succeed on other ones failing to locate zlib. Adding --debug-find to the cmake invocation eventually revealed that it would fail to search in /usr/lib/<triplet>, which is the default library path. This turned out to be a bug in cmake assuming that all linux systems use glibc. libpng1.6 also gained a baseline violation for powerpc and ppc64 by enabling the use of AltiVec there.

The newt package would fail to cross build for many 32-bit architectures whereas it would succeed for armel and armhf due to -Wincompatible-pointer-types. It turns out that this flag was turned into -Werror and it was compiling with a warning earlier. The actual problem is a difference in signedness between wchar_t and FriBidChar (aka uint32_t) and actually affects native building on i386.

Miscellaneous contributions

• Helmut sent 35 patches for cross build failures.
• Stefano Rivera uploaded the Python 3.13.0 final release.
• Stefano continued to rebuild Python packages with C extensions using Python 3.13, to catch compatibility issues before the 3.13-add transition starts.
• Stefano uploaded new versions of a handful of Python packages, including: dh-python, objgraph, python-mitogen, python-truststore, and python-virtualenv.
• Stefano packaged a new release of mkdocs-macros-plugin, which required packaging a new Python package for Debian, python-super-collections (now in NEW review).
• Stefano helped the mini-DebConf Online Brazil get video infrastructure up and running for the event. Unfortunately, Debian’s online-DebConf setup has bitrotted over the last couple of years, and it eventually required new temporary Jitsi and Jibri instances.
• Colin Watson fixed a number of autopkgtest failures to get ansible back into testing.
• Colin fixed an ssh client failure in certain cases when using GSS-API key exchange, and added an integration test to ensure this doesn’t regress in future.
• Colin worked on the Python 3.13 transition, fixing problems related to it in 15 packages. This included upstream work in a number of packages (postgresfixture, python-asyncssh, python-wadllib).
• Colin upgraded 41 Python packages to new upstream versions.
• Carles improved po-debconf-manager: now it can create merge requests to Salsa automatically (created 17, new batch coming this month), imported almost all the packages with debconf translation templates whose VCS is Salsa (currently 449 imported), added statistics per package and language, improved command line interface options. Performed user support fixing different issues. Also prepared an abstract for the talk at MiniDebConf Toulouse.
• Santiago Ruano Rincón continued the organization work for the DebConf 25 conference, to be held in Brest, France. Part of the work relates to the initial edits of the sponsoring brochure. Thanks to Benjamin Somers who finalized the French and English versions.
• Raphaël forwarded a couple of zim and hamster bugs to the upstream developers, and tried to diagnose a delayed startup of gdm on his laptop (cf #1085633).
• On behalf of the Debian Publicity Team, Anupa interviewed 7 women from the Debian community, old and new contributors. The interview was published in Bits from Debian.

The diffoscope maintainers are pleased to announce the release of diffoscope version 283. This version includes the following changes:

[ Martin Abente Lahaye ] * Fix crash when objdump is missing when checking .EFI files.

You find out more by visiting the project homepage.

I wrote in the past about how I'm a pessimist that strives for positive outcomes. One of the things that I gradually learned is is wishing others to succeed. That is something that took me a long time to learn. I did not see the value in positive towards other people's success, but there is. There is one thing to be sceptical to a project or initiative, but you can still encourage the other person and wish them well.

I think not wishing others well is a coping mechanism of sorts. For sure it was for me. As you become more successful in life, it becomes easier to be supportive, because you have established yourself in one way or another and you feel more secure about yourself.

That said, there is something I continue to struggle with, and that are morals. What if the thing the other person is doing seems morally wrong to me? I believe that much of this struggle stems from the fear of feeling complicit in another's choices. Supporting someone — even passively — can feel like tacit approval, and that can be unsettling. Perhaps encouragement doesn't need to imply agreement. Another angle to consider is that my discomfort may actually stem from my own insecurities and doubts. When someone's path contradicts my values, it can make me question my own choices. This reaction often makes it hard to wish them well, even when deep down I want to.

What if my tribe is just wrong on something? I grew up with the idea of “never again”. Anything that remotely looks like fascism really triggers me. There is a well known propaganda film from the US Army called “Don't Be a Sucker” which warns Americans about the dangers of prejudice, discrimination, and fascist rhetoric. I watched this a few times over the years and it still makes me wonder how people can fall for that kind of rhetoric.

But is it really all that hard? Isn't that happening today again? I have a very hard time supporting what Trump or Musk are standing for or people that align with them. Trump's rhetoric and plans are counter to everything I stand for and the remind me a lot of that film. It's even harder for me with Musk. His morals are completely off, he seems to a person I would not want to be friends with, yet he's successful and he's pushing humanity forward.

It's challenging to reconcile my strong opposition to their (and other's) rhetoric and policies with the need to maintain a nuanced view of them. Neither are “literal Hitler”. Equating them with the most extreme historical figures oversimplifies the situation and shuts down productive conversation.

Particularly watching comedy shows reducing Trump to a caricature feels wrong to me. Plenty of his supporters have genuine concerns. I find it very hard to engage with these complexities and it's deeply uncomfortable and quite frankly exhausting.

Life becomes simpler when you just pick a side, but it will strip away the deeper understanding and nuance I want to hold onto. I don’t want to fall into the trap of justifying or defending behaviors I fundamentally disagree with, nor do I want to completely shut out the perspectives of those who support him. This means accepting that people I engage with, might see things very differently, and that maintaining those relationships and wishing them well them requires a level of tolerance I'm not sure I possess yet.

The reason it's particularly hard to me that even if I accept that my tribe maybe wrong in parts, I can see the effects that Trump and others already had on individuals. Think of the Muslim travel ban which kept families apart for years, his border family separation policy, the attempted repeal of Section 230. Some of it was not him, but people he aligned with. Things like the overturning of Roe v. Wade and the effects it had on women, the book bans in Florida, etc. Yes, not quite Hitler, but still deeply problematic for personal freedoms. So I can't ignore the harm that some of these policies have caused in the past and even if I take the most favorable view of him, I have that track record to hold against him.

In the end where does that leave me? Listening, understanding, and standing firm in my values. But not kissing the ring. And probably coping by writing more.

I’ve seen this meme a bunch of times recently. I always reply; what is asserted without evidence may be dismissed without consideration.

Current Gen AI is flawed by hallucination issues, mired in copyright controversy, expensive to run and lacking clear use cases. (Although it’s pretty good at code generation). It’s a massive hype train.

Gen AI, as it is now, was made possible by the invention of “Transformer Architecture” by Google in 2017. We’re seeing fast paced change and development, but all built on that technology.

At some point another quantum breakthrough will change things all over again - and make another step towards AGI. Although it will take several such steps, and order of magnitudes larger models (and multi models), to create anything resembling true AI.

So a huge number of disparate individuals, institutions, governments and companies are pursuing the development of AI. There’s no single cohesive agenda behind it. As new technologies arise we adapt to them, find uses for them, and everyone pursues their agendas with them.

Not particularly special to AI I don’t think.

Python metaclasses, considered advanced programming and Python “black magick” (*) explained in eight words:

The type of a class is a class.

Here’s what knowledge of Object Oriented theory and type systems permit you to deduce from this:

Using the word “class”, instead of “the type of a class is type” or even “the type of a class is a type, classes are types”, implies that a user defined class can be a metaclass. This is indeed the case, and the point of metaclasses in Python.

The type is responsible for creating new instances. So if the type of a class is a class then we can write classes that create classes. Indeed this is the primary usecase for metaclasses.

(Deeper knowledge of Python, and the two phase object creation protocol, may lead you to deduce that this is done by overriding the __new__ method. If you’re familiar with “type” as a class factory you can probably even guess the signature and that you must inherit from type.)

If the type of a class is a class then the type system will permit a type check for the class against its class. And indeed isinstance(klass, metaclass) returns true.

(And deeper knowledge of Python will tell you that the magic methods, the protocol methods, are always looked up on the type. So we can implement behaviour for class objects by providing magic methods on the metaclass.)

All of this implies that classes are themselves objects. Which is true in Python for everything is an object in Python (and everything is a reference).

And so on…

• Type and class are synonyms in Python.
• type(type) is type

And to further round out the type system, these are also Python invariants:

• isinstance(object, object) is True # object is an object
• isinstance(object, type) is True # but also a type
• isinstance(type, object) is True # type is an object
• isinstance(type, type) is True # but also a type

(*) Like all black magick it is useful for understanding the world but never for actual use. Well, except perhaps in very rare circumstances if you know what you’re doing.

📘 Written in 2021.

IronPython in Action was published on the 7th April 2009 and we sold a little over 7000 copies.

Royalties for last quarter amounted to $25.

It took me two years to write thirteen chapters and a couple of appendices, and took Christian Muirhead about the same to write two chapters and an appendix. Jonathan Hartley did the diagrams and illustrations and the worst part was compiling the index.

It took so long because IronPython was still in alpha (!) when we started and it changed several times (including a Silverlight version being released) whilst writing!

After leaving Resolver Systems in 2010 I spent a year contracting on Line of Business apps that ran in Silverlight (Django on the server): Python code running in the browser on the client side. It was glorious.

We even had functional tests on unittest built in to the app.

Work on mock accelerated massively once IronPython in Action was complete. MagickMock was born not long afterwards.

I was also helping maintain the python.org website and adding test discovery to unittest at the time, and speaking at every conference I could find.

It felt like the glory days of the Python community. It’s almost time for PyCon (online) and I’m nostalgic once again.

My first PyCon, the second Dallas PyCon and my first time in the US, there were about 600 attendees. You could almost know everyone.

I shaved my beard to enter Dallas and wore my hair in a pony tail. All I knew was they didn’t like hippies there. It was the nicest greeting at a US airport I’ve ever had.

I went on a road trip with Andrzej Krzywda afterwards trying to find mountains. We found the Ouchita mountains in Oaklahoma and drove back through Arkansas to visit friends of mine in Houston. Along the peaks of the mountains, which are hills really, we found a view called Dead Man’s Vista and we I laughed together at Microsoft.

Not long after this the web explosion happened and Django happened, google adopted Python as an official language and the community started to explode and grow.

That was even before Python became huge as a teaching language and before Python exploded in data science too.

I once paired with Jacob Kaplan Moss at a PyCon sprint and fixed some issue by adding a metaclass to the Django codebase. Which he never committed and found a better way.

That’s the closest I’ve come to deploying a metaclass I think, although I’ve removed a few in my time.

I knew Python had “made it” as a language when one bag stuffing pre-PyCon I met someone who didn’t want to be there. He’d been sent by work. Before that Python was obscure, and only people who really loved it went to PyCon. Which I’m convinced is the secret of Python’s success.

It was built by passion not by money. For the sheer love and the joy of building something beautiful with other people.

I was a Mac user then and had a running joke with Jonathan Hartley about Linux and projectors.

One time he plugged his laptop into the projector prior to his PyCon talk (Testing is a Silver Bullet), tried to fix the x-config from the terminal and rendered his laptop unusable. He did the presentation on mine. The next year Mark Shuttleworth did a keynote talk at PyCon and running some bleeding edge version of Ubuntu also couldn’t plug it into the projector system. Hilarity on my part.

The biggest conference I ever spoke at was a Microsoft one in Brighton where they demoed Silverlight and I demoed IronPython on Silverlight. They didn’t tell me I would be on main stage in front of a few thousand Microsoft devs. I was used to talking to a few hundred at a time!

I had a slide deck built from S5 with reStructured Text markup and a Far Side slide mocking static typing. Which went down a bomb to an audience of C# devs. I still managed, by coincidence, to demo almost the same features of Silverlight as Microsoft bigwig Scott Hanselman who did the keynote.

It was an “interesting experience”, evangelising Python and dynamic languages in “the heart of the beast” as it were. Microsoft went on to step up their involvement with Python and sincere Open Source commitments which they’ve maintained since.

Since I first wrote this Python has finally made it, ranked as the most widely used programming language in the world by TIOBE and PyPL. World number one.

I joined Twitter fourteen years ago and have tweeted over fifty-two thousand times. I follow 1,636 accounts, which is too many, and have 8,670 followers. I use Tweetdeck which is run by Twitter and doesn’t show ads or promoted tweets or mess with tweet order and it lets me use two different accounts.

I use twitter a lot less than I did during my social media and community frenzy whilst I delighted to learn Python, but I still enjoy it.

During that time (2006-2011) I “drank from the firehose”. I read all of slashdot (scanned every headline and read relevant articles), read all of comp.lang.python (every message title - read and replied to many), read all of python-dev (similarly) and all of testing-in-python, blogged almost daily and worked full time as a software engineer commuting to London four times a week and developed mock in my spare time and worked on unittest in the Python standard library. And wrote a book and worked part time doing community liaison and service development for a local charity working with the homeless and disadvantaged. I was Microsoft MVP for three years for my work with IronPython, I spoke at countless conferences and received the Python Software Foundation Community Award for my work running Planet Python and helping out with the Python.org website and mailing infrastructure.

Then in 2011 my first child was born and I started working for Canonical. Three years of large Django web applications then three years of Go and MongoDB and then a year with Red Hat testing Ansible Tower and now four years self employed.

During that time I remembered that the primary drive in my life was spiritual and I started meditating again. One hour a day of mindfulness of breathing. That transformed my life all over again.

I once rode in the back of a beaten up station wagon owned and operated by the creator of the Python programming language whilst sat alongside the creator of Bitorrent, which was written in Python.

I also once had a pub lunch in Oxford with the creator of the Erlang programming language and the creator of the Haskell programming language. We were all three speaking at the ACCU conference. I was speaking on IronPython.

It’s been a fun journey.

I spent past three weeks working on refactoring and fixing legacy code (the oldest of which was from 2013) that handled positioning Plasma desktop icons, and how this data was saved and loaded.

Here's the merge request if you're curious: plasma-desktop: Refactor icon positioner saving and loading

The existing code worked sometimes, but there were some oddities like race conditions (icon positioning happens in weird order) and backend code mixed in with frontend code.

Now I am not blaming anyone for this. Code has tendency to get a bit weird, especially over long periods of time, and especially in open source projects where anyone can tinker with it.

You know how wired earbuds always, always get tangled when you place them in a drawer or your pocket or something for few seconds? Codebases do the exact same thing, when there are multiple people writing on things, fixing each others' bugs. Everyone has a different way of thinking, so it's only natural that things over time get a bit tangled up.

So sometimes you need someone to look at the tangled codebase and try to clear it up a bit.

Reading code is the hardest part

When going through old code, especially some that has barely any comments, it can take a very long time to understand what is actually going on. I honestly spent most of my time trying to understand how the thing even works, what is called when, where the icons positions are updated, and so on.

When I finally had some understanding of what was happening, I could start cleaning things up. I renamed a lot of the old methods to be hopefully more descriptive, and moved backend code — like saving icon positions — from the frontend back to backend.

Screens and icons

Every screen (PC monitor, TV…) tends to have it's own quirks. Some, when connected with display-port adapter, tell your PC it's disconnected if your PC goes to screen saving mode. Some stay connected, but show a blank screen.

One big issue with the icon positions was that when screen got turned off, it thought there was no screen anymore and started removing items from the desktop.

That's fair. Why show desktop icons on a screen that is non-existent? But when you have a monitor that tells your PC, "Okay I'm disconnecting now!" when the PC says it's time to sleep, wrong things would happen.

This condition is now handled by having a check that if the screen is in use or not. Now when screen is not in use, we just do nothing with the icons. No need to touch them at all.

Stripes and screen resolution

Our icon positioning algorithm uses something called "stripes."

Every resolution has it's amount of stripes. Stripes contain an array of icons, or blank spots.

So if your screen resolution is, let's say, 1920x1080, we calculate how many stripes and how many items per stripe will fit on that screen.

Stripe1: 1 2 3 4 5 6 7 Stripe2: 1 2 3 4 5 6 7 Stripe3: 1 2 3 4 5 6 7 And so on..

But when you change your screen resolution or scale factor, how many icon stripes you have and how many icons fit on each stripe will change.

So if you have one of those screens that looks to the system like it's been unplugged when it goes into sleep mode, previously the stripe amount would change to 1 row, 1 column. And the icon positioner would panics and shove all icons in that 1,1 slot.

Then when you'd turn the screen back on, the icon positioner would wonder what just happened and restore the proper stripe number and size. But by that point it would have lost all our positioning coordinate data during the shoving of icons in that one miniscule place, so instead it would reset the icon positions… and this leaves users wondering why their desktop icon arrangement is now gone.

Here we have to also check for the screen being in use or not. But there were other problems.

Saving icon positions

The prior code saved the icon positions every time the positions changed. Makes sense.

But it didn't account for the screen being off… so the icon positions would get saved while the desktop was in a faulty state. This also causes frustration because someone arranges the icons how they wish, but then screen does something weird and they're now saved in wrong places again.

Our icon positions were updated after almost every draw call, if the positions changed. So this would mean the saving would happen rather often and no matter what moved them.

We had to separate the user action from computer action. If computer moves the icons, we ideally do not save their positions, unless something drastic has happened like resolution change.

The icon positions are saved per resolution, so if you move icons around while they're displayed on a 3440x1440 screen and then change the resolution to 1920x1080, both will have their own arrangements. This part of the codebase did not previously work, and it would always override the old configuration, which caused headache.

So now we only save icon positions when:

• The user adds or removes a desktop icon
• The user moves a desktop icon
• The user changes the desktop resolution

This makes the icon position saving much less random, since it's done only after explicit user actions.

Margin errors

The last thing that caused headaches with the icon positioning was that the area available on the desktop for icons was determined before panels were loaded. When the panels loaded, they would reduce the amount of space for desktop icons, and that area would constantly resize until everything is ready.

In previous code, this would cause icons to move, which updates positions, which then saves their positions.

So let's say you arrange your icons nicely, but the next time you boot into plasma, your panels start shoving the poor icon area around and the icons have to move out of the way… and now they're all in the wrong places.

This was already partially fixed by not saving when the computer moves the icons around: we just load the icon positions when the screen is in use and we are done with listing the icons on the desktop. Part of the margin changes happen when screen is off.

We still need to fix the loading part; ideally we load the icon area last, so that it gets the margins it expects and doesn't shuffle around while panels are still appearing. But it was out of scope for this merge request.

Conclusions

It may not sound like much, but this was a lot of work. I spent days just thinking about this problem, trying to understand what is happening now and how to improve it.

Luckily with a lot of help from reviewers and testers I got things to work much better than it used to. I am quite "all-over-the-place" when I solve problems so I appreciate the patience they had with me and my questions. :D

What I mostly wished for when working on this were more inline code comments. You don't need to comment the obvious things, but everything else could use something. It's hard to gauge what is obvious and what is not, but that kind of answers the question: If you don't know if it's obvious or not, it's likely not, so add some comment about it.

I do hope that the desktop icons act more reliably after all these changes. If you spot bugs, do report them at https://bugs.kde.org.

Thanks for reading! :)

PS. The funniest thing to me about all of this is that I do not like having any icons on my desktop. :'D