Feeds
Zero to Mastery: Python Monthly Newsletter đ»đ
Seth Michael Larson: How do I pay the publisher of a web page?
Published 2024-11-24 by Seth Larson
Reading time: minutes
Here's an unanswered question:
I have money and I have a URL, how do I send money to the publisher of that URL?
URLs tell you where to get content on the web, but they don't tell you anything about how to support the person who created the content. This story might sound similar to paying open source maintainers, where a user can almost abstract an entire project to a single download URL.
There are tons of people creating content for the web and plenty of ways to get paid (Patreon, Kofi, GitHub Sponsors, YouTube Paid Membership), but there's no standardized way to direct someone interested in paying for the content of a page in the right direction.
We have HTML meta headers for many things, including where to find an RSS feed or what my Fediverse handle is, but none for enumerating options to pay the creator of the content. I wish I could click a button to easily send a "tip" to someone who created something I enjoy or to browse other options for supporting them.
Existing technology Payment Request APIThere are things like the web "Payment Request API" which gives you a JavaScript API for generating a payment, but this doesn't fit my criteria.
For one: this means that every person creating content for the web needs to add JavaScript to their page. This is a much higher bar than simply linking to existing payment methods that a creator already likely uses to get paid. Being difficult means it's unlikely for large numbers of people to do the work.
I also don't see being able to automate this because of the JavaScript. Web creators likely have existing payment pages that they'd much rather link out to instead of trying to handle payments themselves individually.
Lastly, this API exists and I don't see it being used by creators today. That should say something about either its ease-of-use or return on investment from potential supporters.
Linking to payment methods in the pageYeah, we could scrape the payment URLs we know about embedded in the page. But there's a difference between potential URLs in the page due to non-creator generated content (links in comments, etc) and whatever the "authoritative" URLs are for paying the creator of the page. Being able to set <meta> tags in <head> is typically a higher bar than setting arbitrary URLs in the <body>.
Podcasting 2.0 RSS <podcast:funding> tagPodcasting 2.0 supports basically the exact tag that I want to use which encodes a URL and a human-readable name for that URL into the metadata description of a podcast publication. Really great to see some prior art here.
Thanks to DamonHD for sending me this reference.
FlattrFlattr is a service that tried to turn a "subscription" from users into micro-payouts based on a users' browsing history. Flattr shut down in 2023. This approach isn't one I'm interested in replicating for a few reasons:
- Access to the entire browsing history feels like a privacy nightmare. Yes you receive a "complete" sample of which web pages a user has visited but, yeah this doesn't seem great?
- Flattr tried to create its own payment platform for creators, rather than pushing users to send monetary support through existing stable payment methods like Patreon. They had to do this because of the micro-payments thing.
In general this is making me think micro-payments is extremely hard to do. I think having a handful of dedicated fans for small creators might be enough to "offset" the "loss" of micro-payments? Perhaps there can be a recommendation to note to users when certain creators are "niche" and therefore are receiving fewer payments relative to other creators and thus would benefit more from a contribution / boost?
Thanks to Quentin for sending me this reference.
BraveI know about Brave, and I would like to avoid crypto in my solution. Also many of the creators I pay for don't use crypto but do have multiple payment methods. I don't think the solution should require creators AND users adopt new technology to work.
What happens now?I'm no stranger to standards, so maybe I do some research and write a web standard proposal? Seems like fun! I'm imagining something like:
<head> <!-- ... --> <meta property="financial-support" content="https://patreon.com/c/MatthewCarlson"> </head>Because this is primarily for money, no doubt it will be abused to hell. First-party browsers probably wouldn't do anything with this information for the fear of legitimizing scammers' fake profiles.
The existence of the "Web Payments API" makes me think maybe it's not a huge deal and that whenever money gets involved peoples' spidey-senses start going off about whether a page is legitimate? Not sure.
Let me know what you think!
Have thoughts or questions? Let's chat over email or social:
sethmichaellarson@gmail.com
@sethmlarson@fosstodon.org
Want more articles like this one? Get notified of new posts by subscribing to the RSS feed or the email newsletter. I won't share your email or send spam, only whatever this is!
Want more content now? This blog's archive has ready-to-read articles. I also curate a list of cool URLs I find on the internet.
Find a typo? This blog is open source, pull requests are appreciated.
Thanks for reading! ⥠This work is licensed under CC BY-SA 4.0
︎Welcome to My Blog
Heyho together!
I am from now on writing my posts on GitHub pages. Apart from it being useful to keep my posts versioned using git, I had some issues with my previous blog. The idea was to simply use write.as and publish a post from time to time. This worked well except for more than a month ago me wanting to do a post about my KRunner plugins. It naturally contained a lot of links and thus the publishing was prevented and even the account blocked due to apparent spam. There was no response via mail for over a month.
So here we are not on another blog where I hopefully write more often and also be able to spent more time on KDE!
gnuboot @ Savannah: GNU Boot November 2024 News
A lot has changed since the two last news from the GNU Boot project.
People involved in the GNU Boot project will be organizing a 100% free
software install party within a bigger event that also has a regular
install party. There will also be a presentation about 100% free
software in there. The event will be mainly in French.
More details are available in French and in English in the following
link:
https://lists.gnu.org/archive/html/help-guix/2024-11/msg00112.html
Many changes were made since the RC3 and since then we fixed an
important bug that prevented Trisquel from booting (If during the
Trisquel installation you chose "LVM2" and didn't encrypt the
storage, GNU Boot images with GRUB would not find the Trisquel
installation).
Because of that we decided to do a new RC4 (release candidate 4)
and to publish new GNU Boot images.
There are still some work needed before doing a 0.1 release as we want
to make it easier for less technical users to install and use GNU
Boot, but more and more of the project structure are getting in place
(website, manual, automatic tests, guix, good development procedures,
enabling build on all distributions, etc) which then makes it easier
to contribute.
We also decided to use Guix for more of the software components
we build, and since this is a big change, we will need people to
help more with testing.
The last announcement we made was "Nonfree software found in GNU Boot
releases again, many distros affected"[1].
Some people misunderstood it (maybe we could have been more clear):
the nonfree software that we found was code that GNU Boot didn't use,
so it was easy to remove and it didn't affect the supported devices in
any way.
Finding nonfree software in 100% free distribution is also common:
this is part of the work to ensure these distribution remains 100%
free.
The first time it happened in GNU Boot we publicized it to explain why
we were re-releasing some of the GNU Boot files as it could be very
scary if this happens without any public communication.
The second time we published a news about it mainly to help propagate
the information to the affected distributions and this is probably why
it was misunderstood: it was mainly targeted at GNU Boot users and
maintainers of the affected packages. We also contacted upstream and
some affected distributions directly as well but contacting everybody
takes a lot of time so having a news about it helps. At least Debian
and Trisquel fixed the issue but we still need to contact some
distributions.
After that, and probably thanks to the previous news, Leah Rowe
contacted us on one of the GNU Boot mailing lists[2] to notify us that
she also found additional similar nonfree software in GNU Boot.
So we confirmed that and promptly removed them and re-made again the
source release. And here again even if the work was delayed a bit,
this was fast to do and it doesn't affect the supported devices in any
way.
But we also need help contacting distributions again because one of
the issue she found is very serious because it affects many
distributions and also important devices that GNU Boot doesn't
support.
The ARM trusted firmware ships a nonfree hdcp.bin binary in its source
code. ARM trusted firmware is a dependency of u-boot that is used to
support many ARM computers in other distributions (like Guix, Debian,
etc).
As contacting affected distributions is a tedious task, we also need
help to propagate the information and contact them especially because
we don't know if Leah intend or not to do that work (so far she didn't
reply when asked twice about it), so it's probably up to the GNU Boot
community as a whole (which also includes its maintainers and readers
of this news) to help here.
The details are in the commit 343515aee7ef34695ac45830fad419d9562f9c15
("coreboot: blobs.list: arm-trusted-firmware: Remove RK3399 hdcp.bin
firmware.") in the GNU Boot source code[3].
[1]https://savannah.gnu.org/news/?id=10684
[2]https://lists.gnu.org/archive/html/gnuboot-patches/2024-10/msg00028.html
[3]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=343515aee7ef34695ac45830fad419d9562f9c15
JordĂĄn (isf) has been contributing some Spanish translations of the
most important website pages (the landing, status and how to
contribute pages). This is important as it could help get more
contributors. These contributions also helped us improve the process
for accepting pseudonymous contributions and enabled us to fix issues.
The work on improving the website in general also continued. Many of
the website pages were reviewed and improved (there is a lot of work
there and mentioning it all would make the news way too long).
The website also now shows the git revision from which it is build and
we also helped the FSF fix some server configuration that created
issue with the deployment of the GNU Boot website (more details are in
the commit message[1]) by reporting the issue to them and testing the
fix.
Patches for making a manual are also being reviewed. While there isn't
much in the manual yet, it also enables to better organize the
documentation and it has the potential to make GNU Boot more
accessible to less technical people.
The next goals is to look how to merge part of the website inside the
manual and continue improving both the website and the manual.
[1]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=d1df672383f6eb8d4218fdef7fbe9ec5e41803e4
We now have the ability to verify the source code when downloading it
from git. This is important to avoid certain type of attacks and it
also enables to write code to automatically download, verify and build
the GNU Boot source code.
The source can be verified with the following command (it requires to
have Guix installed):
 $ guix git authenticate $(git rev-parse HEAD) \
 "E23C 26A5 DEEE C5FA 9CDD D57A 57BC 26A3 6871 16F6" \
 -k origin/keyring
If the authentication works it will print a message like that:
   guix git: successfully
   authenticated commit 05c09293d9660ea1f26b5b705a089b466a0aa718
The 05c09293d9660ea1f26b5b705a089b466a0aa718 might be different in
your case.
The "E23C 26A5 DEEE C5FA 9CDD D57A 57BC 26A3 6871 16F6" part in the
command above is Adrien Bourmault (neox)'s GPG key.
How to use that will be documented more in depth in the upcoming GNU
Boot manual that is currently being reviewed. Its importance will also
be explained in more details for people not familiar with the security
issues it's meant to solve. Also note that we also welcome help for
reviewing patches.
The GNU Boot source code has a complex history. It is based on the
last fully free software releases of Libreboot. And the Libreboot
source code history is very complex.
We found some missing authorship information in some of the files that
come from Libreboot and so we started such information from the
various git repositories that were used at some point by Libreboot or
some of the projects it was based on.
To help with this task we also added a page on the GNU Boot website
(https://www.gnu.org/software/gnuboot/web/docs/history/) to track the
status of the reconstruction of the missing authorship and to document
the GNU Boot source code history.
GNU Boot is just a distribution and like most distributions, it tries
to collaborate with various upstream projects whenever possible.
Since GNU Boot relies on Guix, we improved the Guix documentation
directly to help people install Guix on Trisquel and Parabola. We also
helped Trisquel fix security issues in the Guix package by bug
reporting and testing fixes (some bugs still need to be fixed in
Parabola and Debian, and reporting issues upstream takes time).
Since we also advise to use PureOS or Trisquel to build GNU Boot we
also enabled people with Guix to produce PureOS or Trisquel chroots
with Debootstrap. This was done through contributions to Debootstrap,
and to the Guix Debootstrap package. We could then mention that in the
GNU Boot build documentation
(https://www.gnu.org/software/gnuboot/web/docs/build/) and added a
script (in contrib/start-guix-daemon.py) to support building GNU Boot
in chroots. However there are still issue with the build in chroots
that need to be fixed to producing all released files. Instructions on
how to do build in chroots is also lacking.
In addition we also added the ability to build GNU Boot with Trisquel
11 (aramo).
An apt-cacher-ng package was also contributed in Guix upstream as it
can then be used to speed-up one of the automatic tests used in GNU
Boot but the support for apt-cacher-ng was not integrated yet in GNU
Boot. Last year we also contributed a GRUB package in Guix but we
didn't have the occasion to use it yet. It will probably happen soon
though.
How to build GNU Boot has changed a lot since GNU Boot 0.1 RC3.
Before Guix could only be optionally used to build the website.
In addition to that, Guix is now integrated in the build system so we
can now rely on Guix packages to build GNU Boot images. This also
means that you need to install Guix to build GNU Boot images.
We currently use Guix packages to build some tests. We also build some
installation utilities for the I945 ThinkPads (ThinkPad X60, X60s,
X60T and T60) but we don't have documentation for less technical
people yet on how to use them. We also would need help for testing
these computers as we have no idea if they still work fine or which
fully free distributions still work on them in practice.
We now also support the './configure' and 'make' commands to build GNU
Boot but not yet the 'make install' command as to work we would need
to adapt many of the scripts that are used during the build to be
compatible with that.
There is also less visible work that was done, like cleaning up a lot
of code, adding tests for code quality, documenting a bit the GNU Boot
source code structure, and so on.
Work on making GNU Boot reproducible also started. See
https://reproducible-builds.org/ or
https://en.wikipedia.org/wiki/Reproducible_builds for more detail on
the issue.
We took an extremely strict approach and put the checksum of some of
the things we build directly into GNU Boot and verify it the checksum
during the compilation. This enables us to automatically detect
issues without having to do anything.
We started to enable that for easy things, and we also added the
infrastructure to also use that in Guix packages as well by validating
one of the packages we use during automatic testing.
However at one point this guix package stopped being
reproducible. Since we wanted to keep that code (especially as it was
showing a good example of how to do it), we fixed the bug instead of
removing the test.
This then helped us detect a very subtle and interesting bug in one of
the components we use for automatic tests.
The bug could not be caught during testing because some time
information stored inside the FAT32 file system has a granularity of a
day, and since all the testing happened the same day, it was caught
only later on.
This bug was then fixed and the details are in the fix[1]. A bug
report was also opened upstream because bugs were found in diffoscope
along the way[2]. We still need to do some testing though to
understand if the bug is in diffoscope or one of the underlying
libraries (libguestfs) and then to report the remaining bugs to the
distributions we used during this work.
We also made it easier to update the checksum in the Guix package. If
you package software with Guix, this change is also a good example of
how not to break the '--without-tests' option when you override the
tests in the package you contribute. The commit message[3] and the
change have more details and references on all that.
[1]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=4c3de49fbb3b43940b43f8fdccc8e51ee7df8f46
[2]https://salsa.debian.org/reproducible-builds/diffoscope/-/issues/390
[3]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=40fcb94e2f7ab1df8d320f78311e623f801d8602
WodeShengli reported a very important bug[1]: GNU Boot images with
GRUB can't find LVM2 partitions if the partition itself is not
encrypted. For instance if you have LVM2 and no encryption at all or
if the disk is encrypted and that on top you have LVM2, GNU Boot will
not find the partition.
Since this is an extremely serious usability issue (because images
with GRUB are supposed to work out of the box) we spent time to fix
it.
The issue was that the GRUB configuration we ship hardcoded the name
of the LVM volumes to try to boot from. Fixing it required to be able
loop over all the partitions being found, but we found no command to
do that in GRUB (which is probably why the LVM partition names were
hardcoded in the first place).
So we started adding GRUB command options to do that but while the
code worked fine, it didn't integrate in GRUB well. So we contacted
GRUB looking for help as we would have needed to upstream our command
option in GRUB anyway.
And we were told that GRUB already had a way to do what we were
looking for so we used that to fix the issue.
We also added tests that automatically download the Trisquel installer
and installs Trisquel with LVM2 and test if GNU Boot can boot the new
Trisquel installation[2].
While this test is skipped for 32bit computers, it is still good to
have as some people will run it. The test also paves the way to add
more tests that would enable us to improve further the GRUB
configuration without breaking the boot.
[1]https://savannah.gnu.org/bugs/index.php?65663
[2]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=860b00bf1e798d86c8bb2a70d77633599dfa1da2
[3]https://git.savannah.gnu.org/cgit/gnuboot.git/commit/?id=9cc02ddde1e164fabfbddc8bbd3832ef9468d92d
Real Python: How to Iterate Through a Dictionary in Python
Python offers several ways to iterate through a dictionary, such as using .items() to access key-value pairs directly and .values() to retrieve values only.
By understanding these techniques, youâll be able to efficiently access and manipulate dictionary data. Whether youâre updating the contents of a dictionary or filtering data, this guide will equip you with the tools you need.
By the end of this tutorial, youâll understand that:
- You can directly iterate over the keys of a Python dictionary using a for loop and access values with dict_object[key].
- You can iterate through a Python dictionary in different ways using the dictionary methods .keys(), .values(), and .items().
- You should use .items() to access key-value pairs when iterating through a Python dictionary.
- The fastest way to access both keys and values when you iterate over a dictionary in Python is to use .items() with tuple unpacking.
To get the most out of this tutorial, you should have a basic understanding of Python dictionaries, know how to use Python for loops, and be familiar with comprehensions. Knowing other tools like the built-in map() and filter() functions, as well as the itertools and collections modules, is also a plus.
Get Your Code: Click here to download the sample code that shows you how to iterate through a dictionary with Python.
Take the Quiz: Test your knowledge with our interactive âPython Dictionary Iterationâ quiz. Youâll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Python Dictionary IterationDictionaries are one of the most important and useful data structures in Python. Learning how to iterate through a Dictionary can help you solve a wide variety of programming problems in an efficient way. Test your understanding on how you can use them better!
Getting Started With Python DictionariesDictionaries are a cornerstone of Python. Many aspects of the language are built around dictionaries. Modules, classes, objects, globals(), and locals() are all examples of how dictionaries are deeply wired into Pythonâs implementation.
Hereâs how the Python official documentation defines a dictionary:
An associative array, where arbitrary keys are mapped to values. The keys can be any object with __hash__() and __eq__() methods. (Source)
There are a couple of points to notice in this definition:
- Dictionaries map keys to values and store them in an array or collection. The key-value pairs are commonly known as items.
- Dictionary keys must be of a hashable type, which means that they must have a hash value that never changes during the keyâs lifetime.
Unlike sequences, which are iterables that support element access using integer indices, dictionaries are indexed by keys. This means that you can access the values stored in a dictionary using the associated key rather than an integer index.
The keys in a dictionary are much like a set, which is a collection of hashable and unique objects. Because the keys need to be hashable, you canât use mutable objects as dictionary keys.
On the other hand, dictionary values can be of any Python type, whether theyâre hashable or not. There are literally no restrictions for values. You can use anything as a value in a Python dictionary.
Note: The concepts and topics that youâll learn about in this section and throughout this tutorial refer to the CPython implementation of Python. Other implementations, such as PyPy, IronPython, and Jython, could exhibit different dictionary behaviors and features that are beyond the scope of this tutorial.
Before Python 3.6, dictionaries were unordered data structures. This means that the order of items typically wouldnât match the insertion order:
Python >>> # Python 3.5 >>> likes = {"color": "blue", "fruit": "apple", "pet": "dog"} >>> likes {'color': 'blue', 'pet': 'dog', 'fruit': 'apple'} Copied!Note how the order of items in the resulting dictionary doesnât match the order in which you originally inserted the items.
In Python 3.6 and greater, the keys and values of a dictionary retain the same order in which you insert them into the underlying dictionary. From 3.6 onward, dictionaries are compact ordered data structures:
Python >>> # Python 3.6 >>> likes = {"color": "blue", "fruit": "apple", "pet": "dog"} >>> likes {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'} Copied!Keeping the items in order is a pretty useful feature. However, if you work with code that supports older Python versions, then you must not rely on this feature, because it can generate buggy behaviors. With newer versions, itâs completely safe to rely on the feature.
Another important feature of dictionaries is that theyâre mutable data types. This means that you can add, delete, and update their items in place as needed. Itâs worth noting that this mutability also means that you canât use a dictionary as a key in another dictionary.
Understanding How to Iterate Through a Dictionary in Python Read the full article at https://realpython.com/iterate-through-dictionary-python/ »[ Improve Your Python With đ Python Tricks đ â Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
This Week in Plasma: Battery Charge Cycles in Info Center
This week we of course continued the customary bug-fixing, but got some nice new features and UI improvements too!
Let me also remind folks about KDE's end-of-year fundraiser. We're 84% of the way to our goal, and it would be amazing to get all the way to 100% before December! Then we can focus on those stretch goals from December to January.
Anyway, enough of the sales pitch, back to the free stuff!
And isn't that amazing? Let's zoom out a bit here and remind ourselves just how incredible it is that this software is made available for free, with no contract or license agreement, to everyone. To you, to your school, to community organizations, businesses, governments, even our direct competitors to study and examine (which goes both ways, and helped me fix a bug in GTK this week; read on for details). It's kind of wild, if you think about it. But, here we are, and we want to keep on being a light in a tech world that sometimes seems to be darkening. Help us keep that light glowing!
Notable New FeaturesInfo Center now shows your battery's cycle count. (Kai Uwe Broulik, 6.3.0. Link 1 and link 2)
Added the ability to convert to and from the CFP franc currency in KRunner-powered searches. (someone going by the pseudonym "Mr. Athozus", Frameworks 6.9. Link)
Notable UI ImprovementsMiddle-clicking on the Brightness and Color widget no longer does anything when the Night Light hasn't been turned on. (Elias Probst, 6.2.4. Link)
Improved some sources of visual awkwardness in System Monitor: now the loading screen no longer sometimes has a scrollbar; and clicking something selected in a table view visibly de-selects it. (Akseli Lahtinen, 6.2.4. Link 1 and link 2)
Improved the way Discover presents external links to be less visually heavy. (Nate Graham, 6.3.0. Link)
Re-did the "Apply Plasma Settings" dialog on System Settings' Login Screen page to look better and more consistent with other dialogs in QML-based software these days. (Oliver Beard, 6.3.0. Link)
Notable Bug FixesFixed a regression in the Power and Battery widget that broke its ability to notice that power-profiles-daemon was installed instead of TLP after some porting work. (MĂ©ven Car, 6.2.4. Link)
Fixed a regression that caused the Disks & Devices widget to not show the correct actions for non-mounted optical discs after some porting work. (Kai Uwe Broulik, 6.2.4. Link)
Fixed an issue that caused screenshots and screen recordings to look too dim while using HDR mode. (Xaver Hugl, 6.2.4. Link)
Fixed a case where Plasma could crash when logging in with an external screen connected to a laptop via HDMI. (Marco Martin, 6.2.4. Link)
Fixed a rare case where Plasma could crash when copying data to the clipboard. (David Edmundson, 6.3.0. Link)
Fixed a bug affecting people using panels in "Fit content" mode that could, under certain circumstances, cause them to be too small until you manually entered Edit Mode once. (NiccolĂČ Venerandi, 6.3.0, Link)
KWin now behaves better when you plug in a weird defective TV that asks for an inappropriate resolution. (Xaver Hugl, 6.3.0. Link)
Discover once again shows update-able "Get New [Stuff]" content on the updates page. (Harald Sitter, 6.3.0. Link)
XWayland-using apps can no longer crash KWin with ludicrously large icon sizes. (David Redondo, Frameworks 6.9. Link)
Fixed a bizarre and annoying bug that caused text displayed at fractional scale factors in Plasma and QtQuick-based KDE apps and to look, for lack of a better term, wobbly. Wobbly windows good, wobbly text bad! Text has now been put on the straight and narrow. (David Edmundson, Frameworks 6.9. Link)
Fixed a strange Qt bug that manifested as Plasma notifications sometimes being vertically squished. (David Edmundson, Qt 6.8.1. Link)
GTK 3 apps once again have the correct icon for their spinboxes' "decrease the value" buttons when using the Breeze icon theme or any other icon theme whose list-remove icon isn't a minus sign. (Nate Graham, GTK 3.24.44. Link)
Other bug information of note:
- 2 Very high priority Plasma bugs (same as last week). Current list of bugs
- 35 15-minute Plasma bugs (down from 36 last week). Current list of bugs
- 94 KDE bugs of all kinds fixed over the last week. Full list of bugs
The feature to let you record the screen without re-approval now also works for virtual outputs. Additionally, virtual outputs now have a better name that indicates which app records them. (David Redondo, 6.3.0. Link)
Fixed a memory leak caused by having a lot of OverlayFS mounts, e.g. from Docker containers. (Joshua Goins, Frameworks 6.9. Link)
How You Can HelpKDE has become important in the world, and your time and contributions have helped us get there. As we grow, we need your support to keep KDE sustainable.
You can help KDE by becoming an active community member and getting involved somehow. Each contributor makes a huge difference in KDE â you are not a number or a cog in a machine!
You donât have to be a programmer, either. Many other opportunities exist:
- Filter and confirm bug reports, maybe even identify their root cause
- Contribute designs for wallpapers, icons, and app interfaces
- Design and maintain websites
- Translate user interface text items into your own language
- Promote KDE in your local community
- âŠAnd a ton more things!
You can also help us by donating to our yearly fundraiser! Any monetary contribution â however small â will help us cover operational costs, salaries, travel expenses for contributors, and in general just keep KDE bringing Free Software to the world.
To get a new Plasma feature or a bugfix mentioned here, feel free to push a commit to the relevant merge request on invent.kde.org.
Eli Bendersky: GoMLX: ML in Go without Python
In the previous post I talked about running ML inference in Go through a Python sidecar process. In this post, let's see how we can accomplish the same tasks without using Python at all.
How ML models are implementedLet's start with a brief overview of how ML models are implemented under the hood [1]. The model is typically written in Python, using one of the ML frameworks like TensorFlow, JAX or PyTorch. The framework takes care of at least 2 high-level concerns for developers:
- Expressive way to describe the model architecture, including auto-differentiation for training.
- Efficient implementation of computational primitives on common HW: CPUs, GPUs and TPUs.
In-between these two concerns there exists a standardized model definition format (or several) that helps multiple tools interoperate. While it's by no means the only solution [2], let's look at the OpenXLA stack as a way to run models on diverse hardware:
- The top layer are the frameworks that provide high-level primitives to define ML models, and translate them to a common interchange format called StableHLO (where "HLO" stands for High-Level Operations). I've added the gopher on the very right - it will soon become clear why.
- The bottom layer is the HW that executes these models efficiently.
- In the middle is the OpenXLA system, which includes two major components: the XLA compiler translating HLO to HW machine code, and PJRT - the runtime component responsible for managing HW devices, moving data (tensors) between the host CPU and these devices, executing tasks, sharding and so on.
There's a huge amount of complexity hidden by the bottom layers of this diagram. Efficient compilation and code generation for diverse HW - including using fixed blocks and libraries (like cuDNN), runtime management etc. All of this is really something one shouldn't try to re-implement unless there's a really, really good reason to do so. And the best part? There's no Python there - this is C and C++; Python only exists on the upper layer - in the high-level ML frameworks.
GoMLXGoMLX is a relatively new Go package for ML that deserves some attention. GoMLX slots in as one of the frameworks, exactly where the Gopher is in the diagram above [3]. This is absolutely the right approach to the problem. There's no point in re-implementing the low-level primitives - whatever works for TF and JAX will work for Go as well! Google, NVIDIA, Intel and several other companies invest huge resources into these systems, and it's a good idea to benefit from these efforts.
In this post I will showcase re-implementations of some of the samples from the previous post, but with no Python in sight. But first, a few words about what GoMLX does.
GoMLX should be familiar if you've used one of the popular Python ML frameworks. You build a computational graph representing your model - the usual operations are supported and sufficient to implement anything from linear regression to cutting-edge transformers. Since GoMLX wraps XLA, it has access to all the same building blocks TF and JAX use (and it adds its own higher-level primitives, similarly to the Python frameworks).
GoMLX supports automatic differentiation to create the backward propagation operations required to update weights in training. It also provides many helpers for training and keeping track of progress, as well as Jupyter notebook support.
An image model for the CIFAR-10 dataset with GoMLXIn the previous post we built a CNN (convolutional neural network) model using TF+Keras in Python, and ran its inference in a sidecar process we could control from Go.
Here, let's build a similar model in Go, without using Python at all; we'll be training it on the same CIFAR-10 dataset we've used before.
The full code for this sample is here; it is heavily based on GoMLX's own example, with some modifications for simplicity and clarity. Here's the code defining the model graph:
func C10ConvModel(mlxctx *mlxcontext.Context, spec any, inputs []*graph.Node) []*graph.Node { batchedImages := inputs[0] g := batchedImages.Graph() dtype := batchedImages.DType() batchSize := batchedImages.Shape().Dimensions[0] logits := batchedImages layerIdx := 0 nextCtx := func(name string) *mlxcontext.Context { newCtx := mlxctx.Inf("%03d_%s", layerIdx, name) layerIdx++ return newCtx } // Convolution / activation layers logits = layers.Convolution(nextCtx("conv"), logits).Filters(32).KernelSize(3).PadSame().Done() logits.AssertDims(batchSize, 32, 32, 32) logits = activations.Relu(logits) logits = layers.Convolution(nextCtx("conv"), logits).Filters(32).KernelSize(3).PadSame().Done() logits = activations.Relu(logits) logits = graph.MaxPool(logits).Window(2).Done() logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.3), true) logits.AssertDims(batchSize, 16, 16, 32) logits = layers.Convolution(nextCtx("conv"), logits).Filters(64).KernelSize(3).PadSame().Done() logits.AssertDims(batchSize, 16, 16, 64) logits = activations.Relu(logits) logits = layers.Convolution(nextCtx("conv"), logits).Filters(64).KernelSize(3).PadSame().Done() logits.AssertDims(batchSize, 16, 16, 64) logits = activations.Relu(logits) logits = graph.MaxPool(logits).Window(2).Done() logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.5), true) logits.AssertDims(batchSize, 8, 8, 64) logits = layers.Convolution(nextCtx("conv"), logits).Filters(128).KernelSize(3).PadSame().Done() logits.AssertDims(batchSize, 8, 8, 128) logits = activations.Relu(logits) logits = layers.Convolution(nextCtx("conv"), logits).Filters(128).KernelSize(3).PadSame().Done() logits.AssertDims(batchSize, 8, 8, 128) logits = activations.Relu(logits) logits = graph.MaxPool(logits).Window(2).Done() logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.5), true) logits.AssertDims(batchSize, 4, 4, 128) // Flatten logits, and apply dense layer logits = graph.Reshape(logits, batchSize, -1) logits = layers.Dense(nextCtx("dense"), logits, true, 128) logits = activations.Relu(logits) logits = layers.DropoutNormalize(nextCtx("dropout"), logits, graph.Scalar(g, dtype, 0.5), true) numClasses := 10 logits = layers.Dense(nextCtx("dense"), logits, true, numClasses) return []*graph.Node{logits} }As you might expect, the Go code is longer and more explicit (nodes are threaded explicitly between builder calls, instead of being magically accumulated). It's not hard to envision a Keras-like high level library on top of this.
Here's a snippet from the classifier (inference):
func main() { flagCheckpoint := flag.String("checkpoint", "", "Directory to load checkpoint from") flag.Parse() mlxctx := mlxcontext.New() backend := backends.New() _, err := checkpoints.Load(mlxctx).Dir(*flagCheckpoint).Done() if err != nil { panic(err) } mlxctx = mlxctx.Reuse() // helps sanity check the loaded context exec := mlxcontext.NewExec(backend, mlxctx.In("model"), func(mlxctx *mlxcontext.Context, image *graph.Node) *graph.Node { // Convert our image to a tensor with batch dimension of size 1, and pass // it to the C10ConvModel graph. image = graph.ExpandAxes(image, 0) // Create a batch dimension of size 1. logits := cnnmodel.C10ConvModel(mlxctx, nil, []*graph.Node{image})[0] // Take the class with highest logit value, then remove the batch dimension. choice := graph.ArgMax(logits, -1, dtypes.Int32) return graph.Reshape(choice) }) // classify takes a 32x32 image and returns a Cifar-10 classification according // to the models. Use C10Labels to convert the returned class to a string // name. The returned class is from 0 to 9. classify := func(img image.Image) int32 { input := images.ToTensor(dtypes.Float32).Single(img) outputs := exec.Call(input) classID := tensors.ToScalar[int32](outputs[0]) return classID } // ...Now classify is a function that takes an image.Image and runs it through the network, returning the index of the most likely label out of the list of CIFAR-10 labels.
The README file in the sample explains how to run it locally on a GPU; the model trains and runs successfully, with similar results to the TF+Keras model we trained in Python earlier.
Gemma2 with GoMLXFor a (much) more involved example, GoMLX has a full implementation of Gemma2 inference. The model implementation itself is in the transformers package. It should look fairly familiar if you've seen a transformer implementation in another language.
The official example in that repository shows how to run it with weights downloaded from HuggingFace; since I've already downloaded the Gemma2 weights from Kaggle for the previous post, here's a simple adaptation:
var ( flagDataDir = flag.String("data", "", "dir with converted weights") flagVocabFile = flag.String("vocab", "", "tokenizer vocabulary file") ) func main() { flag.Parse() ctx := context.New() // Load model weights from the checkpoint downloaded from Kaggle. err := kaggle.ReadConvertedWeights(ctx, *flagDataDir) if err != nil { log.Fatal(err) } // Load tokenizer vocabulary. vocab, err := sentencepiece.NewFromPath(*flagVocabFile) if err != nil { log.Fatal(err) } // Create a Gemma sampler and start sampling tokens. sampler, err := samplers.New(backends.New(), ctx, vocab, 256) if err != nil { log.Fatalf("%+v", err) } start := time.Now() output, err := sampler.Sample([]string{ "Are bees and wasps similar?", }) if err != nil { log.Fatalf("%+v", err) } fmt.Printf("\tElapsed time: %s\n", time.Since(start)) fmt.Printf("Generated text:\n%s\n", strings.Join(output, "\n\n")) }The complete code together with installation and setup instructions is here.
gomlx/gemma demonstrates that GoMLX has sufficiently advanced capabilities to run a real production-grade open LLM, without Python in the loop.
SummaryThe previous post discussed some options for incorporating ML inference into a Go project via a minimal Python sidecar process. Here, we take it a step further and implement ML inference in Go without using Python. We do so by leveraging GoMLX, which itself relies on XLA and PJRT to do the heavy lifting.
If we strip down a framework like TensorFlow to its layers, GoMLX reuses the bottom layers (which is where most of the magic lies), and replaces the model builder library with a Go variant.
Since GoMLX is still a relatively new project, it may be a little risky for production uses at this point. That said, I find this direction very promising and will be following the project's development with interest.
CodeThe full code for the samples in this post is on GitHub.
[1]This assumes you know the basics of neural network graphs, their training, etc. If not, check out this post and some of my other posts in the Machine Learning category. [2]It's likely the most common production solution, and pretty much the only way to access Google's TPUs. [3]It does so by including Go bindings for both XLA and PJRT; these are wrapped in higher-level APIs for users.parallel @ Savannah: GNU Parallel 20241122 ('Ahoo Daryaei') released
GNU Parallel 20241122 ('Ahoo Daryaei') has been released. It is available for download at: lbry://@GnuParallel:4
Quote of the month:
 GNU parallel is so satisfying
   -- James Coman @jcoman.bsky.social
New in this release:
- --pipe --block works similar to --pipepart --block if --block size is negative.
- DBURLs can be written with / instead of %2F for sqlite and CSV.
- Bug fixes and man page updates.
News about GNU Parallel:
- Embarrassingly GNU parallel https://dengin.xyz/blog/2024/10/24/embarrassingly-gnu-parallel/
- GNU Parallel for Your Terminal Tasks https://erolrecep.github.io/posts/gnuparallel_for_your_terminal_tasks/
- How to leverage GNU parallel to utilize multiple cores while running AUGUSTUS https://lifescienceshub.wixsite.com/lifesciencehub/post/how-to-leverage-gnu-parallel-to-utilize-multiple-cores-while-running-augustus
- GNU Parallel: The Good Parts https://diekmeier.de/posts/2024-11-17-gnu-parallel/
- Put your CPU to work with GNU Parallel https://www.redhat.com/en/blog/gnu-parallel
GNU Parallel - For people who live life in the parallel lane.
If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.
GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.
If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.
GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.
For example you can run this to convert all jpeg files into png and gif files and have a progress bar:
 parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif
Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:
 find . -name '*.jpg' |
   parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200
You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
You can install GNU Parallel in just 10 seconds with:
   $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
      fetch -o - http://pi.dk/3 ) > install.sh
   $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
   12345678 883c667e 01eed62f 975ad28b 6d50e22a
   $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
   cc21b4c9 43fd03e9 3ae1ae49 e28573c0
   $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
   79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
   fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
   $ bash install.sh
Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.
When using programs that use GNU Parallel to process data for publication please cite:
O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.
If you like GNU Parallel:
- Give a demo at your local user group/team/colleagues
- Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
- Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
- Request or write a review for your favourite blog or magazine
- Request or build a package for your favourite distribution (if it is not already there)
- Invite me for your next conference
If you use programs that use GNU Parallel for research:
- Please cite GNU Parallel in you publications (use --citation)
If GNU Parallel saves you money:
- (Have your company) donate to FSF https://my.fsf.org/donate/
GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.
The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.
When using GNU SQL for a publication please cite:
O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.
GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.
Matthew Palmer: Your Release Process Sucks
For the past decade-plus, every piece of software I write has had one of two release processes.
Software that gets deployed directly onto servers (websites, mostly, but also the infrastructure that runs Pwnedkeys, for example) is deployed with nothing more than git push prod main. Iâll talk more about that some other day.
Today is about the release process for everything else I maintain â Rust / Ruby libraries, standalone programs, and so forth. To release those, I use the following, extremely intricate process:
-
Create an annotated git tag, where the name of the tag is the software version Iâm releasing, and the annotation is the release notes for that version.
-
Run git release in the repository.
-
There is no step 3.
Yes, it absolutely is that simple. And if your release process is any more complicated than that, then you are suffering unnecessarily.
But donât worry. Iâm from the Internet, and Iâm here to help.
Sidebar: âannotated what-now?!?âThe annotated tag is one gitâs best-kept secrets. Theyâve been available in git for practically forever (Iâve been using them since at least 2014, which is âpractically foreverâ in software development), yet almost everyone I mention them to has never heard of them.
A âtagâ, in git parlance, is a repository-unique named label that points to a single commit (as identified by the commitâs SHA1 hash). Annotating a tag is simply associating a block of free-form text with that tag.
Creating an annotated tag is simple-sauce: git tag -a tagname will open up an editor window where you can enter your annotation, and git tag -a -m "some annotation" tagname will create the tag with the annotation âsome annotationâ. Retrieving the annotation for a tag is straightforward, too: git show tagname will display the annotation along with all the other tag-related information.
Now that we know all about annotated tags, letâs talk about how to use them to make software releases freaking awesome.
Step 1: Create the Annotated Git TagAs I just mentioned, creating an annotated git tag is pretty simple: just add a -a (or --annotate, if you enjoy typing) to your git tag command, and WHAM! annotation achieved.
Releases, though, typically have unique and ever-increasing version numbers, which we want to encode in the tag name. Rather than having to look at the existing tags and figure out the next version number ourselves, we can have software do the hard work for us.
Enter: git-version-bump. This straightforward program takes one mandatory argument: major, minor, or patch, and bumps the corresponding version number component in line with Semantic Versioning principles. If you pass it -n, it opens an editor for you to enter the release notes, and when you save out, the tag is automagically created with the appropriate name.
Because the program is called git-version-bump, you can call it as a git command: git version-bump. Also, because version-bump is long and unwieldy, I have it aliased to vb, with the following entry in my ~/.gitconfig:
[alias] vb = version-bump -nOf course, you donât have to use git-version-bump if you donât want to (although why wouldnât you?). The important thing is that the only step you take to go from âhere is our current codebase in mainâ to âeverything as of this commit is version X.Y.Z of this softwareâ, is the creation of an annotated tag that records the version number being released, and the metadata that goes along with that release.
Step 2: Run git releaseAs I said earlier, Iâve been using this release process for over a decade now. So long, in fact, that when I started, GitHub Actions didnât exist, and so a lot of the things youâd delegate to a CI runner these days had to be done locally, or in a more ad-hoc manner on a server somewhere.
This is why step 2 in the release process is ârun git releaseâ. Itâs because historically, you canât do everything in a CI run. Nowadays, most of my repositories have this in the .git/config:
[alias] release = push --tagsOlder repositories which, for one reason or another, havenât been updated to the new hawtness, have various other aliases defined, which run more specialised scripts (usually just rake release, for Ruby libraries), but theyâre slowly dying out.
The reason why I still have this alias, though, is that it standardises the release process. Whether itâs a Ruby gem, a Rust crate, a bunch of protobuf definitions, or whatever else, I run the same command to trigger a release going out. It means I donât have to think about how I do it for this project, because every project does it exactly the same way.
The Wiring Behind the ButtonIt wasnât the button that was the problem. It was the miles of wiring, the hundreds of miles of cables, the circuits, the relays, the machinery. The engine was a massive, sprawling, complex, mind-bending nightmare of levers and dials and buttons and switches. You couldnât just slap a button on the wall and expect it to work. But there should be a button. A big, fat button that you could press and everything would be fine again. Just press it, and everything would be back to normal.
- Red Dwarf: Better Than Life
Once youâve accepted that your release process should be as simple as creating an annotated tag and running one command, you do need to consider what happens afterwards. These days, with the near-universal availability of CI runners that can do anything you need in an isolated, reproducible environment, the work required to go from âannotated tagâ to ârelease artifactsâ can be scripted up and left to do its thing.
What that looks like, of course, will probably vary greatly depending on what youâre releasing. I canât really give universally-applicable guidance, since I donât know your situation. All I can do is provide some of my open source work as inspirational examples.
For starters, letâs look at a simple Rust crate Iâve written, called strong-box. Itâs a straightforward crate, that provides ergonomic and secure cryptographic functionality inspired by the likes of NaCl. As itâs just a crate, its release script is very straightforward. Most of the complexity is working around Cargoâs inelegant mandate that crate version numbers are specified in a TOML file. Apart from that, itâs just a matter of building and uploading the crate. Easy!
Slightly more complicated is action-validator. This is a Rust CLI tool which validates GitHub Actions and Workflows (how very meta) against a published JSON schema, to make sure you havenât got any syntax or structural errors. As not everyone has a Rust toolchain on their local box, the release process helpfully build binaries for several common OSes and CPU architectures that people can download if they choose. The release process in this case is somewhat larger, but not particularly complicated. Almost half of it is actually scaffolding to build an experimental WASM/NPM build of the code, because someone seemed rather keen on that.
Moving away from Rust, and stepping up the meta another notch, we can take a look at the release process for git-version-bump itself, my Ruby library and associated CLI tool which started me down the âJust Tag It Alreadyâ rabbit hole many years ago. In this case, since gemspecs are very amenable to programmatic definition, the release process is practically trivial. Remove the boilerplate and workarounds for GitHub Actions bugs, and youâre left with about three lines of actual commands.
These approaches can certainly scale to larger, more complicated processes. Iâve recently implemented annotated-tag-based releases in a proprietary software product, that produces Debian/Ubuntu, RedHat, and Windows packages, as well as Docker images, and it takes all of the information it needs from the annotated tag. Iâm confident that this approach will successfully serve them as they expand out to build AMIs, GCP machine images, and whatever else they need in their release processes in the future.
Objection, Your Honour!I can hear the howl of the âbut, actuallysâ coming over the horizon even as I type. People have a lot of Big Feelings about why this release process wonât work for them. Rather than overload this article with them, Iâve created a companion article that enumerates the objections Iâve come across, and answers them. Iâm also available for consulting if youâd like a personalised, professional opinion on your specific circumstances.
DVD Bonus Feature: Pre-releasesUnless youâre addicted to surprises, itâs good to get early feedback about new features and bugfixes before they make it into an official, general-purpose release. For this, you canât go past the pre-release.
The major blocker to widespread use of pre-releases is that cutting a release is usually a pain in the behind. If youâve got to edit changelogs, and modify version numbers in a dozen places, then youâre entirely justified in thinking that cutting a pre-release for a customer to test that bugfix that only occurs in their environment is too much of a hassle.
The thing is, once youâve got releases building from annotated tags, making pre-releases on every push to main becomes practically trivial. This is mostly due to another fantastic and underused Git command: git describe.
How git describe works is, basically, that it finds the most recent commit that has an associated annotated tag, and then generates a string that contains that tagâs name, plus the number of commits between that tag and the current commit, with the current commitâs hash included, as a bonus. That is, imagine that three commits ago, you created an annotated release tag named v4.2.0. If you run git describe now, it will print out v4.2.0-3-g04f5a6f (assuming that the current commitâs SHA starts with 04f5a6f).
You might be starting to see where this is going. With a bit of light massaging (essentially, removing the leading v and replacing the -s with .s), that string can be converted into a version number which, in most sane environments, is considered ânewerâ than the official 4.2.0 release, but will be superceded by the next actual release (say, 4.2.1 or 4.3.0). If youâre already injecting version numbers into the release build process, injecting a slightly different version number is no work at all.
Then, you can easily build release artifacts for every commit to main, and make them available somewhere they wonât get in the way of the âofficialâ releases. For example, in the proprietary product I mentioned previously, this involves uploading the Debian packages to a separate component (prerelease instead of main), so that users that want to opt-in to the prerelease channel simply modify their sources.list to change main to prerelease. Management have been extremely pleased with the easy availability of pre-release packages; theyâve been gleefully installing them willy-nilly for testing purposes since I rolled them out.
In fact, even while Iâve been writing this article, I was asked to add some debug logging to help track down a particularly pernicious bug. I added the few lines of code, committed, pushed, and went back to writing. A few minutes later (next weekâs job is to cut that in-process time by at least half), the person who asked for the extra logging ran apt update; apt upgrade, which installed the newly-built package, and was able to progress in their debugging adventure.
Continuous Delivery: Itâs Not Just For Hipsters.
â+1, InformativeâHopefully, this has spurred you to commit your immortal soul to the Church of the Annotated Tag. You may tithe by buying me a refreshing beverage. Alternately, if youâre really keen to adopt more streamlined release management processes, Iâm available for consulting engagements.
Matthew Palmer: Invalid Excuses for Why Your Release Process Sucks
In my companion article, I made the bold claim that your release process should consist of no more than two steps:
-
Create an annotated Git tag;
-
Run a single command to trigger the release pipeline.
As I have been on the Internet for more than five minutes, Iâm aware that a great many people will have a great many objections to this simple and straightforward idea. In the interests of saving them a lot of wear and tear on their keyboards, I present this list of common reasons why these objections are invalid.
If you have an objection I donât cover here, the comment box is down the bottom of the article. If you think youâve got a real stumper, Iâm available for consulting engagements, and if you turn out to have a release process which cannot feasibly be reduced to the above two steps for legitimate technical reasons, Iâll waive my fees.
âBut I automatically generate my release notes from commit messages!âThis one is really easy to solve: have the release note generation tool feed directly into the annotation. Boom! Headshot.
âBut all these files need to be edited to make a release!âNo, they absolutely donât. But I can see why you might think you do, given how inflexible some packaging environments can seem, and since âthatâs how weâve always done itâ.
Language PackagesMost languages require you to encode the version of the library or binary in a file that you want to revision control. This is teh suck, but Iâm yet to encounter a situation that canât be worked around some way or another.
In Ruby, for instance, gemspec files are actually executable Ruby code, so I call code (thatâs part of git-version-bump, as an aside) to calculate the version number from the git tags. The Rust build tool, Cargo, uses a TOML file, which isnât as easy, but a small amount of release automation is used to take care of that.
Distribution PackagesIf youâre building Linux distribution packages, you can easily apply similar automation faffery. For example, Debian packages take their metadata from the debian/changelog file in the build directory. Donât keep that file in revision control, though: build it at release time. Everything you need to construct a Debian (or RPM) changelog is in the tag â version numbers, dates, times, authors, release notes. Use it for much good.
The Dreaded ChangelogFinally, thereâs the CHANGELOG file. If itâs maintained during the development process, it typically has an archive of all the release notes, under version numbers, with an âUnreleasedâ heading at the top. Itâs one more place to remember to have to edit when making that âpreparing release X.Y.Zâ commit, and it is a gift to the Demon of Spurious Merge Conflicts if you follow the policy of âevery commit must add a changelog entryâ.
My solution: just burn it to the ground. Add a line to the top with a link to wherever the contents of annotated tags get published (such as GitHub Releases, if thatâs your bag) and never open it ever again.
âBut I need to know other things about my release, too!âFor some reason, you might think you need some other metadata about your releases. Youâre probably wrong â itâs amazing how much information you can obtain or derive from the humble tag â so think creatively about your situation before you start making unnecessary complexity for yourself.
But, on the off chance youâre in a situation that legitimately needs some extra release-related information, hereâs the secret: structured annotation. The annotation on a tag can be literally any sequence of octets you like. How that data is interpreted is up to you.
So, require that annotations on release tags use some sort of structured data format (say YAML or TOML â or even XML if you hate your release manager), and mandate that it contain whatever information you need. You can make sure that the annotation has a valid structure and contains all the information you need with an update hook, which can reject the tag push if it doesnât meet the requirements, and youâre sorted.
âBut I have multiple packages in my repo, with different release cadences and versions!âThis one is common enough that I just refer to it as âthe monorepo dramaâ. Personally, Iâm not a huge fan of monorepos, but you do you, boo. Annotated tags can still handle it just fine.
The trick is to include the package name being released in the tag name. So rather than a release tag being named vX.Y.Z, you use foo/vX.Y.Z, bar/vX.Y.Z, and baz/vX.Y.Z. The release automation for each package just triggers on tags that match the pattern for that particular package, and limits itself to those tags when figuring out what the version number is.
âBut we donât semver our releases!âOh, thatâs easy. The tag pattern that marks a release doesnât have to be vX.Y.Z. It can be anything you want.
Relatedly, there is a (rare, but existent) need for packages that donât really have a conception of âreleasesâ in the traditional sense. The example Iâve hit most often is automatically generated âbindingsâ packages, such as protobuf definitions. The source of truth for these is a bunch of .proto files, but to be useful, they need to be packaged into code for the various language(s) youâre using. But those packages need versions, and while someone could manually make releases, the best option is to build new per-language packages automatically every time any of those definitions change.
The versions of those packages, then, can be datestamps (I like something like YYYY.MM.DD.N, where N starts at 0 each day and increments if there are multiple releases in a single day).
This process allows all the code that needs the definitions to declare the minimum version of the definitions that it relies on, and everything is kept in sync and tracked almost like magic.
Th-th-th-th-thatâs all, folks!I hope youâve enjoyed this bit of mild debunking. Show your gratitude by buying me a refreshing beverage, or purchase my professional expertise and Iâll answer all of your questions and write all your CI jobs.
EuroPython Society: 2024 General Assembly Announcement
We’re excited to invite you to this year’s General Assembly meeting! We’ll gather on Sunday, December 1st, 2024, from 20:00 to 21:00 CET. Just like in recent years, we’ll use Zoom, and additional joining instructions will be shared closer to the date.
The General Assembly is the highest decision making body of the society and EPS membership is required to participate. Membership is open to individuals who wish to actively engage in implementing the EPS mission. If you want to become a member of EuroPython Society you can sign-up here: https://www.europython-society.org/application/
You can find more details about the agenda of the meeting, as it is defined in our bylaws here https://www.europython-society.org/bylaws/ (Article 8).
One of the items on the Agenda is electing the new Board.
What does the Board do?The Board consists of a chairperson, a vice chairperson and 2-7 Board members. The duties and responsibilities of the Board are substantial: the board collectively takes up the fiscal and legal responsibility of the Society.
A major topic is the annual EuroPython conference. While we would like to transition to a model with an independent organising team, we are not there yet. Therefore, the Board still needs to be involved in the conference organisation.
Beyond the conference, the Board also manages several critical areas, including:
- Managing EPS membership
- Overseeing finances and budgets
- Running the grant programme
- Maintaining infrastructure and resources
Furthermore, specifically for 2025, and following the recommendation from the previous Board, we would like to focus on four key topics that are important for the Society&aposs future and sustainability:
- Hiring an Event Manager/Coordinator
- Selecting a location for 2026 and possibly 2027
- Strengthen community outreach
- Improving the fiscal and legal framework
The Society is entirely volunteer-driven and serving on the board requires a significant time commitment. Everyone has a different schedule, so most of the work is usually done asynchronously. However, all board members attend the 1.5-hour board call held every two weeks in the evening, CE(S)T timezone. Everyone&aposs time is valuable and please consider that the less time or effort you can dedicate, the more the workload may shift to other Board members.
All things considered you will need a few hours every week.
Who should apply?You want to invest your time and knowledge into building a better structure for the EuroPython Society? Or you want to work on building connections between different Python-based communities? Then this might be for you! Please keep in mind the time commitments mentioned above.
You are not expected to be perfect in any of the skills needed and you will be supported in learning how things work. That being said, having experience in a non-profit organisation, whether within the Python world (such as EPS, PSF, DSF, local Python communities etc.) or any other similar organisation, would be beneficial for onboarding and understanding the organisational structure, culture and dynamics.
In the past having or willing to learn the following skills helped organising the conference:
- Good communication skills
- Organisation skills
- Experience organising events with more than 1000 people
- Working with volunteer-based communities
- Working in big teams
You get the chance to shape and influence the future of EuroPython
You gain skills useful to run non-profits in different European countries - including cross border challenges
You can help grow and empower local communities
You can build relationships and connections with fellow community members
You can build a more diverse and inclusive Python community by serving the mission of EuroPython Society
I am interested, what should I do?If you’re considering running for the Board or nominating another EPS member, we’d love to hear from you! Although the formal deadline is during the General Assembly, we kindly request you send your nomination as early as possible to board@europython.eu. We will publish the initial list of candidates on Tuesday, 26th of November 2024. If you’re not sure if this is a good idea or not – please email anyway and we will help you figure it out! 🙂
If you&aposre on our EPS Organisers&apos Discord, there&aposs a dedicated channel for interested candidates. Please ask in the general channel, and we’ll be happy to add you.
You can find examples of previous nominations here: https://www.europython-society.org/list-of-eps-board-candidates-for-2023-2024/.
Your nomination should highlight why you want to run for the Board. What is your vision for EPS and in which projects you want to be involved. During the General Assembly, you will have the opportunity to introduce yourself and share with our members why you believe they should vote for you. Each candidate will typically be given one minute to present themselves before members cast their votes.
It sounds a lot, I want to help, but I can’t commit to thatThat’s completely understandable! Serving on the Board comes with significant responsibilities, time commitments, and administrative tasks. If that’s not the right fit for you, but you’re still interested in supporting us, we’d love your help! There are many other ways to get involved. We have several teams (see 2024 Teams Description document, as an example) that work on conference preparations during the months leading up to the event, and we also need volunteers to assist onsite during the conference.
Your help does not need to be limited to the conference. Infrastructure and connections need to be maintained all around the year for example. Your time and support would make a big difference! Stay tuned to our social platforms for announcements about these opportunities.
The Drop Times: It's About Time to Discuss Drupal Event Recipes
mark.ie: My LocalGov Drupal contributions for week-ending November 22nd, 2024
This week, lots of work on the LocalGov News module.
Web Review, Week 2024-47
Let’s go for my web review for the week 2024-47.
The Big Data Center Water ProblemTags: tech, hardware, ecology, economics, energy, water
We always think about the energy consumption, but large data centers gobble billion liters of water too. This would need to be improved.
https://www.asianometry.com/p/the-big-data-center-water-problem
Tags: tech, vr, hardware, foss
Nice to see open hardware for VR hitting such a price point.
Tags: tech, social-media, fediverse, tools
You’re on the fediverse and you want to reach out bluesky users? This might be the right tool for you (unclear if it’ll scale yet though). At least if and when Bluesky turns bad, people will know where to reach friends next.
Tags: tech, social-media, business, politics
Excellent post showing reasons to be skeptical about Bluesky’s future. Despite all their likely sincere claims I don’t see how they’ll escape enclosure and enshittification when their sketchy VCs will want to see money back.
https://www.tbray.org/ongoing/When/202x/2024/11/15/Not-Bluesky
Tags: tech, social-media, politics, twitter
Sad to see people predominantly jumping from Twitter to other tech moguls walled gardens. This feels more and more like a missed opportunity for the fediverse. That said I’m amazed at how efficient Musk has been at killing the network effect of his platform. This proves it’s actually doable.
Tags: tech, social-media, politics, twitter
This is what we get for refusing to regulate social media and for not auditing their algorithms. Their owners can game and bias the platforms as they see fit for their own gains. They became massive forces of manipulation in the process.
https://eprints.qut.edu.au/253211/
Tags: tech, ai, machine-learning, gpt, vendor-lockin
Good reminder that models shouldn’t be used as a service except maybe for prototyping. This has felt obvious to me since the beginning of this hype cycle… but here we are people are falling in the trap today.
https://adriano.fyi/posts/chatgpt-is-slipping/
Tags: tech, python, performance, pandas, data, data-science
OK, the numbers are indeed impressive. And it’s API is fully compatible apparently, looks like a good replacement if you got Pandas code around.
https://hwisnu.bearblog.dev/fireducks-pandas-but-100x-faster/
Tags: tech, tools, debugging
Looks like a nice tool. Maybe it’ll replace my trusty cgdb in some cases.
https://github.com/epasveer/seer
Tags: tech, c++, security
Will we see more deployments of C++ standard library with bound checking by default? It definitely looks tempting.
https://security.googleblog.com/2024/11/retrofitting-spatial-safety-to-hundreds.html?m=1
Tags: tech, php, security
Seeing the amount of PHP code open on the internet, it’s indeed important to harden the runtime (at long last).
https://dustri.org/b/upcoming-hardening-in-php.html
Tags: tech, graphics, gpu
Really nice in depth post. Everything you ever wanted to know about antialiasing but didn’t dare asking.
https://blog.frost.kiwi/analytical-anti-aliasing/
Tags: tech, framework, career, learning
Good advice, no one should be a “React developer”. Make sure you learn more fundamental skills.
https://www.keithcirkel.co.uk/i-dont-have-time-to-learn-react/
Tags: tech, craftsmanship, learning
If you’re just doing the minimum to deal with a task to “mark it done” you’re probably not doing enough and missing out on learning opportunities.
https://edanparker.hashnode.dev/going-a-little-further
Tags: tech, career, learning, engineering
This can change from organization to organization. This post proposes a career ladder which will work in some contexts. What’s clear is that it’s all about scope and impact.
https://matt.blwt.io/post/what-is-a-senior-engineer-anyway/
Tags: tech, engineering, management, learning
Interesting tips to keep learning on the technical side of the job as you get more managerial responsibilities.
Bye for now!
Real Python: The Real Python Podcast â Episode #229: The Joy of Tinkering & Python Free-Threading Performance
What keeps your spark alive for developing software and learning Python? Do you like to try new frameworks, build toy projects, or collaborate with other developers? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.
[ Improve Your Python With đ Python Tricks đ â Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Talk Python to Me: #486: CSnakes: Embed Python code in .NET
Krita for Android Update
We have updated Krita for Android and ChromeOS in the Google Play Store to 5.2.8, an Android/ChromeOS-only emergency release. This release fixes startup problems that happened on some devices with 5.2.6. Krita 5.2.8 for Android is now available both for beta-track users as well as in the "stable" release track. Note, however, that we still recommend treating Krita on Android as a beta release that might have bugs that impair your work, as well as a user interface that is not optimized for touch devices.
Matt Layman: Huey Background Worker - Building SaaS #207
Pages
Recent Publications
- Managing Hidden Dependencies in OO Software: a Study Based on Open Source Projects
- Open Source Communities as Liminal Ecosystems
- Investigating developers' email discussions during decision-making in Python language evolution
- Developers, Quality Control and Download Volume in Open Source Software (OSS) Projects
More...