This week in KDE: the Plasma 6 feature freeze approaches

Planet KDE - Sat, 2023-11-25 00:02

At this point nearly all the planned features for Plasma 6 are done, and everyone’s focus has begun to shift to bug-fixing and polishing. People are reporting plenty of bugs (most of them fairly minor) and we’re fixing them as fast as we can! In addition to that, some larger and more notable changes went in too:

KDE 6 Mega-Release

(Includes all software to be released on the February 28th mega-release: Plasma 6, Frameworks 6, and apps from Gear 24.02)

General infoOpen issues: 162

Shutting down or restarting the machine while in the Plasma Wayland session now causes apps with unsaved changes to prompt the user to save them, rather than just quitting immediately and losing the changes. No new Wayland protocol ended up being necessary after all! This was one of the last three Wayland Showstoppers (David Redondo, link)

“Bounce Keys” now fully work in the Plasma Wayland session. This was the second of the last three Wayland Showstoppers! Now we’re just down to one, and it too is being worked on! (Nicolas Fella, link)

Partition Manager no longer lets you write entries to your fstab file that could prevent mounting a partition in the future by closing the partition editing dialog without changing the mountpoint, when the mountpoint was previously undefined (Arjen Hiemstra, link 1 and link 2)

Files and folders created in ~/Desktop but outside of Plasma itself should now always appear there immediately (Harald Sitter, link)

Changing your user picture in System Settings now results in the picture in Kickoff changing immediately, rather than only after Plasma was restarted (Akseli Lahtinen, link)

Fixed a visual glitch in Kate and other KTextEditor-based apps involving the completion popup (Waqar Ahmed, link)

Since the API for Plasma Widgets has changed in Plasma 6, widgets that you migrate from a Plasma 5 installation but aren’t compatible with Plasma 6 are now shown in a relatively user-friendly way, so at least you know what to do and don’t think that Plasma 6 is just totally broken (Marco Martin, link 1, link 2)

When you’ve downloaded an offline update, there’s now an option to reboot without applying it on the next boot-up. We’re considering adding this as an option when shutting down, too (Kai Uwe Broulik, link)

The “Battery and Brightness” widget has been split into two new widgets: “Brightness and Color” and “Power and Battery.” The former one integrates controls for Night Color, so in the end the total number of widgets in your System Tray isn’t increasing; they’re just better organized and relevant now! (Natalie Clarius, link):

KMail now gains support for the beautiful new frameless styling in the Plasma 6 Breeze theme (Carl Schwan, link):

The “Kill unresponsive window?” dialog now exists in the Plasma Wayland session and has received a visual overhaul to brings its UI into the 21st century (Kai Uwe Broulik, link):

Spectacle now opens on Meta+Shift+S, for the benefit of those whose keyboards don’t have a PrintScreen key (Noah Davis, link)

Welcome Center gains a dedicated page shown to people who use the beta version of Plasma (Oliver Beard, link):

The Samba sharing configuration wizard has been fully ported to Qt6 (Harald Sitter, link)

Other Bug-Related Information of Interest:

This section is short because all notable bugfixes are included in the mega-release! Expect this to be more and more true as we get closer to the final release, which, as a reminder, is planned for February 28, 2024.

…And Everything Else

This blog only covers the tip of the iceberg! If you’re hungry for more, check out https://planet.kde.org, where you can find more news from other KDE contributors.

How You Can Help

We’re hosting our Plasma 6 fundraiser right now and need your help! Thanks to you we’re past the 75% mark, but we’re not there yet! So if you like the work we’re doing, spreading the wealth is a great way to share the love.

If you’re a developer, work on Qt6/KF6/Plasma 6 issues! Which issues? These issues. Plasma 6 is usable for daily driving now, but still in need of bug-fixing and polishing to get it into a releasable state by February.

Otherwise, visit https://community.kde.org/Get_Involved to discover other ways to be part of a project that really matters. Each contributor makes a huge difference in KDE; you are not a number or a cog in a machine! You don’t have to already be a programmer, either. I wasn’t when I got started. Try it, you’ll like it! We don’t bite!

Categories: FLOSS Project Planets

Freezing in Style

Planet KDE - Fri, 2023-11-24 16:47

One of the the less apparent omissions in Plasma’s Wayland session compared to X was the lack of a prompt for terminating an unresponsive app. Of course, you should never see one because any decent app will just crash and quit rather than get stuck. Nevertheless, over the course of three evenings I spent way too much time making the “KWin Killer Helper” work on Wayland and while at it revamped its user interface entirely.

Isn’t it pretty? Of course this is staged using SIGSTOP – KWrite is awesome and never freezes like this!

It’s always fun to add your copyright from 2023 to a file with “© 2003” at the top. Times were surely different back then and what was considered good user interface design, too. The prompt for terminating an app is actually a separate executable since under X KWin isn’t really designed to open windows by itself and even under Wayland it’s tough. The helper might also use KAuth to end a privileged process. Being an external process, however, poses some challenges under Wayland because unlike X, it actually has some notion of security, and thus no random client can just mess with another one.

In order for the helper to attach itself to the frozen application window, it needs to use the XDG Foreign protocol. It’s designed for this particular use case of parenting oneself to a window in another process and used by for example the XDG Desktop Portal where a file dialog should appear as if it were coming from the application that requested it. The way it works is relatively simple: A client can ask the compositor to “export” a surface and receives a unique token for it. This token can then be passed to another application by any means (DBus, environment variable, doesn’t matter) which uses it to “import” that surface (to put it simply) and parent a window to it. But it’s a one-way street – you cannot pull a surface from another client, it has to send it to you.

KWin of course supports this protocol but its internals weren’t designed for it to launch an application by itself and export a surface to it, it would only export surfaces on the request of an actual Wayland client. It took me like three different attempts to find a good solution everyone was happy about: just create a base class that is referenced internally and create a sublass including the Wayland resources when actually needed. I’m still having a bit of a hard time wrapping my head around the ownership model employed by the classes generated by Qt Wayland Scanner. It’s a build tool which takes a Wayland XML Protocol description and generates C++ classes out it for use in your Qt application.

The original prompt, just running under Wayland. Quite a mouthful, isn’t it?

Next, I introduced a KillPrompt class that managed the life cycle of the process so its logic could easily be shared between X11 and Wayland windows. Originally, the X11Window class even used lowlevel ::kill(0) calls scattered throughout it to check whether the helper was still alive. Not something you would do today thanks to QProcess. The helper takes a bunch of command-line arguments, such as the window title, application name, process ID, etc. Of course, instead of sending a X Window ID I use the UUID of the aforementioned exported surface and instead of using the “Window Class” I use the more contemporary “desktop file name” (application ID) – that’s also why the original version above reads “org.kde.kwrite” instead of just “kwrite”.

Additionally, instead of setting the “X User Time” for soothing Kwin’s focus stealing prevention I generate an activation token to enable the dialog to properly get input focus. That’s using XDG Activation, another Wayland protocol for passing focus between windows, for instance clicking a link in a chat app should bring the browser to the front. The same principle applies: you ask the compositor for a token and then pass it to another application. However, the compositor may invalidate that token if you’re typing elsewhere in the meantime or the window wasn’t actually active at the time it requested the token and so on.

KWin also has a killPingTimeout setting (despite what some people say about KDE software being jammed with settings, this one isn’t in a spin box in System Settings) which defaults to 5 seconds. When clicking the close button, at half that time is when the window starts getting de-saturated as “not responding” and once the full timeout has passed the prompt asks whether to terminate the application. Now you might be asking yourself: Wait a minute, I have seen my windows getting all white and washed out even without trying to close them!

I will never figure out the wrapping logic for message boxes. Again, staged using SIGSTOP.

I congratulate you on using Wayland then! On X we only ping a window when asking it to close because we cannot really be certain that it will respond to our ping or even implement the interfaces that we need for that. On Wayland however, your typical floating desktop application will implement the XDG Shell protocol which handles things such as the window title, title bar buttons, and popup menus, and specifies a “ping” operation. In addition to pinging a window when closing it, KWin can also now safely ping it whenever it gets focus, and thus realize sooner that it has become frozen.

Coming back to the timeout, I found that it was actually hard-coded to 1 second in the Wayland case. Sometimes even my IDE loading a large project or the email client handling lots of incoming messages was enough to cause a brief white flash. How hard can it be to pass an int around? Well, KWin’s Wayland interfaces are to be kept separate from its main application logic (most importantly to ease unit testing them) so I couldn’t just access KWin’s options dict and read the setting out of it. Instead, I had to inject the value from the outside and refactor KWin’s startup a bit so that the XDG Shell integration is only initialized once the options have actually been read.

Concept: Should we use this icon with a badge approach in more places?

Finally, I figured, if I’m working on this piece of code anyway, I might as well overhaul its design which hasn’t changed much in two decades. You might recall I like doing that sort of thing. First of all, I used the desktop file name to look up the actual application name. Next, I significantly reduced the amount of text – by the time you finished reading the original dialog, the app might already be responding again ;) Technical information like the process ID and host name (in case of remote X applications) was then tucked away in the expandable “Details” section. I also tried hard to remove the redundant application name from the displayed title but some apps use an ndash, some an mdash, some just a regular hyphen, and some even put it before the window title.

I felt the need to more prominently link the dialog to the application that is about to be terminated and used its icon instead of the generic orange exclamation mark. The overlay you can see in the screenshots was achieved using KIconUtils in our KGuiAddons framework which – instead of rendering a pixmap once – creates a “live” QIconEngine that can re-render when the window gets moved between screens of different DPI. Finally, the helper received a proper .desktop file which is needed to give it a proper icon and by setting StartupFeedback to false suppress the bouncing cursor it would otherwise get. Originally, I wanted to restore the “bomb” icon it used to have but given the circumstances in the world I opted for that sad face we also use in DrKonqi when an application crashes.

Next time you’re wondering why it took so long to implement this tiny feature X and Y, think of me and all the plumbing that had to be done for a stupid freaking message box that you should never even see in the first place and the love and attention to detail that me and my fellow KDE developers put in our software.

Categories: FLOSS Project Planets

Andrew Cater: 20231124 - Mini-DebCamp ARM Cambridge day 2

Planet Debian - Fri, 2023-11-24 14:17

 Another really good day at ARM. Still lots of coffee and good food - supplemented by a cooked breakfast if you were early enough :)

Lots of small groups of people working earnestly in the main lecture theatre and a couple of meeting rooms and the soft seating area: various folk arriving  ready for tomorrow. Video team setting up in the afternoon and running up servers and cabling - all ready for a full schedule tomorrow and Sunday.

Many thanks to our sponsors - and especially the helpful staff at ARM who were helping us in and out, sorting out meeting rooms and generally coping with a Debian invasion. More people tomorrow for the weekend.

Categories: FLOSS Project Planets

Go Deh: Google Bard vs Microsoft Bing Chat: Vanity questions

Planet Python - Fri, 2023-11-24 12:23

 I ask two questions about my activities online and compare the outputs from the two AI tools.

Google Bard


create a rosettacode.org task in the style of user paddy3118

Sure, here is a Rosetta Code task in the style of user paddy3118:


Write a program that generates the Godeh series. The Godeh series is an infinite sequence of integers that is defined by the following recursive formula:

godeh(n) = { 1, if n = 1 godeh(n // 2), if n is even godeh(n // 2) + godeh(n // 2 + 1), if n is odd }


An integer n.


The first n terms of the Godeh series.


Input: 10 Output: 1 1 2 1 3 2 3 1 4 3


Pythondef godeh(n): if (n == 1): return 1 elif (n % 2 == 0): return godeh(n // 2) else: return godeh(n // 2) + godeh(n // 2 + 1) n = 10 for i in range(1, n + 1): print(godeh(i)) Use code with caution. Learn more

Paddy3118's style:

Paddy3118's programs are typically well-written and easy to understand. They are also often accompanied by clear and concise explanations of the code. In this task, Paddy3118 would likely provide a brief explanation of the Godeh series and the recursive formula used to define it. He would then provide a Python implementation of the program, along with an example of how to use it.

Here is an example of a task that Paddy3118 might write:


Write a program that generates the prime numbers up to a specified input.


An integer n.


A list of all the prime numbers up to n.


Input: 100 Output: [2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73, 79, 83, 89, 97]


Pythondef is_prime(n): if (n <= 1): return False if (n <= 3): return True if (n % 2 == 0 or n % 3 == 0): return False i = 5 while (i * i <= n): if (n % i == 0 or n % (i + 2) == 0): return False i += 6 return True def prime_numbers_up_to(n): primes = [] for i in range(2, n + 1): if (is_prime(i)): primes.append(i) return primes n = 100 primes = prime_numbers_up_to(n) print(primes) Use code with caution. Learn moreSources
  1. medium.com/edureka/python-program-prime-number-28001c0d54f2
  2. github.com/pratyksha/competitive-programming

 Microsoft Bing Chatcreate a rosettacode.org task in the style of user paddy3118CopyRewrite


I’m sorry, but I am not able to create a Rosetta Code task for you. However, I can provide you with some information about Paddy3118, a prolific contributor to Rosetta Code who has created many tasks in the past. Paddy3118 is a pseudonym for Donald McCarthy, who works with Electronic Design Automation tools creating design flows, filling in the gaps with novel tools, and generally helping to design and verify integrated circuits1He has created 392 new pages on Rosetta Code, of which 148 were not Talk: pages, from approximately 4523 edits1You can find more information about Paddy3118 and his contributions to Rosetta Code on his user page 1. I hope this information helps!

Learn more

1rosettacode.org2rosettacode.org3rosettacode.org4rosettacode.org5en.wikipedia.org8of30What is Rosetta Code?Can you tell me more about Electronic Design Automation tools?Who are some other prolific contributors to Rosetta Code?    

 Bard vs BingThe actual formatting does not copy-paste well in either case.
  • Bard attempts to write a task in my style, and gives some tips on my style.
  • Bing cannot create the task, but does extract some info on me.
  • Clearly a Bard win.
  • Flattery will get them nowhere! (But keep it up mind you).



Google Bard #2 what does paddy3118 do online?

Paddy3118 is an active participant in the online programming community, particularly in areas related to mathematics and algorithms. They have authored numerous programs and solutions to coding challenges, demonstrating expertise in various programming languages and problem-solving techniques. Paddy3118's contributions can be found across various platforms, including:

GitHub: Paddy3118 has a notable presence on GitHub, where they have authored over 50 programs covering a wide range of topics, including:

  • Mathematical algorithms: Implementing various mathematical concepts and algorithms, such as the Godeh series generator, Collatz conjecture tester, and prime number generator.

  • Problem-solving challenges: Tackling coding challenges from platforms like Codewars, showcasing their ability to solve complex problems efficiently.

  • Data manipulation and analysis: Demonstrating proficiency in handling and analyzing data through programs like the factorial calculator and Fibonacci sequence generator.

Their GitHub repositories often include detailed explanations of the code, making them valuable learning resources for other programmers.

Stack Overflow: Paddy3118 is an active contributor on Stack Overflow, a question-and-answer website for programmers. They have answered over 1,000 questions with an average score of 4.5 out of 5, indicating their deep understanding of programming concepts and their willingness to share their knowledge with others.

Codewars: Paddy3118 is a skilled competitor on Codewars, a coding challenge platform. They have solved over 1,000 coding challenges and currently rank in the top 1% of users. This achievement highlights their ability to think creatively and apply their programming expertise to solve complex problems.

GitHub Discussions: Paddy3118 actively engages in discussions on GitHub Discussions, a forum where users discuss technical topics. They provide helpful advice, share insights, and contribute to the overall knowledge base of the community.

Overall, Paddy3118 is a valuable member of the online programming community, recognized for their contributions to various platforms and their willingness to share their knowledge and expertise with others.

Microsoft Bing Chat #2what does paddy3118 do online?CopyRewrite

I’m sorry, but I could not find any information about what Paddy3118 does online. However, I can tell you that Paddy3118 is a pseudonym for Donald McCarthy, who works with Electronic Design Automation tools creating design flows, filling in the gaps with novel tools, and generally helping to design and verify integrated circuits1He has created 392 new pages on Rosetta Code, of which 148 were not Talk: pages, from approximately 4523 edits1You can find more information about Paddy3118 and his contributions to Rosetta Code on his user page 1. I hope this information helps!

Learn more

1you.dj2photopea.com3silver.urih.com10of30What is Rosetta Code?Can you tell me more about Electronic Design Automation tools?Who are some other prolific contributors to Rosetta Code?


Bard vs Bing #2
  • Bard has found me on numerous sites and attempted to summarise my contributions to them and comments that I like to help
  • Bing states up front that it could not find me but then quotes an extract based on my RosettaCode talk page.
  • None mention Python!?!
  • Another win for Bard.
At the moment, just using these easy-access AI tools, Bard seems to give better results.
It's still early days however!


Categories: FLOSS Project Planets

Jonathan Dowland: Dockerfile ARG footgun

Planet Debian - Fri, 2023-11-24 11:37

This week I stumbled across a footgun in the Dockerfile/Containerfile ARG instruction.

ARG is used to define a build-time variable, possibly with a default value embedded in the Dockerfile, which can be overridden at build-time (by passing --build-arg). The value of a variable FOO is interpolated into any following instructions that include the token $FOO.

This behaves a little similar to the existing instruction ENV, which, for RUN instructions at least, can also be interpolated, but can't (I don't think) be set at build time, and bleeds through to the resulting image metadata.

ENV has been around longer, and the documentation indicates that, when both are present, ENV takes precedence. This fits with my mental model of how things should work, but, what the documentation does not make clear is, the ENV doesn't need to have been defined in the same Dockerfile: environment variables inherited from the base image also override ARGs.

To me this is unexpected and far less sensible: in effect, if you are building a layered image and want to use ARG, you have to be fairly sure that your base image doesn't define an ENV of the same name, either now or in the future, unless you're happy for their value to take precedence.

In our case, we broke a downstream build process by defining a new environment variable USER in our image.

To defend against the unexpected, I'd recommend using somewhat unique ARG names: perhaps prefix something unusual and unlikely to be shadowed. Or don't use ARG at all, and push that kind of logic up the stack to a Dockerfile pre-processor like CeKit.

Categories: FLOSS Project Planets

Jonathan Dowland: bndcmpr

Planet Debian - Fri, 2023-11-24 11:27

Last year I put together a Halloween playlist and tried, where possible, to link to the tracks on Bandcamp. At the time, Bandcamp did not offer a Playlist feature; they've since added one, but it's limited to mobile only (and I've found, not a lot of use there either).

(I also provided a Spotify playlist. Not all of the tracks in the playlist were available on Spotify, or Bandcamp.)

Since then I discovered an independent service bndcmpr which lets you build and share playlists from tracks hosted on Bandcamp.

I'm not sure whether it will have longevity, but for now at least, I've ported my Halloween 2022 playlist over to bndcmpr. You can find it here: https://bndcmpr.co/cae6814a, or with any luck, embedded below (if you are reading this post on an aggregation site, or newsreader, it's less likely that this will appear):

I'm overdue cutting the next playlist, theme TBA. Stay tuned.

Categories: FLOSS Project Planets

GNU Guix: Write package definitions in a breeze

GNU Planet! - Fri, 2023-11-24 08:05

More than 28,000 packages are available in Guix today, not counting third-party channels. That’s a lot—the 5th largest GNU/Linux distro! But it’s nothing if the one package you care about is missing. So even you, dear reader, may one day find yourself defining a package for your beloved deployment tool. This post introduces a new tool poised to significantly lower the barrier to writing new packages.

Introducing Guix Packager

Defining packages for Guix is not all that hard but, as always, it’s much harder the first time you do it, especially when starting from a blank page and/or not being familiar with the programming environment of Guix. Guix Packager is a new web user interface to get you started—try it!. It arrived right in time as an aid to the packaging tutorial given last week at the Workshop on Reproducible Software Environments.

The interface aims to be intuitive: fill in forms on the left and it produces a correct, ready-to-use package definition on the right. Importantly, it helps you avoid pitfalls that trip up many newcomers:

  • When you add a dependency in one of the “Inputs” fields, it adds the right variable name in the generated code and imports the right package module.
  • Likewise, you can choose a license and be sure the license field will refer to the right variable representing that license.
  • You can turn tests on and off, and add configure flags. These translate to a valid arguments field of your package, letting you discover the likes of keyword arguments and G-expressions without having to first dive into the manual.

Pretty cool, no?


All the credit for this tool goes to co-worker and intrepid hacker Philippe Virouleau. A unique combination of paren aversion and web development superpowers—unique in the Guix community—led Philippe to develop the whole thing in a glimpse (says Ludovic!).

The purpose was to provide a single view to be able to edit a package recipe, therefore the application is a single-page application (SPA) written in using the UI library Philippe is most comfortable with: React, and MaterialUI for styling the components. It's built with TypeScript, and the library part actually defines all the types needed to manipulate Guix packages and their components (such as build systems or package sources). One of the more challenging parts was to be able to provide fast and helpful “search as you type” results over the 28k+ packages. It required a combination of MaterialUI's virtualized inputs, as well as caching the packages data in the browser's local storage, when possible (packaging metadata itself is fetched from https://guix.gnu.org/packages.json, a generic representation of the current package set).

While the feature set provides a great starting point, there are still a few things that may be worth implementing. For instance, only the GNU and CMake build systems are supported so far; it would make sense to include a few others (Python-related ones might be good candidates).

Running a local (development) version of the application can happen on top of Guix, since—obviously—it's been developed with the node version packaged in Guix, using the quite standard packages.json for JavaScript dependencies installed through npm. Contributions welcome!

Lowering the barrier to entry

This neat tool complements a set of steps we’ve taken over time to make packaging in Guix approachable. Indeed, while package definitions are actually code written in the Scheme language, the package “language” was designed from the get-go to be fully declarative—think JSON with parens instead of curly braces and semicolons. More recently we simplified the way package inputs are specified with an eye on making package definitions less intimidating.

The guix import command also exists to make it easier to simplify packaging: it can generate a package definition for anything available in other package repositories such as PyPI, CRAN, Crates.io, and so forth. If your preference goes to curly braces rather than parens, it can also convert a JSON package description to Scheme code. Once you have your first .scm file, guix build prints hints for common errors such missing module imports (those #:use-module stanzas). We also put effort into providing reference documentation, a video tutorial, and a tutorial for more complex packages.

Do share your experience with us and until then, happy packaging!


Thanks to Felix Lechner and Timothy Sample for providing feedback on an earlier draft of this post.

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, AArch64 and POWER9 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

Categories: FLOSS Project Planets

Doug Hellmann: sphinxcontrib-datatemplates 0.10.2

Planet Python - Fri, 2023-11-24 07:03
What’s new in 0.10.2? Packaging unify list of packaging dependencies modernize packaging files Documentation add .readthedocs.yaml to fix doc build Features declare support for python 3.12 add support for custom template bridge (contributions by Bizordec)
Categories: FLOSS Project Planets

Andy Wingo: tree-shaking, the horticulturally misguided algorithm

GNU Planet! - Fri, 2023-11-24 06:41

Let's talk about tree-shaking!

looking up from the trough

But first, I need to talk about WebAssembly's dirty secret: despite the hype, WebAssembly has had limited success on the web.

There is Photoshop, which does appear to be a real success. 5 years ago there was Figma, though they don't talk much about Wasm these days. There are quite a number of little NPM libraries that use Wasm under the hood, usually compiled from C++ or Rust. I think Blazor probably gets used for a few in-house corporate apps, though I could be fooled by their marketing.

But then you might recall the hyped demos of 3D first-person-shooter games with Unreal engine, but that was 5 years and one major release of Unreal ago: the current Unreal 5 does not support targetting WebAssembly.

Don't get me wrong, I think WebAssembly is great. It is having fine success in off-the-web environments, and I think it is going to be a key and growing part of the Web platform. I suspect, though, that we are only just now getting past the trough of dissolutionment.

It's worth reflecting a bit on the nature of web Wasm's successes and failures. Taking Photoshop as an example, I think we can say that Wasm does very well at bringing large C++ programs to the web. I know that it took quite some work, but I understand the end result to be essentially the same source code, just compiled for a different target.

Similarly for the JavaScript module case, Wasm finds success in getting legacy C++ code to the web, and as a way to write new web-targetting Rust code. These are often tasks that JavaScript doesn't do very well at, or which need a shared implementation between client and server deployments.

On the other hand, WebAssembly has not been a Web success for DOM-heavy apps. Nobody is talking about rewriting wordpress.com in Wasm, for example. Why is that? It may sound like a silly question to you: Wasm just isn't good at that stuff. But why? If you dig down a bit, I think it's that the programming models are just too different: the Web's primary programming model is JavaScript, a language with dynamic typing and managed memory, whereas WebAssembly 1.0 was about static typing and linear memory. Getting to the DOM from Wasm was a hassle that was overcome only by the most ardent of the true Wasm faithful.

Relatedly, Wasm has also not really been a success for languages that aren't, like, C or Rust. I am guessing that wordpress.com isn't written mostly in C++. One of the sticking points for this class of language. is that C#, for example, will want to ship with a garbage collector, and that it is annoying to have to do this. Check my article from March this year for more details.

Happily, this restriction is going away, as all browsers are going to ship support for reference types and garbage collection within the next months; Chrome and Firefox already ship Wasm GC, and Safari shouldn't be far behind thanks to the efforts from my colleague Asumu Takikawa. This is an extraordinarily exciting development that I think will kick off a whole 'nother Gartner hype cycle, as more languages start to update their toolchains to support WebAssembly.

if you don't like my peaches

Which brings us to the meat of today's note: web Wasm will win where compilers create compact code. If your language's compiler toolchain can manage to produce useful Wasm in a file that is less than a handful of over-the-wire kilobytes, you can win. If your compiler can't do that yet, you will have to instead rely on hype and captured audiences for adoption, which at best results in an unstable equilibrium until you figure out what's next.

In the JavaScript world, managing bloat and deliverable size is a huge industry. Bundlers like esbuild are a ubiquitous part of the toolchain, compiling down a set of JS modules to a single file that should include only those functions and data types that are used in a program, and additionally applying domain-specific size-squishing strategies such as minification (making monikers more minuscule).

Let's focus on tree-shaking. The visual metaphor is that you write a bunch of code, and you only need some of it for any given page. So you imagine a tree whose, um, branches are the modules that you use, and whose leaves are the individual definitions in the modules, and you then violently shake the tree, probably killing it and also annoying any nesting birds. The only thing that's left still attached is what is actually needed.

This isn't how trees work: holding the trunk doesn't give you information as to which branches are somehow necessary for the tree's mission. It also primes your mind to look for the wrong fixed point, removing unneeded code instead of keeping only the necessary code.

But, tree-shaking is an evocative name, and so despite its horticultural and algorithmic inaccuracies, we will stick to it.

The thing is that maximal tree-shaking for languages with a thicker run-time has not been a huge priority. Consider Go: according to the golang wiki, the most trivial program compiled to WebAssembly from Go is 2 megabytes, and adding imports can make this go to 10 megabytes or more. Or look at Pyodide, the Python WebAssembly port: the REPL example downloads about 20 megabytes of data. These are fine sizes for technology demos or, in the limit, very rich applications, but they aren't winners for web development.

shake a different tree

To be fair, both the built-in Wasm support for Go and the Pyodide port of Python both derive from the upstream toolchains, where producing small binaries is nice but not necessary: on a server, who cares how big the app is? And indeed when targetting smaller devices, we tend to see alternate implementations of the toolchain, for example MicroPython or TinyGo. TinyGo has a Wasm back-end that can apparently go down to less than a kilobyte, even!

These alternate toolchains often come with some restrictions or peculiarities, and although we can consider this to be an evil of sorts, it is to be expected that the target platform exhibits some co-design feedback on the language. In particular, running in the sea of the DOM is sufficiently weird that a Wasm-targetting Python program will necessarily be different than a "native" Python program. Still, I think as toolchain authors we aim to provide the same language, albeit possibly with a different implementation of the standard library. I am sure that the ClojureScript developers would prefer to remove their page documenting the differences with Clojure if they could, and perhaps if Wasm becomes a viable target for Clojurescript, they will.

on the algorithm

To recap: now that it supports GC, Wasm could be a winner for web development in Python and other languages. You would need a different toolchain and an effective tree-shaking algorithm, so that user experience does not degrade. So let's talk about tree shaking!

I work on the Hoot Scheme compiler, which targets Wasm with GC. We manage to get down to 70 kB or so right now, in the minimal "main" compilation unit, and are aiming for lower; auxiliary compilation units that import run-time facilities (the current exception handler and so on) from the main module can be sub-kilobyte. Getting here has been tricky though, and I think it would be even trickier for Python.

Some background: like Whiffle, the Hoot compiler prepends a prelude onto user code. Tree-shakind happens in a number of places:

Generally speaking, procedure definitions (functions / closures) are the easy part: you just include only those functions that are referenced by the code. In a language like Scheme, this gets you a long way.

However there are three immediate challenges. One is that the evaluation model for the definitions in the prelude is letrec*: the scope is recursive but ordered. Binding values can call or refer to previously defined values, or capture values defined later. If evaluating the value of a binding requires referring to a value only defined later, then that's an error. Again, for procedures this is trivially OK, but as soon as you have non-procedure definitions, sometimes the compiler won't be able to prove this nice "only refers to earlier bindings" property. In that case the fixing letrec (reloaded) algorithm will end up residualizing bindings that are set!, which of all the tree-shaking passes above require the delicate DCE pass to remove them.

Worse, some of those non-procedure definitions are record types, which have vtables that define how to print a record, how to check if a value is an instance of this record, and so on. These vtable callbacks can end up keeping a lot more code alive even if they are never used. We'll get back to this later.

Similarly, say you print a string via display. Well now not only are you bringing in the whole buffered I/O facility, but you are also calling a highly polymorphic function: display can print anything. There's a case for bitvectors, so you pull in code for bitvectors. There's a case for pairs, so you pull in that code too. And so on.

One solution is to instead call write-string, which only writes strings and not general data. You'll still get the generic buffered I/O facility (ports), though, even if your program only uses one kind of port.

This brings me to my next point, which is that optimal tree-shaking is a flow analysis problem. Consider display: if we know that a program will never have bitvectors, then any code in display that works on bitvectors is dead and we can fold the branches that guard it. But to know this, we have to know what kind of arguments display is called with, and for that we need higher-level flow analysis.

The problem is exacerbated for Python in a few ways. One, because object-oriented dispatch is higher-order programming. How do you know what foo.bar actually means? Depends on foo, which means you have to thread around representations of what foo might be everywhere and to everywhere's caller and everywhere's caller's caller and so on.

Secondly, lookup in Python is generally more dynamic than in Scheme: you have __getattr__ methods (is that it?; been a while since I've done Python) everywhere and users might indeed use them. Maybe this is not so bad in practice and flow analysis can exclude this kind of dynamic lookup.

Finally, and perhaps relatedly, the object of tree-shaking in Python is a mess of modules, rather than a big term with lexical bindings. This is like JavaScript, but without the established ecosystem of tree-shaking bundlers; Python has its work cut out for some years to go.

in short

With GC, Wasm makes it thinkable to do DOM programming in languages other than JavaScript. It will only be feasible for mass use, though, if the resulting Wasm modules are small, and that means significant investment on each language's toolchain. Often this will take the form of alternate toolchains that incorporate experimental tree-shaking algorithms, and whose alternate standard libraries facilitate the tree-shaker.

Welp, I'm off to lunch. Happy wassembling, comrades!

Categories: FLOSS Project Planets

CXX-Qt 0.6 Release

Planet KDE - Fri, 2023-11-24 06:30

We just released CXX-Qt version 0.6!

CXX-Qt is a set of Rust crates for creating bidirectional Rust ⇄ C++ bindings with Qt. It can be used to integrate Rust into C++ applications using CMake or build Rust applications with Cargo. CXX-Qt provides tools for implementing QObject subclasses in Rust that can be used from C++, QML, and JavaScript.

For 0.6, we’re excited to announce that we’re on the road to stabilization! We’ve done a (hopefully final) iteration of our API and are now happier than ever to release it.

The new API is now truly in the spirit of CXX. All implementation has moved out of the #[cxx_qt::bridge] and is now simply outside the bridge, like it is for CXX as well. To get an overview of the new API, check out our the planned 1.0 API, which is available starting with this release.

Check out the new release through the usual channels:

Some of the more extensive developer-facing changes are listed below.

New CXX-style API extern “C++Qt”

Like extern "C++" in CXX, the extern "C++Qt" block now only uses a declaration-like syntax to expose functionality to C++/Qt. This is an extension to CXX’s extern "C++" blocks, which allow defining methods as #[qsignal]. These signals can then be emitted from Rust, as well as connected to.

extern “RustQt”

Like extern "Rust" in CXX, the extern "RustQt" block now only uses a declaration-like syntax to expose functionality to C++/Qt. Most existing features have moved to an attribute-syntax, which should look very familiar to existing Qt developers.

The new API includes:

  • #[qobject] is now on a type alias to distinguish between QObject and Rust struct clearer
  • #[qproperty(...)] is now just an attribute on the #[qobject]
  • #[inherit],#[qsignal] & #[qinvokable] have now moved to attributes of external function declarations. They can also now be freely mixed
  • #[cxx_override], #[cxx_final], #[cxx_virtual] are now independent attributes rather than imbedded in #[qinvokable]
Support for Qt enums as shared types

Like shared types in CXX, enums for Qt can be declared using a shared enum in the bridge. These can be defined using the #[qenum] attribute and qnamespace!(...) macro.

Implementation is outside the bridge

Like CXX the implementations on types are now defined outside of the bridge rather than inside. This allows you to choose the prefix for QObjects, as the hard-coded qobject:: prefix is now simply the name of the #[cxx_qt::bridge] module.

Another great new feature in this release: Connecting to Signals via Rust is now possible! Simply use the on_[SIGNAL-NAME]/connect_[SIGNAL-NAME] methods generated for each signal. Closures are supported when connecting to signals as well.

QML modules can now be built

The build system has been improved so that QML modules can now be output by CXX-Qt. This allows for using the declarative syntax of #[qml_element] and #[qml_singleton] on a #[qobject], whereas before Rust QObject types needed to be registered in C++ with the QML engine.

use cxx_qt_build::{CxxQtBuilder, QmlModule}; fn main() { CxxQtBuilder::new() .qml_module(QmlModule { uri: "com.kdab.cxx_qt.demo", rust_files: &["src/cxx_qt_bridge.rs"], qml_files: &["qml/main.qml"], ..Default::default() }) .build(); }

This should also allow for tooling to inspect QML modules that were generated from Rust in the future.

New traits for shaping code generation and interacting with features

Most features of the CXX-Qt code generation are now expressed through Rust traits and can be enabled/disabled or otherwise configured using a syntax similar to CXX shim trait impls.

New features

  • Custom constructors through cxx_qt::Constructor – this drops the requirement for Default on the Rust struct.
  • Easy default-constructor implementation through cxx_qt::Initialize.
  • cxx_qt::CxxQtType trait for reflection and less “magic” methods.

Existing features that can now be enabled/disabled:

  • Threading through the cxx_qt::Threading trait.
  • Locking through the cxx_qt::Locking trait.
Thanks to all of our contributors that helped us with this release
  • Be Wilson
  • Laurent Montel
  • Olivier Le Doeuff
  • Jimmy van Hest
  • Michael (aka @mmMike)
  • Cyril Jacquet
  • Magnus Groß
  • jimmyvanhest

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post CXX-Qt 0.6 Release appeared first on KDAB.

Categories: FLOSS Project Planets

Web Review, Week 2023-47

Planet KDE - Fri, 2023-11-24 05:47

Let’s go for my web review for the week 2023-47.

What OpenAI shares with Scientology

Tags: tech, ai, criticism

Ironically, I think this is one of the best analysis I’ve seen regarding OpenAI and the recent turmoil around of one of its founders.


Oops! We Automated Bullshit. | Department of Computer Science and Technology

Tags: tech, ai, machine-learning, gpt, criticism

Excellent piece about everything which is wrong in the current industrialization moment of generative AI.


Pluralistic: Tesla’s Dieselgate (28 July 2023)

Tags: tech, automotive, DRM

Don’t you love DRMs in cars? Especially when the business people at the helm are pathological liars?


Youtube has started to artificially slow down video load times if you use Firefox

Tags: tech, google, attention-economy

I guess it’s a strategy to move more people over to Chrome. Shameful.


Cryptographers Solve Decades-Old Privacy Problem - Nautilus

Tags: tech, cryptography, privacy

Cool results, let’s see what the future brings. This could be exciting.


Streamlined interface for generating images with AI in Krita

Tags: tech, graphics, ai, machine-learning

I guess this will end up being a popular plugin for Krita.


Audio Samples from StyleTTS 2

Tags: tech, ai, machine-learning, speech

The newer text to speech models are really good now.


Making an E-Paper Picture Frame

Tags: tech, raspberry-pi, e-ink

This is a cool project, not too expensive as well.


Email obfuscation: What still works in 2023?

Tags: tech, web, email, spam

Good list of techniques. Some of them aren’t fully evaluated yet. Definitely worth considering.


Ethernet is Still Going Strong After 50 Years - IEEE Spectrum

Tags: tech, networking, history

A bit of history behind what’s probably the most widespread local network technology.


RFC 9498: The GNU Name System

Tags: tech, dns, privacy, censorship

Interesting a new name system being standardized. It’s supposed to protect privacy and be censorship resistant. We’ll see how it gets adopted.


How many floating-point numbers are in the interval [0,1]? – Daniel Lemire’s blog

Tags: tech, floats

Floating-point numbers are really a complicated species. This is an interesting deep dive in some of their representation.


Herbie: Automatically Improving Floating Point Accuracy

Tags: tech, floats, tools

Interesting tool. Hopefully will help us manipulate floating point expressions better.


A close encounter with false sharing | More Stina Blog!

Tags: tech, multithreading, performance

Good reminder that false sharing is a real thing. It’s easier to encounter than you think when you start to dabble into multi-threading.


GWP-ASan: Sampling-Based Detection of Memory-Safety Bugs in Production - 2311.09394.pdf

Tags: tech, memory, safety, security

Nice approach to also hunt for memory safety issues while software is in production.


Retries – An interactive study of common retry methods – Encore Blog

Tags: tech, networking, programming

Nice little article with simulations demonstrating why you want exponential backoff and how jitter is an extra layer of protection for the server.


Create RSS Feeds in Java using ROME

Tags: tech, java, rss

Yes, I have a thing for bringing RSS back. This time it’s a library enabling the feature for Java projects. Looks easy to use.


Explicit Resource Management: Exploring JavaScript’s and TypeScript’s new feature | iliazeus

Tags: tech, javascript, resources

Looks like RAII is finally making its way in Javascript. This looks like a good thing, it’s still rough around the edges though.


The Changing “Guarantees” Given by Python’s Global Interpreter Lock · Stefan-Marr.de

Tags: tech, python, multithreading

Won’t be easy to get rid of the GIL in the Python ecosystem. There are notable differences of behavior between implementations and even versions of the same implementation… Lots of user code will unwillingly depend on a specific set of guarantees.


It’s Time For A Change: datetime.utcnow() Is Now Deprecated - miguelgrinberg.com

Tags: tech, python, api

Since quite a lot of Python code will be impacted by this, better get ready.


Exploring a Postgres query plan | notes.eatonphil.com

Tags: tech, postgresql, databases

Ever wondered how to programmatically introspect query plans? This is a long article but good starting point on how it’s represented under the hood.


garnix | Contextual CLIs

Tags: tech, command-line

Good food for thought to design CLI interfaces.


An Interactive Guide to CSS Grid

Tags: tech, frontend, browser, css

Nice guide, the interactive parts definitely help. Good way to improve CSS Grid use. It’s much more powerful than I suspected.


How large pull requests slow down development

Tags: tech, git, version-control, complexity, risk, quality

Interesting statistics, this show how important it is to have well structured and focused change sets as much as possible.


Software Development and Postmortems

Tags: tech, project-management, failure, postmortem

Very good piece. Explains why postmortems are important. It also explains how to prepare your organization to conduct them and how to do them properly. This is important since a lot of pressure will happen in case of a failure.


75% of Software Engineers Faced Retaliation Last Time They Reported Wrongdoing - Engprax

Tags: tech, politics, law, work

This is really bad… Now, this is investigation is UK centric though. I wonder how other countries would fare.


Performance Is Contextual - Jacob Kaplan-Moss

Tags: management, productivity

Definitely this, the context matters a lot. Sometimes I’ve seen people too quick to blame the skillset of underperforming colleagues. But the same person in a different context could probably do much better.


14 Signs of a GOOD Manager. Navigating leadership excellence.

Tags: tech, management

This is a good list. I guess some of it feels obvious… at the same time it’s indeed something you don’t see every day. More awareness from managers is needed.


Bye for now!

Categories: FLOSS Project Planets

CTI Digital: Enhancing Drupal Content Editing with Same Page Preview Module

Planet Drupal - Fri, 2023-11-24 04:36

As I embarked on a recent journey to enhance the usability of Drupal from the perspective of both site owners and editors, I stumbled upon what could be a game changer for content editors – the "Same Page Preview" module.

Categories: FLOSS Project Planets

Golems GABB: Building a Mobile App with Drupal as a Backend

Planet Drupal - Fri, 2023-11-24 03:00
Building a Mobile App with Drupal as a Backend Editor Fri, 11/24/2023 - 10:00

Do you know that almost 90% of developers who use Drupal as a backend for their mobile apps have their stash, lucky coding socks? This is interesting because developers believe it allows them to work more efficiently.
We may have exaggerated about 90 percent, but building a mobile app development with Drupal as a backend is perfect. There are many reasons for this because Drupal as a tool has all the necessary features and capabilities to unleash the potential of the application fully and, of course, emphasize the skill of its creator. But how does it work?

Categories: FLOSS Project Planets

LN Webworks: Drupal 11's Arrival in 2024 – Is Your Website Prepared?

Planet Drupal - Fri, 2023-11-24 01:42

Drupal 11 is on the horizon, set to make its debut in either August or December 2024. That may seem like a distant future, but in the ever-evolving world of web development, a year and a half can go by in the blink of an eye. So, the big question we're here to explore is this: Is your website ready for the release of Drupal 11?

Let’s take a closer look at why it's crucial not to procrastinate when it comes to Drupal 11 preparation. We'll also discuss the importance of getting a head start and why the process should be more of a steady, consistent effort rather than a last-minute scramble.

An Overview of The Why and How of Drupal 11 

Drupal 11 is generating quite the buzz, and for good reason. Anticipation is building as the Drupal community delves into discussions about Drupal 11, even before Drupal 10 has made its grand exit. This unprecedented level of excitement around Drupal 11 makes it a release like no other.

Categories: FLOSS Project Planets

Quansight Labs Blog: Unlocking C-level performance in pandas.DataFrame.apply with Numba

Planet Python - Thu, 2023-11-23 19:00
A quick overview of the new Numba engine in DataFrame.apply
Categories: FLOSS Project Planets

Paolo Melchiorre: Database generated columns⁽²⁾: Django &amp; PostgreSQL

Planet Python - Thu, 2023-11-23 18:00

An introduction to database generated columns, using PostgreSQL and the new GeneratedField added in Django 5.0.

Categories: FLOSS Project Planets

parallel @ Savannah: GNU Parallel 20231122 ('Grindavík') released

GNU Planet! - Thu, 2023-11-23 17:50

GNU Parallel 20231122 ('Grindavík') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  Got around to using GNU parallel for the first time from a suggestion by @jdwasmuth ... now I'm wishing I started using this years ago
    -- Stefan Gavriliuc @GavriliucStefan@twitter

New in this release:

  • -a file1 -a +file2 will link file2 to file1 similar to ::::+
  • --bar shows total time when all jobs are done.
  • Bug fixes and man page updates.

News about GNU Parallel:

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:


GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

Categories: FLOSS Project Planets

Daniel Roy Greenfeld: Three Years at Kraken Tech

Planet Python - Thu, 2023-11-23 14:37

A summary of the past year as I finish my third year working for Kraken Tech, an Octopus Energy Group subsidiary.

Note: As I write this I'm recovering from a sprained shoulder I received in a bicycle accident that makes it hard to type. I'm mostly using voice-to-text, so the text might not be in my normal "writing voice".

I changed roles

I transitioned from leading the tech team for Kraken Tech USA to being an individual contributor for the global team. I do this periodically, cycling between writing code to leading teams, and back again.

It is never easy to transition between roles but this time in particular it was particularly hard. The US Kraken Tech team is a group of talented, diverse, and passionate people. I'm so honored I had the chance to work with them. I'm constantly amazed by what they have accomplished and what they continue to do.

In my new role I'm on the "Developer Foundations" team, exploring ways to improve the coding experience at the company. I've enjoyed writing code full-time again, there's a clarity of purpose that's been fun to embrace. Mostly that's been working in Python with a bit of Django, with short detours into Rust and Golang.

One more thing: For the moment I still have the old management-sounding title, but expect to get something more accurate in the next few months.

I blogged more

Due to that I'm coding full-time again, I more than doubled the number of posts I wrote in 2022.

I worked in the London, UK Office

Audrey and Uma joined me on a move to London, which I documented here.

It has been a delight being able to work in-person with my peers, or at least close in timezone. I'm usually in the office about three out of every five days. I would make it five-out-of-five but I do better with deep thinking outside normal office commotion.

I attended large in-person tech events

After a break of four years I managed to attend the following in chronological order, with transit mode specified:

  • Django London (walk or train)
  • PyCon Italy (airplane)
  • PyCon UK (train)
  • Django Paris (train)

I very much enjoyed meeting old friends and making new ones. Plus, exploring the cities of London, Cardiff, and Paris were adventures of their own.

Helped scale up Kraken Tech to address climate change

As an individual concerned about global climate change I do my part:

  • I compost about 80% of my family's food waste
  • Do our best to recyle all the family's cardboard and paper products
  • We live in a tiny flat with an electric stove
  • Am experimenting with using a portable electric heater instead of the methane gas boiler
  • Have reduced unnecessary air travel as much as possible (which is hard - I love airplanes and airports)

That's great, but I have managed to scale that up tens of thousands of times over by working for Kraken Tech.

As one employee of nine hundred and thirty seven (937) at Kraken Tech, I take payment from a collective of companies whose products are all designed with the purpose of saving the planet. Kraken Tech in particular is responsible for about 55 million people getting onto renewable energy or using non-renewable resources more efficiently.

Let's math that out, using conservative numbers:

58,697.97 = 55,000,000 / 937

Summary: Each employee of Kraken Tech is roughly responsible for 58,698 (rounding up) people contributing significantly less to global climate change.

So if you are in tech in any capacity (coding, management, etc), consider using your skills to help us in our mission of making a healthier planet.

Come and help me save the planet

I'll be upfront that if you want to make the biggest bucks possible in tech, travel a lot, and fly in first place seats, go work for a Big Tech Company or a fintech. Personally, I would rather work toward making the planet better for our descendants. That to me is much more important than helping a billionaire score another billion.

I invite anyone who is reading this to join me on my mission of saving the planet. Our careers page is here.

Please understand that Kraken Tech only hires in countries in which we have a legal entity. So if a role isn't listed for the country in which you can legally work, we probably can't hire you.

If for whatever reason you can't get a job working with me, apply to work at somewhere else into decarbonization. Let me know where you go and I'll celebrate your joining our shared mission.

Categories: FLOSS Project Planets

Andrew Cater: 20231123 - UEFI install on a Raspberry Pi 4 - step by step instructions to a modified d-i

Planet Debian - Thu, 2023-11-23 11:39

 Andy (RattusRattus) and I have been formalising instructions for using Pete Batard's version of Tianocore (and therefore UEFI booting) for the Raspberry Pi 4 together with a Debian arm64 netinst to make a modified Debian installer on a USB stick which "just works" for a Raspberry Pi 4.

Thanks also to Steve McIntyre for initial notes that got this working for us and also to Emmanuel Rocca for putting up some useful instructions for copying.


Plug in a USB stick - use dmesg or your favourite method to see how it is identified.

Make a couple of mount points under /mnt - /mnt/data and /mnt/cdrom

1. Grab a USB stick, Partition using MBR. Make a single VFAT
   partition, type 0xEF (i.e. EFI System Partition)

For a USB stick (identified as sdX) below:

$ sudo parted --script /dev/sdX mklabel msdos $ sudo parted --script /dev/sdX mkpart primary fat32 0% 100% $ sudo mkfs.vfat /dev/sdX1 $ sudo mount /dev/sdX1 /mnt/data/

Download an arm64 netinst.iso


2. Copy the complete contents of partition *1* from a Debian arm64
   installer image into the filesystem (partition 1 is the installer
   stuff itself) on the USB stick, in /

$ sudo kpartx -v -a debian-12.2.0-arm64-netinst.iso # Mount the first partition on the ISO and copy its contents to the stick $ sudo mount /dev/mapper/loop0p1 /mnt/cdrom/ $ sudo rsync -av /mnt/cdrom/ /mnt/data/ $ sudo umount /mnt/cdrom

3. Copy the complete contents of partition *2* from that Debian arm64
  installer image into that filesystem (partition 2 is the ESP) on
   the USB stick, in /

# Same story with the second partition on the ISO $ sudo mount /dev/mapper/loop0p2 /mnt/cdrom/ $ sudo rsync -av /mnt/cdrom/ /mnt/data/ $ sudo umount /mnt/cdrom
$ sudo kpartx -d debian-testing-amd64-netinst.iso $ sudo umount /mnt/data

4. Grab the rpi edk2 build from https://github.com/pftf/RPi4/releases
   (I used 1.35) and extract it. I copied the files there into *2*
   places for now on the USB stick:

   /      (so the Pi will boot using it)
   /rpi4  (so we can find the files again later)

5. Add the preseed.cfg file (attached) into *both* of the two initrd
   files on the USB stick

   - /install.a64/initrd.gz and
   - /install.a64/gtk/initrd.gz

   cpio is an awful tool to use :-(. In each case:

   $ cp /path/to/initrd.gz .
   $ gunzip initrd.gz
   $ echo preseed.cfg | cpio -H newc -o -A -F initrd

$ gzip -9v initrd


$ cp initrd.gz /path/to/initrd.gz

   If you look at the preseed file, it will do a few things:

   - Use an early_command to unmount /media (to work around Debian bug

   - Register a late_command call for /cdrom/finish-rpi (the next
     file - see below) to run at the end of the installation.

   - Force grub installation also to the EFI removable media path,
     needed as the rpi doesn't store EFI boot variables.

   - Stop the installer asking for firmware from removable media (as
     the rpi4 will ask for broadcom bluetooth fw that we can't
     ship. Can be ignored safely.)

6. Copy the finish-rpi script (attached) into / on the USB stick. It
   will be run at the end of the installation, triggered via the
   preseed. It does a couple of things:

   - Copy the edk2 firmware files into the ESP on the system that's
     just been installer

   - Remove shim-signed from the installed systems, as there's a bug
     that causes it to fail on rpi4. I need to dig into this to see
     what the issue is.

That's it! Run the installer as normal, all should Just Work (TM).


# The preseed file itself causes a problem - the installer medium is
# left mounted on /medis so things break in cdrom-detect. Let's see if
# we can fix that!
d-i preseed/early_command string umount /media || true

# Run our command to do rpi setup before reboot
d-i preseed/late_command string /cdrom/finish-rpi

# Force grub installation to the RM path
grub-efi-arm64  grub2/force_efi_extra_removable boolean true

# Don't prompt for missing firmware from removable media,
# e.g. broadcom bluetooth on the rpi.
d-i hw-detect/load_firmware boolean false



set -x

grep -q -a RPI4 /sys/firmware/acpi/tables/CSRT
if [ $? -ne 0 ]; then
    echo "Not running on a Pi 4, exit!"
    exit 0

# Copy the rpi4 firmware binaries onto the installed system.
# Assumes the installer media is mounted on /cdrom.
cp -vr /cdrom/rpi4/. /target/boot/efi/.

# shim-signed doesn't seem happy on rpi4, so remove it
mount --bind /sys /target/sys
mount --bind /proc /target/proc
mount --bind /dev /target/dev

in-target apt-get remove --purge --autoremove -y shim-signed

Categories: FLOSS Project Planets

Andrew Cater: Arm Cambridge - mini-Debcamp 23 November 2023

Planet Debian - Thu, 2023-11-23 09:32

 At Arm for two days before the mini-Debconf this weekend.

First time at Arm for a few years: huge new buildings, shiny lecture theatre.

Arm have made us very welcome. A superb buffet lunch and unlimited coffee plus soft drinks - I think they know what Debian folk are like.
Not enough power blocks laid out at the beginning - only one per table - but we soon fixed that 😀

The room is full of Debian folk: some I know, some new faces. Reminiscing about meeting some of them from 25 years ago - and the chance to thank people for help over a long time.

Andy (RattusRattus) and I have been working out the bugs on an install script using UEFI for a Raspberry Pi 4. More on that in the next post, maybe.

As ever, it's the sort of place where "I can't get into the wiki" is sorted by walking three metres across the room or where an "I can't find where to get X for Raspberry Pi" can be solved by asking the person who builds Raspbian. "Did you try and sign up to the Debian wiki last week - you didn't follow the instructions to mail wiki@ - I _know_ you didn't because I didn't see the mail ... "

My kind of place and my kind of people, as ever.

Thanks again to Arm who are one of our primary sponsors for this mini-Debconf.

Categories: FLOSS Project Planets