FLOSS Project Planets

tl;dr

Starting with curl 8.0.0-2, you can now use HTTP3.

curl --http3-only https://example.com

Or, if you would like to try it out in a container:

podman run debian:unstable apt install --update -y curl && curl --http3-only https://example.com

(in case you haven't noticed, apt now has the --update option for the upgrade and install commands, although not available on stable yet)

Availability

• Debian unstable - Since 2024-07-02
• Debian testing - Coming up between the second and third week of July 2024 (estimate if no issues are spotted).
• Debian 12/bookworm backports - As soon as the package gets to Debian testing, I'll upload it to bookworm.
• Debian 12/bookworm - Due to the mechanisms we have in place to make sure Debian stable is in fact stable, we will never be able to ship this in the regular repository. Users can make use of the backports repositories instead.
• Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.

The challenge

HTTP3 is fresh new, well... not really, but at least fresh enough that I'm not aware of any other Linux distribution supporting it on curl, the reason is likely two-fold:

1. OpenSSL is not there yet

   OpenSSL still doesn't have proper HTTP3 support, and given that OpenSSL is so widely used, almost every curl distributor/packager will build curl with it and thus changing the TLS backend to something else is risky.

   Unfortunately, proper support for the OpenSSL libcurl is unlikely to come anytime before the end of this year, the OpenSSL performance is not good enough yet as of version 3.3.

   Daniel Stenberg has written about the state of this multiple times, most recently at HTTP/3 in curl mid 2024, if you're interested, I suggest reading through his other posts as well.

   Some might have noticed that nginx does support HTTP3 through OpenSSL, although when you look closely, it's not exactly perfect:

   An SSL library that provides QUIC support is recommended to build nginx, such as BoringSSL, LibreSSL, or QuicTLS. Otherwise, the OpenSSL compatibility layer will be used that does not support early data.

   As you can see, they don't recommend using OpenSSL, and when doing so, you don't get complete support.

2. HTTP3 support for GnuTLS/nghttp3/ngtcp2 is recent

   The non-experimental support arrived back in October 2023, and so that's when I started seriously planning for this.

   curl has been working on HTTP3 support for years, and so it did support other TLS backends before that, but out of them, the one most feasible for a distribution to ship would be GnuTLS, which gets HTTP3 support through ngctp2 and nghttp3.

How it was done

The Debian curl package has historically shipped at least two variants of libcurl, an OpenSSL and a GnuTLS one.

The OpenSSL libcurl can't support HTTP3 for the reasons explained above, but the GnuTLS libcurl can (with ngtcp2 and nghtp3).

Debian packages can choose which version of libcurl to link against (without having to modify any upstream source code). Debian's "git" package being a famous example of a package that links against the GnuTLS libcurl.

Enabling HTTP3 on curl was done in three steps:

1. Make sure all required dependencies fulfill the minimum requirements.
2. Enable HTTP3 for GnuTLS libcurl.
3. Change the libcurl used by the curl CLI, from OpenSSL to GnuTLS.

curl's HTTP3 support requires a somewhat recent version of nghttp3 and updating that required a transition (due to the SONAME bump), while we've also had months of freeze for transitions due to the time_t transition.

After the dependencies were in place, enabling HTTP3 for the GnuTLS libcurl was straightforward.

Then, for the last part, we had to switch the TLS backend used by the curl CLI. Doing the swap is also quite easy on the packaging level, but we have to consider the chances of this change breaking our users' environments.

Ensuring there are no breakages

The first thing to consider regarding breakages is that this change is not going to be pushed directly to the current Debian stable releases, it will be present in the next stable release (13/trixie) but the current one will stick to the version that's already shipped.

Secondly, we have to consider the risk of losing the ability to use certain parameters from the curl CLI which could be limited to the OpenSSL backend. During curl-up 2024, the curl developers pointed out the existence of a page that lists the TLS related options and the backends they work with.

Analysing that page, ignoring all of the options that are suffixed with "BLOB" (only pertinent to the library, not the CLI), the only one left which is attention worthy is CURLOPT_ECH.

This experimental feature requires a special build of OpenSSL, as ECH is not yet supported in OpenSSL releases. In contrast ECH is supported by the latest BoringSSL and wolfSSL releases.

As it turns out, Encrypted Client Hello is experimental and it's not supported by the vanilla OpenSSL.

This was enough of an investigation for me to go ahead with the change. Noting that even in the worst case scenario (we find a horrible regression), we can rollback without having affected a single stable release.

Now that the package is on Debian unstable, the CI tests (autopkgtest) of every package that depends on curl is currently running, the results are compared against the migration-reference (in this case, the curl CLI with OpenSSL, before the change).

If everything goes right, curl with HTTP3 support will migrate to Debian testing in around 5 days. If we spot any issues, we'll have to solve them first and it's going to be hard to predict how long it takes, although it's fair to expect less than a month.

Feedback

Feel free to join the Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org
We have historically spoken Portuguese in the room but we'll switch to English in case anyone joins.

Acknowledgements

It took us a bit longer than expected to be able to enable HTTP3, nonetheless it's still early enough to be excited about.

A lot of people were crucial to make this happen.

I should recognize in the first place, obviously, the curl developers and the developers of the supporting libraries: GnuTLS, nghttp3, ngtcp2. Participating in the curl-up 2024 conference helped me get motivated to push this through, besides becoming aware of the right documentation to research for impact.

On the Debian side, Sakirnth Nagarasa <sakirnth> was responsible for updating and taking care of the transition for nghttp3 and ngtcp2.

Also on the Debian side, I've got loads of help and support from the co-maintainers of the curl package: Sergio Durigan Junior <sergiodj> and Carlos Henrique Lima Melara <charles>.

Here comes the 2nd article of the 5-episode blog post series written by Guido Berhörster, member of staff at my company Fre(i)e Software GmbH.

Enjoy also this read on Guido's work on Polis,
Mike

Table of Contents of the Blog Post Series

1. Introduction
2. Initial evaluation and adaptation (this article)
3. Issues extending Polis and adjusting our goals
4. Creating (a) new frontend(s) for Polis
5. Current status and roadmap

Polis - Initial evaluation and adaptation

The Polis code base consists of a number of components, the administration and participation interfaces, a common web backend, and a statistics processing server. Both frontends and the backend are written in a mixture of JavaScript and TypeScript, only the statistics processing server is written in Clojure.

In case of self hosting the preferred method of deployment is via Docker containers using Docker Compose or any other orchestrator. The participation frontend for conversations can either be used as a standalone web page or be embedded via an iframe.

For our planned use case we initially defined the following goals:

• custom branding and the integration into different content management systems (CMS)
• better support for mobile devices
• mandatory authentication and support for a broader range of authentication methods, including self-hosted solutions and DigiD
• support for alternative email sending services
• GDPR compliance

After a preliminary evaluation of our own and consulting with Policy Lab UK who were also evaluating and testing Polis and had already made a range of improvements related to self-hosting as well as bug fixes and modernization changes we decided to take their work as a base for our adaptations with the intent of submitting generally useful changes back to the Polis project.

Subsequently, a number of changes were implemented, including the removal of hardcoded domain names, the elimination of unnecessary cookies and third-party requests, support for an alternative email sending service, and the option of disabling Facebook and X integration.

For the branding our approach was to add an option allowing websites which are embedding conversations in an iframe to load an alternative stylesheet for overriding the native Polis branding. For this to be practical we intended to use CSS custom properties for defining branding-related styles such as colors and fonts. That approach turned out to be problematic because although the Polis participation frontend stylesheet is generated via SCSS and some of the colors are parameterized, however, they are not used consistently throughout the SCSS stylesheets, unfortunately. In addition the frontend templates contain a large amount of hardcoded style attributes. While we succeeded in implementing user-defined stylesheets, it took a disproportionate amount of development resources to parameterize all used colors and fonts via CSS custom properties aggravated by the fact that the SCSS and template files are huge and contain many unused rules and code.

derive-deftly, the template-based derive-macro facility for Rust, has been a great success.

It’s coming up to time to declare a stable 1.x version. If you’d like to try it out, and have final comments / observations, now is the time.

• Introduction to derive-deftly
• Status
• History
• Plans - call for review/testing

Introduction to derive-deftly

Have you ever wished that you could that could write a new derive macro without having to mess with procedural macros?

You can!

derive-deftly lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!:

use derive_deftly::{define_derive_deftly, Deftly}; define_derive_deftly! { ListVariants: impl $ttype { fn list_variants() -> Vec<&'static str> { vec![ $( stringify!( $vname ) , ) ] } } } #[derive(Deftly)] #[derive_deftly(ListVariants)] enum Enum { UnitVariant, StructVariant { a: u8, b: u16 }, TupleVariant(u8, u16), } assert_eq!( Enum::list_variants(), ["UnitVariant", "StructVariant", "TupleVariant"], );Status

derive-deftly has a wide range of features, which can be used to easily write sophisticated and reliable derive macros. We’ve been using it in Arti, the Tor Project’s reimplementation of Tor in Rust, and we’ve found it very useful.

There is comprehensive reference documentation, and more discursive User Guide for a more gentle introduction. Naturally, everything is fully tested.

History

derive-deftly started out as a Tor Hackweek project. It used to be called derive-adhoc. But we renamed it because we found that many of the most interesting use cases were really not very ad-hoc at all.

Over the past months we’ve been ticking off our “1.0 blocker” tickets. We’ve taken the opportunity to improve syntax, terminology, and semantics. We hope we have now made the last breaking changes.

Plans - call for review/testing

In the near future, we plan to declare version 1.0. After 1.x, we intend to make breaking changes very rarely.

So, right now, we’d like last-minute feedback. Are there any wrinkles that need to be sorted out? Please file tickets or MRs on our gitlab. Ideally, anything which might imply breaking changes would be submitted on or before the 13th of August.

In the medium to long term, we have many ideas for how to make derive-deftly even more convenient, and even more powerful. But we are going to proceed cautiously, because we don’t want to introduce bad syntax or bad features, which will require difficult decisions in the future about forward compatibility.

comments

Fellow Drupalists!

We have exciting news. As a way to express our appreciation for our community, the Drupal Association is pleased to announce some changes to the Individual Membership program, now called Ripple Makers!

Our goal is to increase communication and transparency, and create more opportunities for engagement.

The Drupal Association is a United States incorporated 501(c)3 nonprofit organization. Donations to Ripple Makers are tax deductible, where applicable. We raise funding primarily through event registration, event sponsorship, the Drupal Certified Partner program, some grants, and individual contributions. We do not have a for profit parent corporation telling us what to do, nor do we ever charge to download Drupal. In fact, it is our mission to ensure that it will always be free. 

Drupal has given members of our community opportunities beyond just a career. The values adopted by open source communities create innovation, collaboration, and creation across the globe. Drupal itself is recognized as a Digital Public Good. Drupal is a lever for change and makes a difference preserving the integrity of the internet. 

At the heart of the Ripple Makers transformation is a commitment to fostering deeper connections with each and every member of our community, recognizing the invaluable role that sustaining donors play in advancing our shared mission and impact. By revamping our sustaining donors club to be more inclusive and communicative, we aim to create a culture of belonging where every donor feels valued, heard, and empowered to make a difference. 

Ripple Makers is your chance to give back—and to celebrate the Drupal community and the open source ecosystem that helps it thrive. We invite you to make a monthly donation in support of the Drupal Association.

I became interested in Network Time Protocol (NTP) last year after learning how fundamental this protocol is to the functioning of the global Internet. NTP helps synchronize clocks on devices over the Internet, which is essential for secure browsing, timestamping, keeping everyone in sync or just checking what time it is. Computers usually have a hardware real-time clock (RTC) but that deviates over time, so an occasional sync over NTP is required to keep the time accurate. Many network and IoT devices don’t have hardware RTC so have even more reliance on NTP.

Accurate time keeping starts with reference clocks like atomic clocks, GPS etc. Multiple government standard agencies host these reference clocks, which are regarded as Stratum 0. Stratum 1 servers are known as primary servers, and directly connect to Stratum 0 clocks for time. Stratum 1 servers then distribute time to Stratum 2 and further down the hierarchy. Computers typically connects to one or more Stratum 1/2/3… servers to get their time.

Someone has to host these public Stratum 1,2,3… NTP servers. That’s what NTP pool, a global effort by volunteers does. They provide NTP servers for the public to use. As of today, there are 4700+ servers in the pool which are free to use for anyone.

Now let’s come to the reason for writing this post. Indian Computer Emergency Response Team (CERT-In) in April 2022 released a set of cybersecurity directions which set the alarm bells ringing. Internet Society (and almost everyone else) wrote about it.

And then there was this specific section about NTP:

All service providers, intermediaries, data centres, body corporate and Government organisations shall connect to the Network Time Protocol (NTP) Server of National Informatics Centre (NIC) or National Physical Laboratory (NPL) or with NTP servers traceable to these NTP servers, for synchronisation of all their ICT systems clocks. Entities having ICT infrastructure spanning multiple geographies may also use accurate and standard time source other than NPL and NIC, however it is to be ensured that their time source shall not deviate from NPL and NIC.

CSIR-National Physical Laboratory (NPL) is the official timekeeper for India and hosts the only public Stratum 1 clock in India, according to NTP pool website. So I was naturally curious to know what kind of infrastructure they’re running for NTP. India has a Right to Information (RTI) Act which, like the Freedom of Information Act (FOIA) in the United States, gives citizens rights to request information from governmental entities, to which they have to respond in under 30 days. So last year, I filed two sets of RTI (one after the first reply came) inquiring about NPL’s public NTP server setup.

The first RTI had some generic questions:

First RTI. Click to enlarge

This gave a vague idea about the setup, so I sat down and came with some specific questions in the next RTI.

Second RTI. Click to enlarge

Feel free to make your conclusions from it now. Bear in mind these were filled last year so things might have changed. Do let me know if you have more information about it.

Since its introduction, JSON has rapidly emerged as the predominant standard for the exchange of information. Whether you want to transfer data with an API or store information in a document database, it’s likely you’ll encounter JSON. Fortunately, Python provides robust tools to facilitate this process and help you manage JSON data efficiently.

In this tutorial, you’ll learn how to:

• Understand the JSON syntax
• Convert Python data to JSON
• Deserialize JSON to Python
• Write and read JSON files
• Validate JSON syntax
• Prettify JSON in the terminal
• Minify JSON with Python

While JSON is the most common format for data distribution, it’s not the only option for such tasks. Both XML and YAML serve similar purposes. If you’re interested in how the formats differ, then you can check out the tutorial on how to serialize your data with Python.

Free Bonus: Click here to download the free sample code that shows you how to work with JSON data in Python.

Take the Quiz: Test your knowledge with our interactive “Working With JSON Data in Python” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Working With JSON Data in Python

In this quiz, you'll test your understanding of working with JSON in Python. JSON has become the de facto standard for information exchange, and Python provides easy-to-use tools to handle JSON data.

Introducing JSON

The acronym JSON stands for JavaScript Object Notation. As the name suggests, JSON originated from JavaScript. However, JSON has transcended its origins to become language-agnostic and is now recognized as the standard for data interchange.

The popularity of JSON can be attributed to native support by the JavaScript language, resulting in excellent parsing performance in web browsers. On top of that, JSON’s straightforward syntax allows both humans and computers to read and write JSON data effortlessly.

To get a first impression of JSON, have a look at this example code:

JSON hello_world.json { "greeting": "Hello, world!" } Copied!

You’ll learn more about the JSON syntax later in this tutorial. For now, recognize that the JSON format is text-based. In other words, you can create JSON files using the code editor of your choice. Once you set the file extension to .json, most code editors display your JSON data with syntax highlighting out of the box:

The screenshot above shows how VS Code displays JSON data using the Bearded color theme. You’ll have a closer look at the syntax of the JSON format next!

Examining JSON Syntax

In the previous section, you got a first impression of how JSON data looks. And as a Python developer, the JSON structure probably reminds you of common Python data structures, like a dictionary that contains a string as a key and a value. If you understand the syntax of a dictionary in Python, you already know the general syntax of a JSON object.

Note: Later in this tutorial, you’ll learn that you’re free to use lists and other data types at the top level of a JSON document.

The similarity between Python dictionaries and JSON objects is no surprise. One idea behind establishing JSON as the go-to data interchange format was to make working with JSON as convenient as possible, independently of which programming language you use:

[A collection of key-value pairs and arrays] are universal data structures. Virtually all modern programming languages support them in one form or another. It makes sense that a data format that is interchangeable with programming languages is also based on these structures. (Source)

To explore the JSON syntax further, create a new file named hello_frieda.json and add a more complex JSON structure as the content of the file:

JSON hello_frieda.json 1{ 2 "name": "Frieda", 3 "isDog": true, 4 "hobbies": ["eating", "sleeping", "barking"], 5 "age": 8, 6 "address": { 7 "work": null, 8 "home": ["Berlin", "Germany"] 9 }, 10 "friends": [ 11 { 12 "name": "Philipp", 13 "hobbies": ["eating", "sleeping", "reading"] 14 }, 15 { 16 "name": "Mitch", 17 "hobbies": ["running", "snacking"] 18 } 19 ] 20} Copied!

In the code above, you see data about a dog named Frieda, which is formatted as JSON. The top-level value is a JSON object. Just like Python dictionaries, you wrap JSON objects inside curly braces ({}).

In line 1, you start the JSON object with an opening curly brace ({), and then you close the object at the end of line 20 with a closing curly brace (}).

Note: Although whitespace doesn’t matter in JSON, it’s customary for JSON documents to be formatted with two or four spaces to indicate indentation. If the file size of the JSON document is important, then you may consider minifying the JSON file by removing the whitespace. You’ll learn more about minifying JSON data later in the tutorial.

Inside the JSON object, you can define zero, one, or more key-value pairs. If you add multiple key-value pairs, then you must separate them with a comma (,).

Read the full article at https://realpython.com/python-json/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Long gone are the days of using HTML tables for page layouts. That time almost seems shrouded in myth. One of the (many) reasons tables fell out of use was the advent of responsive design to meet the needs of different contexts and screen sizes. Tables are rigid. Their presence naturally hampers the responsiveness of a website to conform to a smaller screen size.

But tables are still useful. In some cases, they are necessary.

In this quiz, you’ll test your understanding of Python’s Magic Methods.

By working through this quiz, you’ll revisit the concept of magic methods in Python, how they work, and how you can use them to customize the behavior of your classes.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Experience Builder (XB) is set to redefine how we build and launch Drupal sites. As a cornerstone of the Starshot initiative, the XB module bridges the gap between Drupal installation and site launch, simplifying the process for all users. Kazima Abbas, sub-editor at The Drop Times, interviews Lauri Eskola, the Product Lead for the XB initiative. Lauri shares the inspiration behind XB, its integration with Starshot, and how it enhances Drupal's user experience and accessibility.

tl;dr

Whenever you need to download files through the terminal and don't feel like using wget:

wcurl example.com/filename.txt

Manpage:
https://manpages.debian.org/unstable/curl/wcurl.1.en.html

Availability (comes installed with the curl package):

• Debian unstable - Since 2024-07-02
• Debian testing - Coming up between the second and third week of July 2024.
• Debian 12/bookworm backports - As soon as the package gets to Debian testing, I'll upload it to bookworm.
• Debian 12/bookworm - Depends on whether Debian's release team will approve it, it could be available in the next point release.
• Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.

If you don't want to wait for the package update to arrive, you can always copy the script and place it in your /usr/bin, the code is here:
https://github.com/Debian/wcurl/blob/main/wcurl
https://salsa.debian.org/debian/wcurl/-/blob/main/wcurl?ref_type=heads

Smoother CLI experience

Starting with curl version 8.8.0-2, the Debian's curl package now ships a wcurl executable.

wcurl is the solution for those who just need to download files without having to remember curl's parameters for things like automatically naming the files.

Some people, myself included, would fall back to using wget whenever there was a need to download a file. Sometimes even installing wget just for that usecase. After all, it's easier to remember "apt install wget" rather than "curl -L -O -C - ...".

wcurl consists of a simple shell script that provides sane defaults for the curl invocation, for when the use case is to just download files.

By default, wcurl will:

• Encode whitespaces in URLs;
• Download multiple URLs in parallel;
• Follow redirects;
• Automatically choose a filename as output;
• Perform retries;
• Resume from broken/interrupted downloads.
• Set the downloaded file timestamp to the value provided by the server, if available;

Example, to download a single file:

wcurl example.com/filename.txt

If you ever need to set a custom flag, you can make use of the -o/--opts wcurl option, anything set there will be passed to the curl invocation. Just beware that if you need to set any custom flags, it's likely you will be better served by calling curl directly. The -o/--opts options are there to allow for some flexibility in unforeseen circumstances.

The need for wcurl

I've always felt a bit ashamed of not remembering curl's parameters for downloading a file and automatically naming it, having resorted to wget most of the times this was needed (even installing wget when it wasn't there, just for this). I've spoken to a few other experienced people I know and confirmed what could be obvious to others: a lot of people struggle with this.

Recently, the curl project released the results of 2024's curl survey, which also showed this is as a much needed feature, just look at some of the answers:

Q: Which curl command line option do you think needs improvement and how?

-O, I really want wget like functionality where I don't have to specify the name

Downloading a file (like wget) could be improved - with automatic naming of the file

downloading files - wget is much cleaner

I wish the default behaviour when GETting a binary was to drop it on disk. That's the only reason 'wget foo.tgz" is still ingrained in my muscle memory .

Maybe have a way to download without specifying something in -o (the only reason i used wget still)

--remote-time should be default

--remote-name-all could really use a short flag

Q: If you miss support for something, tell us what!

"Write the data to the file named in the URL (or in redirects if I'm feeling daring), and timestamp the file to the last-modified-date". This is the main reason I'm still using wget.

I can finally feel less bad about falling back to wget due to not remembering the parameters I want.

Idealization vs. reality

I don't believe curl will ever change its default behavior in such a way that would accommodate this need, as that would have a side-effect of breaking things which expect the current behavior (the blast radius is literally the solar system).

This means a new executable needs to be shipped side-by-side with curl, an opportunity to start fresh and work with a more focused use case (to download files).

Ideally, this new executable would be maintained by the curl project, make use of libcurl under-the-hood, and be available everywhere. Nobody wants to worry if their systems have the tool or not, it should always be there.

Given I'm just a Debian Developer, with not as much free time as I wish, I've decided to write a simple shell script wrapper calling the curl CLI under-the-hood.

wcurl will come installed with the curl package from now on, and I will check with the release team about shipping it on the current Debian stable as well. Shipping wcurl in other distros will be up to them (Debian-derivatives should pick it up automatically, though).

We've tried to make it easy for anyone to ship this by using the curl license, keeping the script POSIX-compliant, and shipping a manpage.

Maybe if there's enough interest across distributions, someone might sign up for implementing this in upstream curl and increase its reach. I would be happy with the curl project reusing the wcurl name when that happens. It's unlikely that wcurl would be shipped by curl upstream as it is, assuming they would prefer a solution that uses libcurl direclty (more similar to curl the CLI, to maintain).

In the worst case, wcurl becomes a Debian-specific tool that only a few people are aware of, in the best case, it becomes the new go-to CLI tool for simply downloading files. I would be happy if at least someone other than me finds it useful.

Naming is hard

When I started working on it, I was calling the new executable "curld" (stands for "curl download"), but then when discussing this in one of our weekly calls in the Debian Brasília community, it was mentioned that this could be confused for a daemon.

We then settled for the name "wcurl", which doesn't really stand for anything, but it's very easy to remember.

You know... "it's that wget alternative for when you want to use curl instead" :)

Feedback

I'm hosting the code on Github and Debian's GitLab instance, feel free to open an issue to provide feedback.
https://salsa.debian.org/debian/wcurl
https://github.com/Debian/wcurl

We also have a Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org
We have historically spoken Portuguese in the room but we'll switch to English in case anyone joins.

Acknowledgments

The idea for wcurl came a few days before the curl-up conference 2024. I've been thinking a lot about developer productivity in the terminal lately, different tools and better defaults. Before curl-up, I was also thinking about packaging improvements for the curl package. I don't remember what exactly happened, but I likely had to download something and felt a bit ashamed of maintaining curl and not remembering the parameters to download files the way I wanted.

I first discussed this idea in the conference, where I asked the participants about it and there were no concerns raised, and some people said I should give it a go. Participating in curl-up was a really great experience and I'm thankful for the interactions I've had there.

On the Debian side, I've got reviews of the code and manpage by Sergio Durigan Junior <sergiodj>, Guilherme Puida Moreira <puida> and Carlos Henrique Lima Melara <charles>. Sergio ended up rewriting the tool to be POSIX-compliant (my version was written in bash), so he takes all the credit for the portability.

Calamares is an indepdendent Linux distro-installer, and we just released Calamares 3.3.7. There’s a couple of known issues that need tracking down, but it is a slow process – one entirely dependent on how much time volunteers are able and willing to put into careful bug reporting (and reproduction) and then dealing with code to fix them. Anyway, here’s some semi-coherent notes about Calamares and distro’s and issues and things.

Installing Kubuntu

I recently had to install an “emergency PC” for my mom, and picked Kubuntu – fairly arbitrarily, I must say. The install-Kubuntu icon does not look like Calamares, and first comes up with some other selection dialog in a different style, so I was pleasantly surprised to see it using Calamares after that.

It picked up Dutch from geo-location, which was fine. Less fine is that the Dutch translations aren’t complete. I’m pretty sure I could fix that.

Installation was straightforward, although I always pick “erase whole disk” and click through nearly all of the defaults. No complicated installs for me, and it worked without a hitch.

Mom didn’t need the emergency PC after all, so now I do have a spare low-end desktop machine with Kubuntu. I might turn it into a try-real-installs box for a while.

Building Calamares on KDE Neon

KDE Neon was a CI target for Calamares for a long time, but I switched it off in March 2024. Dependencies were a mess at the time, and the automatic CI builds were failing every night. Other distro’s didn’t have that struggle, so I put more CI weight on Fedora.

I did keep one of my desktop machines installed with KDE Neon, as a gaming machine. At least I can see what Plasma 6 is supposed to be like, as a reference for when we land it in FreeBSD. Untangling the package mess to return it to a development machine was just too much of a hassle.

At some point I removed Qt5 – and everything that depended on it – and then reinstalled some bits and did a pkcon update and whatever and the machine finally ended up in a workable state for development again, but don’t ask me exactly what I did and don’t talk to me about the upgrade experience.

Building Calamares on FreeBSD

It’s possible, just probably not very useful. Clang spits out mountains of warnings, which I occasionally try to address.

Not all of the warnings are all that useful – when dealing with command-line arguments, for instance. The relationship between argc and argv (conventional names for parameters) in main() is clear, but there’s nothing in the type of either to express that, so you get warnings like this one:

src/libcalamares/geoip/test_geoip.cpp:37:45: warning: unsafe buffer access [-Wunsafe-buffer-usage] 37 | QString selector = argc == 3 ? QString( argv[ 2 ] ) : QString(); Building Calamares on EndeavourOS

The live-ISO for EndeavourOS is one of my favorites. I don’t know why they ship an ISO with git in the live-image, but it means that I trivially have a system with a working Calamares configuration, which I can update with the latest version:

git clone http://github.com/calamares/calamares cd calamares sudo ./ci/deps-endeavouros.sh sudo pacman -Scc export CMAKE_ARGS="-DWITH_QT6=ON -DCMAKE_BUILD_TYPE=Debug" export BUILDDIR=build ./ci/build.sh

I have a VM with the live-ISO, but also two virtual disks attached. By doing the checkout and package download on one of the disks, and doing test-installs to the other, this is by far the easiest and most pleasant develop-and-test setup I have right now.

And the wallpaper is pretty nice, too.

We just released skrub 0.2.0. This release markedly simplifies learning on complex dataframes.

model = tabular_learner(‘classifier’) Simple, yet solid default baseline

The highlight of the release is the tabular_learner function, which facilitates creating pipelines that readily perform machine learning on dataframes, adding preprocessing to a scikit-learn compatible learner. The function packs defaults and heuristics to transform all forms of dataframes to a representation that is well suited to a learner, and it can adapt these transformation: tabular_learner(HistGradientBoostingClassifier()) encodes categories differently than tabular_learner(LogisticRegression()).

The heuristics are tuned based on much benchmarking and experience shows that they give good tradeoffs. The default tabular_learner(‘classifier’) is often a strong baseline.

The benefit are visible in a really simple example:

>>> # First retrieve data >>> from skrub.datasets import fetch_employee_salaries >>> dataset = fetch_employee_salaries() >>> df = dataset.X >>> y = dataset.y >>> # The dataframe is a quite rich and complex dataframe, with various columns >>> df

We can then easily build a learner that applies readily to this dataframe, without any transformation:

>>> from skrub import tabular_learner >>> learner = tabular_learner('regressor') >>> # The resulting learner can apply all the machine-learning conveniences (eg cross-validation) directly on the dataframe >>> from sklearn.model_selection import cross_val_score >>> cross_val_score(learner, df, y) array([0.89370447, 0.89279068, 0.92282557, 0.92319094, 0.92162666]) transformer = TableVectorizer() Making encoding complex dataframes easy

Behind the hood, the work is done by the skrub.TableVectorizer(), a scikit-learn compatible transformer that facilitates combining multiple transformations on the different columns of a dataframe. The TableVectorizer is not new in the 0.2.0 release, but we have completely revamped its internals to cover really well edge cases. Indeed, one challenge is to make sure that nothing different or strange happens at test time. Actually, enforcing consistency between train-time and test-time transformation is the real value of skrub compared to using pandas or polars to do transformation.

Increasing support of polars Short-term goal of optimized support for pandas and polars

We have implemented a new mechanism for supporting both pandas and polars. It has not been applied on all the codebase, hence the support is still imperfect. However, we are seeing increasing support for polars in skrub, and our goal in the short term is to provide rock-solid polar support.

Try skrub out! It’s still young, but in mind opinion, it provides a lot of value to tabular learning.

#636 – JULY 2, 2024
View in Browser »

Build a GUI Calculator With PyQt and Python

In this video course, you’ll learn how to create graphical user interface (GUI) applications with Python and PyQt. Once you’ve covered the basics, you’ll build a fully functional desktop calculator that can respond to user events with concrete actions.
REAL PYTHON course

Satellites Spotting Ships

Umbra Space has released a data set consisting of satellite based radar images of shipping. This article from Mark shows you how to grab the data, visualize, and annotate it.
MARK LITWINTSCHIK

Discover the Power of Observability With Pydantic Logfire

Logfire, by the makers of Pydantic, is an observability platform that will help you understand your app’s behavior with less code and time. Built on OpenTelemetry, it features user-friendly dashboards, SQL querying, and Python-specific integrations. Get started today →
PYDANTIC sponsor

Modern Good Practices for Python Development

This is a very detailed list of best practices for developing in Python. It includes tools, language features, application design, which libraries to use an more.
STUART ELLIS

PSF Board Candidates for 2024

PYTHON SOFTWARE FOUNDATION

Python 3.13.0 Beta 3 Released

CPYTHON DEV BLOG

Django 5.1 Beta 1 Released

DJANGO SOFTWARE FOUNDATION

PyBay 2024 Call for Proposals

PYBAY

Articles & Tutorials Build a Guitar Synthesizer: Play Musical Tablature in Python

In this tutorial, you’ll build a guitar synthesizer using the Karplus-Strong algorithm in Python. You’ll model vibrating strings, simulate strumming techniques, read musical notation and tablature, and apply audio effects. By the end, you’ll have created a digital guitar that can play any song. This tutorial was also discussed on Real Python Podcast Episode #210.
REAL PYTHON

Python’s Security Model After the xz-utils Backdoor

The backdoor introduced to the xz-utils compression project through social engineering was one of the topics at the Python Language Summit. Participants discussed what can be done to prevent similar social engineering attacks on the Python source.
PYTHON SOFTWARE FOUNDATION

Authentication Your Whole Team Will Love

“With PropelAuth, I think I’ve spent about a day – total – on auth over the past year.” PropelAuth is easy to integrate and provides all the tools your team needs to manage your users - dashboards, user insights, impersonation, SSO and more →
PROPELAUTH sponsor

Running Prettier Against Django or Jinja Templates

“Prettier” is a JavaScript based linting tool for templates. For folks not familiar with the world of npm, it can be a bit daunting to get it going. Simon fiddled with it so you don’t have to and posted how he got it working on his system.
SIMON WILLISON

Write Less Code, You Must

An often overlooked aspect of software development is architecture at the module & function level. It is important to write code that is simple and easy to move from one place to another.
DAVIDVUJIC.BLOGSPOT.COM • Shared by David Vujic

Quickstart for Playing With LLMs Locally

This is a simple, quick guide to getting started running LLMs on your local computer. It covers the basics of the powerful libraries Ollama and LangChain for controlling these AI models.
JOSHUA COOK • Shared by Joshua Cook

Under the Hood of Python’s set Data Structure

This tutorial covers hash tables, collision handling, performance optimization and how it relates to the implementation of the set data structure in Python.
ABHINAV UPADHYAY

Ways to Have an Atomic Counter in Django

Keeping a counter across objects in Django means having to be careful about race conditions. This article outlines several approaches to the problem.
GONÇALO VALÉRIO

Saying Thanks to Open Source Maintainers

Brett talks about the different ways you can support the many maintainers of open source projects, and often times just saying “thanks” means a lot.
BRETT CANNON

A Guide to Python’s Weak References

Learn all about weak references in Python: reference counting, garbage collection, and practical uses of the weakref module
MARTIN HEINZ • Shared by Martin Heinz

Creating Great README Files for Your Python Projects

In this tutorial, you’ll learn how to create, organize, and format high-quality README files for your Python projects.
REAL PYTHON

Get Terminal Size

This quick TIL post from Rodrigo shows you how to get information about the terminal size from the shutil module.
RODRIGO GIRÃO SERRÃO

A Complete Guide to pytest Fixtures

Learn how to use pytest fixtures for writing maintainable and isolated tests.
STANLEY ULILI

Projects & Code burr: Build Apps That Make Decisions (Chatbots, Agents, etc)

GITHUB.COM/DAGWORKS-INC

Lazy f-strings

GITHUB.COM/POMPONCHIK • Shared by pomponchik

oxo: Security Scanning Orchestrator

GITHUB.COM/OSTORLAB

jax: Composable Transformations of Python+NumPy Programs

GITHUB.COM/GOOGLE

dbt-utils: Utility Functions for DBT Projects

GITHUB.COM/DBT-LABS

Events Weekly Real Python Office Hours Q&A (Virtual)

July 3, 2024
REALPYTHON.COM

Canberra Python Meetup

July 4, 2024
MEETUP.COM

Sydney Python User Group (SyPy)

July 4, 2024
SYPY.ORG

EuroPython 2024

July 8 to July 15, 2024
EUROPYTHON.EU

SciPy US 2024

July 8 to July 14, 2024
SCIPY.ORG

PyCon Nigeria 2024

July 10 to July 14, 2024
PYCON.ORG

Happy Pythoning!
This was PyCoder’s Weekly Issue #636.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

For a while now I'd see an annoying problem when trying to analyze data. I would be trying to import into numpy an innocuous-looking data file like this:

# image x y z temperature image1.png 1 2 5 34 image2.png 3 4 1 35

As usual, I would be using vnlog.slurp() (a thin wrapper around numpy.loadtxt()) to read this in, but that doesn't work: the image filenames aren't parseable as numerical values. Up until now I would work around this by using the suprocess module to fork off a vnl-filter -p !image and then slurp that, but it's a pain and slow and has other issues. I just solved this conclusively using the numpy structured dtypes. I can now do this:

dtype = np.dtype([ ('image', 'U16'), ('x y z', int, (3,)), ('temperature', float), ]) arr = vnlog.slurp("data.vnl", dtype=dtype)

This will read the image filename, the xyz points and the temperature into different sub-arrays, with different types each. Accessing the result looks like this:

print(arr['image']) ---> array(['image1.png', 'image2.png'], dtype='<U16') print(arr['x y z']) ---> array([[1, 2, 5], [3, 4, 1]]) print(arr['temperature']) ---> array([34., 35.])

Notes:

• The given structured dtype defines both how to organize the data, and which data to extract. So it can be used to read in only a subset of the available columns. Here I could have omitted the temperature column, for instance
• Sub-arrays are allowed. In the example I could say either dtype = np.dtype([ ('image', 'U16'), ('x y z', int, (3,)), ('temperature', float), ])

  or

  dtype = np.dtype([ ('image', 'U16'), ('x', int), ('y', int), ('z', int), ('temperature', float), ])

  The latter would read x, y, z into separate, individual arrays. Sometime we want this, sometimes not.

• Nested structured dtypes are not allowed. Fields inside other fields are not supported, since it's not clear how to map that to a flat vnlog legend
• If a structured dtype is given, slurp() returns the array only, since the field names are already available in the dtype

We still do not support records with any null values (-). This could probably be handled with the converters kwarg of numpy.loadtxt(), but that sounds slow. I'll look at that later.

This is available today in vnlog 1.38.

Dear Debian community,

Statement on Daniel Pocock

The Debian project has successfully taken action to secure its trademarks and interests worldwide, as detailed in our press statement. I would like to personally thank everyone in the community who was involved in this process. I would have loved for you all to have spent your volunteer time on more fruitful things.

Debian Boot team might need help

I think I've identified the issue that finally motivated me to contact our teams: for a long time, I have had the impression that Debian is driven by several "one-person teams" (to varying extents of individual influence and susceptibility to burnout). As DPL, I see it as my task to find ways to address this issue and provide support.

I received private responses from Debian Boot team members, which motivated me to kindly invite volunteers to some prominent and highly visible fields of work that you might find personally challenging. I recommend subscribing to the Debian Boot mailing list to see where you might be able to provide assistance.

/usrmerge

Helmut Grohne confirmed that the last remaining packages shipping aliased files inside the package set relevant to debootstrap were uploaded. Thanks a lot for Helmut and all contributors that helped to implement DEP17.

Contacting more teams

I'd like to repeat that I've registered a BoF for DebConf24 in Busan with the following description:

This BoF is an attempt to gather as much as possible teams inside Debian to exchange experiences, discuss workflows inside teams, share their ways to attract newcomers etc.

Each participant team should prepare a short description of their work and what team roles (“openings”) they have for new contributors. Even for delegated teams (membership is less fluid), it would be good to present the team, explain what it takes to be a team member, and what steps people usually go to end up being invited to participate. Some other teams can easily absorb contributions from salsa MRs, and at some point people get commit access. Anyway, the point is that we work on the idea that the pathway to become a team member becomes more clear from an outsider point-of-view.

I'm lagging a bit behind my team contacting schedule and will not manage to contact every team before DebConf. As a (short) summary, I can draw some positive conclusions about my efforts to reach out to teams. I was able to identify some issues that were new to me and which I am now working on. Examples include limitations in Salsa and Salsa CI. I consider both essential parts of our infrastructure and will support both teams in enhancing their services.

Some teams confirmed that they are basically using some common infrastructure (Salsa team space, mailing lists, IRC channels) but that the individual members of the team work on their own problems without sharing any common work. I have also not read about convincing strategies to attract newcomers to the team, as we have established, for instance, in the Debian Med team.

DebConf attendance

The amount of money needed to fly people to South Korea was higher than usual, so the DebConf bursary team had to make some difficult decisions about who could be reimbursed for travel expenses. I extended the budget for diversity and newcomers, which enabled us to invite some additional contributors. We hope that those who were not able to come this year can make it next year to Brest or to MiniDebConf Cambridge or Toulouse

tag2upload

On June 12, Sean Whitton requested comments on the debian-vote list regarding a General Resolution (GR) about tag2upload. The discussion began with technical details but unfortunately, as often happens in long threads, it drifted into abrasive language, prompting the community team to address the behavior of an opponent of the GR supporters. After 560 emails covering technical details, including a detailed security review by Russ Allbery, Sean finally proposed the GR on June 27, 2024 (two weeks after requesting comments).

Firstly, I would like to thank the drivers of this GR and acknowledge the technical work behind it, including the security review. I am positively convinced that Debian can benefit from modernizing its infrastructure, particularly through stronger integration of Git into packaging workflows.

Sam Hartman provided some historical context [1], [2], [3], [4], noting that this discussion originally took place five years ago with no results from several similarly lengthy threads. My favorite summary of the entire thread was given by Gregor Herrmann, which reflects the same gut feeling I have and highlights a structural problem within Debian that hinders technical changes. Addressing this issue is definitely a matter for the Debian Project Leader, and I will try to address it during my term.

At the time of writing these bits, a proposal from ftpmaster, which is being continuously discussed, might lead to a solution. I was also asked to extend the GR discussion periods which I will do in separate mail.

Talk: Debian GNU/Linux for Scientific Research

I was invited to have a talk in the Systems-Facing Track of University of British Columbia (who is sponsoring rack space for several Debian servers). I admit it felt a bit strange to me after working more than 20 years for establishing Debian in scientific environments to be invited to such a talk "because I'm DPL". 😉

Kind regards Andreas.

GNU direvent version 5.4 is available for download.

New in this version:

Simultaneous execution limits

It is possible to limit number of command instances that are allowed to run simultaneously for a particular watcher.  This is done using
the max-instances statement in watcher section.

Restore the "nowait" default

In previous version, watchers waited for the handler to terminate, unless given the nowait option explicitly.  It is now fixed and nowait is the default, as described in the documentation.

Fix bug in generic to system event translation

Fix sentinel code

In some cases setting the sentinel effectively removed the original watcher.  That happened if the full file name of the original watcher
and its directory part produced the same hash code.

GNU dbm version 1.24 is available for download. New in this version:

New gdbm_load option: --update

The --update (-U) option instructs gdbm_load to update an existing database.

Fix semantics of gdbm_load -r

The --replace (-r) is valid only when used together with --update.

Use getline in gdbmtool shell

New function: gdbm_load_from_file_ext

In contrast to gdbm_load and gdbm_load_from_file, which derive the value of the flag parameter for gdbm_open from the value of their replace argument, this function allows the caller to specify it explicitly. 

Bugfixes

• Fix binary dump format for key and/or data of zero size (see bug 656)
• Fix location tracking and recover command in gdbtool (see bug 566)
• Fix possible buffer underflow in gdbmload.
• Ensure any padding bytes in avail_elem structure are filled with 0. This fixes debian bug 1031276.
• Improve the documentation.

This is the first article of a 5-episode blog post series written by Guido Berhörster, member of staff at my company Fre(i)e Software GmbH. Thanks, Guido for being on the Polis project.

Enjoy the read on the work Guido has been doing over the past months,
Mike

A team lead by Raoul Kramer/BetaBreak is currently adapting Polis for evaluation and testing by several Dutch provincial governments and central government ministries. Guido Berhörster (author of this article) who is an employee at Fre(ie) Software GmbH has been involved in this project as the main software developer. This series of blog posts describes how and why Polis was initially modified and adapted, what issues the team ran into and how this ultimately lead them to start a new Open Source project called Particiapp for accelerating the development of alternative Polis frontends compatible to but independent from the upstream project.

Table of Contents of the Blog Post Series

1. Introduction (this article)
2. Initial evaluation and adaptation
3. Issues extending Polis and adjusting our goals
4. Creating (a) new frontend(s) for Polis
5. Current status and roadmap

Polis - The Introduction What is Polis?

Polis is a platform for participation which helps to gather, analyze and understand viewpoints of large groups of participants on complex issues. In practical terms participants take part in “conversations” on a predefined topic by voting on statements or submitting their own statements (referred to as “comments” in Polis) for others to vote on1.

Through statistical analysis including machine learning participants are sorted into groups based on similarities in voting behavior. In addition, group-informed and overall consensus statements are identified and presented to participants in real-time. This allows for participants to react to and refine statements and either individually or through a predefined process to come to an overall consensus.

Furthermore, the order in which statements are presented to participants is influenced by a complex weighting system based on a number of factors such as variance, recency, and frequency of skipping. This so called “comment routing” is intended to facilitate a meaningful contribution of participants without requiring them to vote on each of a potentially huge number of statements 2.

Polis open-ended nature sets it apart from online surveys using pre-defined questions and allows its users to gather a more accurate picture of the public opinion. In contrast to a discussion forum or comment section where participants directly reply to each other, it discourages unproductive behavior such as provocations or personal attacks by not presenting statements in chronological order in combination with voting. Finally, its “comment routing” is intended to provide scalability towards a large number of participants which generate a potentially large number of statements.

The project was developed and is maintained by The Computational Democracy Project, a USA-based non-profit organization which provides a hosted version and offers related services. It is also released as Open Source software under the AGPL 3.0 license.

Polis has been used in a variety of different contexts as part of broader political processes facilitating broader political participation and opinion-forming, and gathering feedback and creative input.

Use of Polis in Taiwan

One prominent use case of Polis is its adoption as part of the vTaiwan participatory governance project. Established by the g0v civic tech community in the wake of the 2014 mass protests by the Sunflower movement, the vTaiwan project enables consultations on proposed legislation among a broad range of stakeholders including government ministries, lawmakers, experts, interest groups, civil society as well as the broader public. Although the resulting recommendations are non-binding, they exert pressure on the government to take action and recommendations have been adopted into legislation.345

vTaiwan uses Polis for large-scale online deliberations as part of a structured participation process. These deliberations take place after identifying and involving stakeholders and experts and providing through information about the topic at hand to the public. Citizens are then given the opportunity to vote on statements or provide alternative proposals which allows for the refinement of ideas and ideally leads to a consensus at the end. The results of these online deliberations are then curated, discussed in publicly broadcast face-to-face meetings which ultimately produce concrete policy recommendations. vTaiwan has in numerous cases given impulses resulting in government action and provided significant input e.g. on legislation regulating Uber or technological experiments by Fintech startups.35

See also

• Polis Case Studies
• BBC Click: Can Taiwan Reboot Democracy?
• Polis: Scaling Deliberation by Mapping High Dimensional Opinion Spaces

1. https://compdemocracy.org/Polis/ ↩

2. https://compdemocracy.org/comment-routing/ ↩

3. https://info.vtaiwan.tw/ ↩ ↩

4. https://www.theguardian.com/world/2020/sep/27/taiwan-civic-hackers-polis-consensus-social-media-platform ↩

5. https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/ ↩ ↩