Feeds

#612 – JANUARY 16, 2024
View in Browser »

Exploring Python in Excel

Are you interested in using your Python skills within Excel? Would you like to share a data science project or visualization as a single Office file? This week on the show, we speak with Principal Architect John Lam and Sr. Cloud Developer Advocate Sarah Kaiser from Microsoft about Python in Excel.
REAL PYTHON podcast

Python 3.13 Gets a JIT

This article does a deeper dive into the JIT recently added to the CPython 3.13 pre-release. This JIT is a bit different, it is called a copy-and-patch JIT, and the post explains what that means.
ANTHONY SHAW

NumPy 2 Is Coming: Preventing Breakage, Updating Your Code

NumPy 2 is coming, and it’s backwards incompatible. Learn how to keep your code from breaking, and how to upgrade.
ITAMAR TURNER-TRAURING

Build Invincible Apps With Temporal’s Python SDK

Get an introduction to Temporal’s Python SDK by walking through our easy, free tutorials. Learn how to build Temporal applications using Python, including building a data pipeline Workflow and a subscription Workflow. Get started here →
TEMPORAL sponsor

PSF Says: EU’s Cyber Resilience Act Has Wins for Open Source

PYTHON SOFTWARE FOUNDATION

Articles & Tutorials Learn From 2023’s Most Popular Python Tutorials and Courses

Revisit your favorite Real Python tutorials and video courses from 2023. Explore various topics, from Python basics to web development, machine learning, and effective coding environments. It’s been a busy year of learning, and there’s something for everyone to discover and build upon in 2024.
REAL PYTHON

Python’s Array: Working With Numeric Data Efficiently

In this tutorial, you’ll dive deep into working with numeric arrays in Python, an efficient tool for handling binary data. Along the way, you’ll explore low-level data types exposed by the array module, emulate custom types, and even pass a Python array to C for high-performance processing.
REAL PYTHON

Data Deduplication in Python with RecordLinkage

Duplicate detection is a critical process in data preprocessing, especially when dealing with large datasets. In this tutorial, you will explore data deduplication using Python’s RecordLinkage package, paired with Pandas for data manipulation.
PATRYK SZLAGOWSKI • Shared by Izabela Pawlik

The Curious Case of Pydantic and the 1970s Timestamps

When parsing Unix timestamps, Pydantic guesses whether to interpret them in seconds or milliseconds. While this is certainly convenient and works most of the time, it can drastically (and silently) distort timestamps from a few decades ago.
ARIE BOVENBERG • Shared by Arie Bovenberg

A Critical Supply Chain Attack on PyTorch

This post describes how coders found an exploit in the PyTorch supply chain, leaving the repo and its maintainers up for attack. Full details on what the vulnerability was and how to avoid the problem in your own repos is covered.
JOHN STAWINSKI

Comparing Coroutines, by Example, in Kotlin and Python

This series of 3 articles compares Python and Kotlin, with a focus on coroutines and generators. It compares, through examples, how coroutines are used in both languages, to read files and perform network requests.
MEDIUM.COM • Shared by Carmen Alvarez

Enhance Your Flask Web Project With a Database

Adding a database to your Flask project elevates your web app to the next level. In this tutorial, you’ll learn how to connect your Flask app to a database and how to receive and store posts from users.
REAL PYTHON

SQLALchemy vs Django ORM

If you are working with Django ORM most of the time and then switching to SQLAlchemy, you may face some unexpected behavior. This post describes the most important differences between them.
ALEXEY EVSEEV

Annotating *args and **kwargs in Python

“Typing *args and **kwargs has always been a pain since you couldn’t annotate them precisely before.” This article shows you what your options are when typing function signatures.
REDOWAN DELOWAR

Python Gotcha: Modifying a List While Iterating

Python makes it easy to modify a list while you are iterating through it’s elements. This will bite you. Read on to find out how and what can be done about it.
ANDREW WEGNER

A Deep Dive Into Python’s functools.wraps Decorator

Take a deep dive into Python’s functools.wraps decorator to learn how it maintains metadata in your code. A concise guide to effective decorator use.
JACOB PADILLA

max() is broken

The built-in function max in Python is broken and this article explains why, drawing parallels with other programming and mathematics concepts.
MATHSPP.COM • Shared by Rodrigo Girão Serrão

All PyCon 2023 (US and AU) Talks Sorted by the View Count

A full list of PyCon talks given in the US and Australia which are available on YouTube, and sorted by popularity.
SUBSTACK.COM

Projects & Code PikaPython: Python Interpreter in 4KB of RAM

GITHUB.COM/PIKASTECH

Fontimize: Optimize Fonts to the Glyphs on Your Site

GITHUB.COM/VINTAGEDAVE

instructor: Structured Outputs for LLMS

GITHUB.COM/JXNL

Pint: Units for Python

PYPI.ORG

Events Weekly Real Python Office Hours Q&A (Virtual)

January 17, 2024
REALPYTHON.COM

PyData Bristol Meetup

January 18, 2024
MEETUP.COM

PyLadies Dublin

January 18, 2024
PYLADIES.COM

Chattanooga Python User Group

January 19 to January 20, 2024
MEETUP.COM

IndyPy: Models & AI For Dummies (Hybrid)

January 23, 2024
MEETUP.COM • Shared by Laura Stephens

Happy Pythoning!
This was PyCoder’s Weekly Issue #612.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

At The DropTimes (TDT), we constantly evolve. The formative steps for us were challenging. We might be just out of infancy, but still in childhood. That does not give us any excuse not to be a better version of what we were yesterday. As we align with the build-in-open philosophy, let us record our recent decisions in response to community feedback.

Authored by: Nadiia Nykolaichuk.

Upon entering the new year, many of us have started working towards resolutions or aspiring achievements. In the digital space, Drupal websites, too, can aspire to be on the cutting edge of trends and provide the most compelling online journeys to users.

My to-read shelf

This year, with respect to my ever-growing reading backlog, I'm going to try something new: when I acquire a new book, I'm going to try to read at least a few pages of it immediately. My theory is this will help me to have a better idea of what to expect when I come to pick the next book to start, later on. A few pages may not be very representative of a full book (In "How to read a Novel: A User's Guide John Sullivan suggests reading 69 pages before giving up on a book), but it may be better than nothing.

I'll report back if it seems to work.

<p><a target="_blank" href="https://drupal.org/project/rjsf">RJSF</a> is a bit of a unique module in that there is more JavaScript than php in the module. This presents some challenges when trying to host and package the module from a Drupal.org repository. Like that, a release on d.o is a copy of a specific commit and not a build artifact controlled by the repo. Because of this RJSF ships the JS part of the module as an NPM package that is only updated when a release is tagged. This has some significant downsides like if you want to use the module you need to install both the module and the npm package. Or to use the dev version of the module you have to build the JS yourself. Or to tag a release I have to manually package a release for NPM. Recently I've been thinking about how to simplify the installation and release process for RJSF and decided to start by attempting to include the built js with the module instead of as an NPM package.</p><h2>The Goal</h2><p><ul><li>Include built javascript when the module is installed via composer</li><li>Update the built javascript whenever the source changes</li><li>Do it all automatically without any manual steps</li></ul></p>

A classic childhood game is tic-tac-toe, also known as naughts and crosses. It’s simple and enjoyable, and coding a version of it with Python is an exciting project for a budding programmer. Now, adding some artificial intelligence (AI) using Python can make an old favorite even more thrilling.

In this comprehensive tutorial, you’ll construct a flexible game engine. This engine will include an unbeatable computer player that employs the minimax algorithm to play tic-tac-toe flawlessly. Throughout the tutorial, you’ll explore concepts such as immutable class design, generic plug-in architecture, and modern Python coding practices and patterns.

In this video course, you’ll learn how to:

• Develop a reusable Python library containing the tic-tac-toe game engine
• Create a Pythonic code style that accurately models the tic-tac-toe domain
• Implement various artificial players, including one using the powerful minimax algorithm
• Construct a text-based console front end for the game, enabling human players to participate
• Discover effective strategies for optimizing performance

Are you ready to embark on this step-by-step adventure of building an extensible game engine with an unbeatable AI player using the minimax algorithm?

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Pamela Fox is a Python Cloud Developer Advocate at Microsoft. 

Topics include:

• Girl Develop It
• Django Girls
• Girls Who Code
• Teaching a language vs teaching a tool
• What a dev advocate does
• Accessibility (A11y) testing
• Playwright
• axe-core
• Snapshot testing
• pytest plugin authoring
• Flask SQLAlchemy
• Relearning Go

Links from the show:

• Python Bytes 323 with Pamela: AI search wars have begun
• Python Test 199 with Pamela: Is Azure Right for a Side Project?
• gdi: Girl Develop It
• Django Girls
• Girls Who Code
• "Automated accessibility audits" - Pamela Fox (North Bay Python 2023)
• Playwright
• axe-core
• pytest-axe-playwright-snapshot, plugin from Pamela
• pytest-crayons plugin is from a PyCascades talk about building plugins
• pytest-check, yet another plugin
• FlaskSQLAlchemy
• Concurrency is not Parallelism by Rob Pike

The Complete pytest Course

• Level up your testing skills and save time during coding and maintenance.
• Check out courses.pythontest.com

★ Support this podcast on Patreon ★ <p>Pamela Fox is a Python Cloud Developer Advocate at Microsoft. </p><p><br>Topics include:</p><ul><li>Girl Develop It</li><li>Django Girls</li><li>Girls Who Code</li><li>Teaching a language vs teaching a tool</li><li>What a dev advocate does</li><li>Accessibility (A11y) testing</li><li>Playwright</li><li>axe-core</li><li>Snapshot testing</li><li>pytest plugin authoring</li><li>Flask SQLAlchemy</li><li>Relearning Go</li></ul><p>Links from the show:</p><ul><li><a href="https://pythonbytes.fm/episodes/show/323/ai-search-wars-have-begun">Python Bytes 323 with Pamela: AI search wars have begun</a></li><li><a href="https://podcast.pythontest.com/episodes/199-is-azure-right-for-a-side-project">Python Test 199 with Pamela: Is Azure Right for a Side Project?</a></li><li><a href="https://girldevelopit.com">gdi: Girl Develop It</a></li><li><a href="https://djangogirls.org/en/">Django Girls</a></li><li><a href="https://girlswhocode.com">Girls Who Code</a></li><li><a href="https://www.youtube.com/watch?v=J-4Qa6PSomM">"Automated accessibility audits" - Pamela Fox (North Bay Python 2023)</a></li><li><a href="https://playwright.dev">Playwright</a></li><li><a href="https://github.com/dequelabs/axe-core">axe-core</a></li><li><a href="https://github.com/pamelafox/pytest-axe-playwright-snapshot">pytest-axe-playwright-snapshot</a>, plugin from Pamela</li><li><a href="https://www.youtube.com/watch?v=kevcz8NRcQU">pytest-crayons plugin is from a PyCascades talk about building plugins</a></li><li><a href="https://github.com/okken/pytest-check">pytest-check</a>, yet another plugin</li><li><a href="https://flask-sqlalchemy.palletsprojects.com/en/3.1.x/">FlaskSQLAlchemy</a></li><li><a href="https://www.youtube.com/watch?v=oV9rvDllKEg">Concurrency is not Parallelism by Rob Pike</a></li></ul><p><br></p> <br><p><strong>The Complete pytest Course</strong></p><ul><li>Level up your testing skills and save time during coding and maintenance.</li><li>Check out <a href="https://courses.pythontest.com/p/complete-pytest-course">courses.pythontest.com</a></li></ul> <strong> <a href="https://www.patreon.com/PythonPeople" rel="payment" title="★ Support this podcast on Patreon ★">★ Support this podcast on Patreon ★</a> </strong>

As part of my role in the Drupal Association, we are trying to find new ways to unleash innovation. Innovation as it happens is a key goal for the Drupal Association. What surprised me when I started with the Drupal Association was to meet companies that were contributors, (some of them known for being long-time contributors) or that are very interested in contributing, but then not knowing how they could maximize their contributions or even where they should be contributing to.

I don’t think that these are a few isolated cases, as it’s not the first time I've seen this trend. Back when I was working for a 100+ developer consultancy firm there was a big corporate push to increase our contribution to open source. And contribute we did. We started “Pizza Fridays”, which meant we were spending Fridays contributing, doing presentations between us, and having pizza for lunch. We had fun, but we lacked structure, purpose, and higher goals (and a healthy diet on Fridays). Our plan was not aligned with anything other than our own appetite to experiment or learn something.

If we had a structure that aligned us to the project we were contributing to, our contributions would have been more impactful, business would have benefited in a more meaningful way, and the whole team would have probably been allowed to contribute even further and longer in time. We did amazing things, don’t get me wrong, but the impact of those could have been much bigger.

That’s why, today, we are introducing the credit bounty program. The idea is to do an initial experiment, and if it has an impact on Drupal moving forward, we’ll tweak it if needed and continue with new iterations.

I expect that the issues and projects that we are promoting will change over time, so we’ll share soon how you can get updated information.

If you are a maintainer and you would like us to include your issues in this pilot program, that may be a possibility as well, so please send me an email: alex.moreno@association.drupal.org. Depending on how this first phase goes, we may start promoting contributed module issues as well based on the popularity of the modules, usage on sites, complexity, how innovative they are, etc, etc

For now, this is the list of issues where (core for now) maintainers need your help. The amount of credit we’ll be given is 50 credits, which is 5 times the normal amount of credits we would grant (normal core issues get 10 credits).

• Cannot save translated nodes after upgrading to 8.8 due to invalid path
  • https://www.drupal.org/project/drupal/issues/3101344
• Composer scaffolding fails when permissions on default.settings.yml or default.settings.php is not writable
  •  https://www.drupal.org/project/drupal/issues/3091285
• Configuration language being overwritten during module install
  • https://www.drupal.org/project/drupal/issues/2905295
• Prevent saving config entities when configuration overrides are applied
  • https://www.drupal.org/project/drupal/issues/2910353

Maintainers will grant credit as normal on these issues, and *all* of the contributing organizations and individuals that the maintainers credit will receive the full bounty. 

Make sure to read Drupal Core's Issue Etiquette for core contribution, and the Contributor Guide. 

Have questions or ideas? Please ping me: alex.moreno@association.drupal.org
 

Mastering Drupal API for Your Next Big Project Success kanapatrick Tue, 01/16/2024 - 14:23

Embarking on a large-scale web development project demands a robust framework that not only supports your vision but also enhances your efficiency and scalability. With over 20 years of development, Drupal has become one of the leading content management systems for building complex and robust websites.

In the next few weeks, the OSI board of directors will renew three of its seats with an open election process among its full individual members and affiliates. There will be two elections in March, running in parallel:

• The affiliate organizations will elect one director
• Individual members will elect two directors

The results of elections for both Individual and Affiliate member board seats are advisory with the OSI Board making the formal appointments to open seats based on the community’s votes.

Signup now to become a full individual member (Supporting or Professional) to qualify as a candidate when the application opens on Feb 5th.

2024 elections timeline The role of the board of directors

The board of directors is the ultimate authority responsible for the Open Source Initiative as a California public benefit corporation, with 501(c)3 tax-exempt status. The board’s responsibilities include oversight of the organization, approving the budget and supporting the executive director and staff to fulfill its mission. The OSI isn’t a volunteer-run organization anymore and the role of the directors has changed accordingly.

Each director is expected to be a counsel and a guide for staff rather than an active contributor. Directors should guide discussions, support the vision and mission of the organization, and advocate for the OSI. They’re also asked to support the fundraising efforts however they feel comfortable doing.

The board is governed by the bylaws. Each board member is expected to sign the board member agreement. Depending on expertise and availability, directors are expected to serve on the active committees: the license, fundraising, standards and financial committees.

Candidates will be asked to share their ideas on how they’ll contribute to the vision and mission, and the 2024 strategic objectives.

The rules for how OSI runs the elections are published on our website. We’ll communicate more details in the coming weeks: stay tuned for announcements on our social media channels (Fediverse, LinkedIn, Twitter.)

Affiliate organizations will receive instructions via email.

The post <span class='p-name'>How OSI will renew its board of directors in 2024</span> appeared first on Voices of Open Source.

It’s a pleasure to be on the OpenUK New Year’s Honours list for 2024. There’s some impressive names on there such as Richard Hughes of Packagekit and other projects at Red Hat, Colin Watson who was at Ubuntu with me and I see is now freelance, Mike McQuaid was previously of KDE but is now trying a startup with Mac packager Workbrew for Homebrew.

OpenUK run various activities for open tech in UK countries and KDE currently needs some more helpers for a stall at their State of Open Con in London on Feb 6 and 7 February, if you can help do get in touch.

KDE’s 6th releases will happen next month bringing with it the refresh of code and people that a new major version number can bring, I think KDE’s software in the coming year will continue to impress.

My life fell apart after some family loss last year so I’ve run away to the end of the world at Finesterre in Galicia in Spain for now, let me know if you’re in the area.

Happy 2024!

DAIS have continued their programme of posthumous Coil remasters and re-issues. Constant Shallowness Leads To Evil was remastered by Josh Bonati in 2021 and re-released in 2022 in a dizzying array of different packaging variants. The original releases in 2000 had barely any artwork, and given that void I think Nathaniel Young has done a great job of creating something compelling.

A limited number of the original re-issue have special lenticular covers, although these were not sold by any distributors outside the US. I tried to find a copy on my trip to Portland in 2022, to no avail.

Last year DAIS followed Constant… with Queens Of The Circulating Library, same deal: limited lenticular covers, US only.

Both are also available digital-only, e.g. on Bandcamp: Constant…, Queens…. The original, pre-remastered releases have been freely available on archive.org for a long time: Constant…, Queens…

Both of these releases feel to me that they were made available by the group somewhat as an afterthought, having been produced primarily as part of their live efforts. (I'm speculating freely here, it might not be true). Live takes of some of this material exist in the form of Coil Presents Time Machines, which has not (yet) been reissued. In my opinion this is a really compelling recording. I vividly remember listening to this whilst trying to get an hour's rest in a hotel somewhere on a work trip. It took me to some strange places!

I'll leave you from one of my favourite moments from "Colour Sound Oblivion", Coil's video collection of live backdrops. When this was performed live it was also called "Constant Shallowness Leads To Evil", although it's distinct from the material on the LP:

also available on archive.org. A version of this Constant… made it onto a Russian live bootleg, which is available on Spotify and Bandcamp complete with some John Balance banter: we only do this on religious holidays…

Constant Shallowness Leads to Evil by Coil

Posted on January 16, 2024 Tags: debian, free software, nix, life

The nix package manager is available in Debian since May 2020. Why would one use it in Debian?

• learn about nix
• install software that might not be available in Debian
• install software without root access
• declare software necessary for a user’s environment inside $HOME/.config

Especially the last point nagged me every time I set up a new Debian installation. My emacs configuration and my Desktop setup expects certain software to be installed.

Please be aware that I’m a beginner with nix and that my config might not follow best practice. Additionally many nix users are already using the new flakes feature of nix that I’m still learning about.

So I’ve got this file at .config/nixpkgs/config.nix1:

with (import <nixpkgs> {}); { packageOverrides = pkgs: with pkgs; { thk-emacsWithPackages = (pkgs.emacsPackagesFor emacs-gtk).emacsWithPackages ( epkgs: (with epkgs.elpaPackages; [ ace-window company org use-package ]) ++ (with epkgs.melpaPackages; [ editorconfig flycheck haskell-mode magit nix-mode paredit rainbow-delimiters treemacs visual-fill-column yasnippet-snippets ]) ++ [ # From main packages set ] ); userPackages = buildEnv { extraOutputsToInstall = [ "doc" "info" "man" ]; name = "user-packages"; paths = [ ghc git (pkgs.haskell-language-server.override { supportedGhcVersions = [ "94" ]; }) nix stack thk-emacsWithPackages tmux vcsh virtiofsd ]; }; }; }

Every time I change the file or want to receive updates, I do:

nix-env --install --attr nixpkgs.userPackages --remove-all

You can see that I install nix with nix. This gives me a newer version than the one available in Debian stable. However, the nix-daemon still runs as the older binary from Debian. My dirty hack is to put this override in /etc/systemd/system/nix-daemon.service.d/override.conf:

[Service] ExecStart= ExecStart=@/home/thk/.local/state/nix/profile/bin/nix-daemon nix-daemon --daemon

I’m not too interested in a cleaner way since I hope to fully migrate to Nix anyways.

1. Note the nixpkgs in the path. This is not a config file for nix the package manager but for the nix package collection. See the nixpkgs manual.↩︎

2025 may seem like a distant future, but for anyone managing a website on Drupal 7, January 5th 2025 is a date to mark on your calendar: Support and security updates for Drupal 7 are scheduled to end on that day. To stay relevant and secure, it's time to upgrade.

The new year has just begun and we have six weeks left before the final release! The most noticeable change since my last post is obviously that we have decided on the wallpaper to be used in Plasma 6.0! But of course there’s more going on under the hood than just that.

My desktop isn’t usually that tidy

I actually spent most of my time in Qt Wayland rather than KDE code lately but more on that in an upcoming blog post once all my changes have been integrated. Nevertheless, there are still plenty of Wayland-related and other improvements on the Plasma, Frameworks, and KDE Gear side to talk about here.

XDG Foreign Everywhere

After my previous experience of revamping the KWin Window Killer and having learned how to use the XDG Foreign Protocol (a Wayland protocol for exporting a surface to enable a different process to attach to it), I looked at all the places we have a helper application show a window in another application. This is actually done a lot more often than I thought and thus I added API in KWindowSystem for both exporting and importing windows on Wayland.

Since the export side is only really needed on Wayland, I added (un)exportWindow(QWindow*) functions in KWaylandExtras (a utility namespace with Wayland-specific windowing system functionality) along with a windowExported signal once that has been performed. Setting a foreign parent windows can already be done by using KWindowSystem::setMainWindow or even just QWindow::fromWinId. Both of them take a WId (a long int usually) which means that a string-based handle received from the compositor doesn’t play well with the existing APIs.

In case of KWindowSystem I just added a QString overload. The clever part is that it also understands a long int in a string, thus you just feed a token received as a string from QCommandLineParser or stdin verbatim into the API (even supports using 0x and 0b prefixes) and have it do the right thing on all platforms. That way you only have to special-case the export part for Wayland but the importing side will “just work”. Lifetime of the objects is tied to the actual window and there’s no additional resource tracking logic needed on the application side besides calling those functions, pretty neat.

Both kdialog and keditfiletype support XDG Foreign handles for their relevant attach/parent argument now. It is important to have all “public” command-line tools adjusted for 6.0, thereby knowing it’s a Qt 6 build is enough to tell whether it will understand the string or refuse to start on being unable to parse it as a number. If you know of any other tools that might need adjustment, please do tell. The KAuth Framework for executing privileged tasks as well as the KDE PolicyKit agent also learned how to create and understand those handles. This ensures the password prompt is attached to the window it came from (e.g. when changing settings). There’s still plenty of places where KAuth is used without setting a parent window on the executed action but at least the infrastructure is all there now.

More fractional scaling goodness

Of course I accomplished a few more fixes for fractional scaling. The pixmap created by Item.grabToImage now captures it with the proper scale factor. I also made a fix for then using that grab result in a Drag handler (not merged yet). Together with another fix for Plasma’s Folder View and a KWin change I did a while ago we should finally have crisp pixmaps when dragging icons on the desktop. That was a lot more entangled than I anticipated with the scale factor discarded at pretty much every opportunity along the way.

Left to right: Original item, original blurry capture, fixed rendering (salmon for illustration)

TextInput and TextEdit also re-render themselves immediately when the scale factor changes. This fixes sticky notes on the desktop being blurry until interacted with as well as spin boxes and other editable controls in apps like System Settings. The issue was addressed for labels some time ago but similar treatment was needed for input fields, too.

It’s the little things…

I talked about Dolphin’s icon rendering in September and now file thumbnails are properly rendered with fractional scaling, too, both in the main file view and the information sidebar. Méven Car worked on high-dpi support for thumbnailers previously and the switch to Frameworks 6 was a good opportunity to change the wire-format used for communication between app and thumbnailer to use a floating-point number for the scale factor instead of an integer. While at it, I also fixed the “tick” icon to accept input on the address bar. Furthermore, mouse cursor theme previews in System Settings are also scaled smoothly. Incidentally, KWin’s bouncing cursor (startup feedback) is scaled according to the cursor size on Wayland, too.

Speaking of Dolphin, the Places sidebar no longer lets you drag one place into another one. While I found that Windows also lets you do that (macOS doesn’t), I don’t really see the point, unless you want to drag your Documents folder to an external hard drive directly? More importantly, though, it makes re-arranging places very finicky since the drop area in-between places is very small. Obviously, it is still possible to drag files and folders from the main view and elsewhere onto an entry in the Places panel to copy/move/link it to the folder or hard drive in question.

In order to fix KWin’s screen edge triggering inadvertently while selecting text in an application and nudging the corner of the screen, I disabled them completely when a mouse cursor is pressed. However, I wasn’t entirely happy with this (and people in the comments weren’t either) and now it’s possible to drag a file into a screen edge and peek at the desktop to drop it there. Making the new overview effects work with drag and drop is going to be a lot more work, so it’s left for a later time. I’d also love to be able to switch virtual desktops while dragging a file, just like we do when touching the screen edge while dragging a window.

That irksome Konsole font bug finally squashed!

On the subject of broken rendering, I also found a way to improve font rendering in Konsole after the removal of QFont::ForceIntegerMetrics in Qt 6. From what I can tell Konsole entirely relies on the fact that all characters are the same width. Apparently, even with a Monospace font under certain conditions thanks to Hinting and Kerning this may not be entirely the case. Forcing full hinting on the font used seems to ensure that no such trickery is going on. If you still see broken fonts when text is highlighted or selecting it, check that your distribution doesn’t force any particular font settings here.

Anything else Okular’s annotation bar now showing proper page numbers and annotation contents

As a heavy user of Okular’s fantastic annotation feature where you can add labels, sticky notes, shapes, lines, and all sorts of other markers to a (PDF) document, I slightly improved the Annotation sidebar: Custom page numbers are now displayed (pages in a PDF aren’t necessarily strictly numbered but there can be Roman numerals for the index, for example, just like in a real book) and the textual content of the annotation, if any, is shown as well to more easily identify which item is which.

Libksysguard learned SI prefixes for Ampere, Volt, Watt, and Watt-hour units. Should I ever upgrade to a Petawatt solar installation, System Monitor has got me covered. KMessageDialog gained a beep function for playing the relevant notification sound (warning, error, etc), for applications that implement a custom message box and already depend on KWidgetsAddons but don’t want to pull in KNotification just for this. It is now used by Kate’s “Save?” dialog which looks like a message box when saving a single file but because it may also display a list of files is a custom implementation. I also had a look at how to make Qt’s own QMessageBox play the KDE sounds but this is routed though the Qt accessibility framework and I wasn’t sure how to hook into that without jeopardizing more important components of it like the AT-SPI interface.

Do you want to install updates or what?

With the prevalence of Offline Updates (i.e. restarting the system and installing updates in a minimal environment) there’s now a dedicated “Reboot & Install Updates” button on the logout screen to skip installing updates and just reboot. This could surely be extended in the future, e.g. shut down without installing updates or install them now instead of on next boot and so on but at least you can now easily reboot the system without installing updates if you want to.

One of the first changes exclusive to Plasma 6.1: WiFi channel display.

Finally, even though we’re all busy squashing any remaining bugs for the 6.0 release, it has been branched off into the “stable” release branch and the repositories are again open for gentle feature development (which includes anything that needs new translations). I just merged a tiny change exclusive to Plasma 6.1: displaying the WiFi channel number next to its frequency in connection details.

Discuss this post on KDE Discuss.

<strong>Topics covered in this episode:</strong><br> <ul> <li><a href="https://world.hey.com/dhh/we-have-left-the-cloud-251760fb"><strong>Leaving the cloud</strong></a></li> <li><a href="https://peps.python.org/pep-0723/"><strong>PEP 723 - Inline script metadata</strong></a></li> <li><a href="https://flet.dev/blog/flet-for-android"><strong>Flet for Android</strong></a></li> <li><a href="https://github.com/tconbeer/harlequin"><strong>harlequin: The SQL IDE for Your Terminal.</strong></a></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=qjl95MJwW1A' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="367">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by <strong>Bright Data</strong> : <a href="https://pythonbytes.fm/brightdata"><strong>pythonbytes.fm/brightdata</strong></a></p> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too.</p> <p><strong>Michael #1:</strong> <a href="https://world.hey.com/dhh/we-have-left-the-cloud-251760fb"><strong>Leaving the cloud</strong></a></p> <ul> <li>Also see <a href="https://world.hey.com/dhh/five-values-guiding-our-cloud-exit-638add47">Five values guiding our cloud exit</a> <ul> <li>We value independence above all else.</li> <li>We serve the internet. </li> <li>We spend our money wisely. </li> <li>We lead the way. </li> <li>We seek adventure.</li> </ul></li> <li>And <a href="https://world.hey.com/dhh/we-stand-to-save-7m-over-five-years-from-our-cloud-exit-53996caa">We stand to save $7m over five years from our cloud exit</a></li> <li>Slice our new monster 192-thread Dell R7625s into isolated VMs</li> <li>Which added a combined <a href="https://world.hey.com/dhh/the-hardware-we-need-for-our-cloud-exit-has-arrived-99d66966">4,000 vCPUs with 7,680 GB of RAM and 384TB of NVMe storage</a> to our server capacity</li> <li>They <a href="https://kamal-deploy.org">created Kamal</a> — Deploy web apps anywhere</li> <li>A lot of these ideas have changed how I run the infrastructure at Talk Python and for Python Bytes. </li> </ul> <p><strong>Brian #2:</strong> <a href="https://peps.python.org/pep-0723/"><strong>PEP 723 - Inline script metadata</strong></a></p> <ul> <li>Author: Ofek Lev</li> <li>This PEP specifies a metadata format that can be embedded in single-file Python scripts to assist launchers, IDEs and other external tools which may need to interact with such scripts.</li> <li>Example: <pre><code> # /// script # requires-python = "&gt;=3.11" # dependencies = [ # "requests&amp;lt;3", # "rich", # ] # /// import requests from rich.pretty import pprint resp = requests.get("https://peps.python.org/api/peps.json") data = resp.json() pprint([(k, v["title"]) for k, v in data.items()][:10]) </code></pre></li> </ul> <p><strong>Michael #3:</strong> <a href="https://flet.dev/blog/flet-for-android"><strong>Flet for Android</strong></a></p> <ul> <li>via Balázs</li> <li><a href="https://talkpython.fm/episodes/show/378/flet-flutter-apps-in-python">Remember Flet</a>?</li> <li>Here’s a <a href="https://flet.dev/docs/guides/python/drag-and-drop">code sample</a> (scroll down a bit).</li> <li>It’s amazing but has been basically impossible to deploy. </li> <li>Now we have Android.</li> <li>Here’s a good <a href="https://www.youtube.com/watch?v=Hj09tFCdjSw">YouTube video</a> showing the build process for APKs.</li> </ul> <p><strong>Brian #4:</strong> <a href="https://github.com/tconbeer/harlequin"><strong>harlequin: The SQL IDE for Your Terminal.</strong></a></p> <ul> <li>Ted Conbeer &amp; other contributors</li> <li>Works with DuckDB and SQLite</li> <li>Speaking of SQLite <ul> <li><a href="https://mastodon.social/@webology/111766195410833730">Jeff Triplett and warnings of using Docker and SQLite in production</a></li> <li><a href="https://blog.pecar.me/">Anže’s post</a></li> <li>and and article: <a href="https://blog.pecar.me/django-sqlite-dblock">Django, SQLite, and the Database is Locked Error</a></li> </ul></li> </ul> <p><strong>Extras</strong> </p> <p><strong>Brian</strong>:</p> <ul> <li>Recent <a href="https://pythonpeople.fm">Python People</a> episodes <ul> <li>Will Vincent</li> <li>Julian Sequeira</li> <li>Pamela Fox</li> </ul></li> </ul> <p><strong>Michael</strong>:</p> <ul> <li>PageFind and <a href="https://fosstodon.org/@mkennedy/111637520985150159">how I’m using it</a></li> <li>When "<a href="https://socket.dev/blog/when-everything-becomes-too-much?utm_source=tldrnewsletter">Everything" Becomes Too Much</a>: The npm Package Chaos of 2024</li> <li>Essay: <a href="https://mkennedy.codes/posts/michael-kennedys-unsolicited-advice-for-mozilla-and-firefox/">Unsolicited Advice for Mozilla and Firefox</a></li> <li><a href="https://fosstodon.org/@matthewfeickert/111763520503201675">SciPy 2024 is coming to Washington</a> </li> </ul> <p><strong>Joke:</strong> Careful with that <a href="https://trello.com/1/cards/655ef44fcc1657159ad4102c/attachments/655ef452b9b27b86253285c2/download/1700711828998blob.jpg">bike lock combination code</a></p>

Content editors are always looking for ways to get more efficient and faster at creating/updating content. It can always be challenging to start with a blank page. Having a predefined set of templates to begin with makes the job much easier and quicker. Let us introduce you to the Drupal 10 CKEditor Templates module. CKEditor is a popular rich text editor used in many content management systems, including Drupal. Currently, Drupal 10 supports CKEditor version 5 and it comes with a great user experience, it is highly customizable and extensible. CKEditor offers many plugins to extend its functionality. One such plugin in focus today is  “Content Templates” which is extremely useful for content creators and editors.  The CKEditor Templates module is an extension/plugin for the CKEditor WYSIWYG editor in Drupal, that allows you to easily insert pre-defined templates or content snippets into your content. We are grateful for the wonderful Drupal community contributors who created this module! How to Install the CKEditor Templates Module The installation of the Drupal 10 CKEditor Templates module is the same as any other Drupal module. It has dependencies on the CKEditor template plugin. If the plugin is needed, add it to the libraries folder at the root of the Drupal setup. After installing and adding the library, enable the module. Now, two main steps need to be followed to work with the module: Configure CKEditor to include the CKEditor Templates button in the toolbar. Create and manage templates in the template library. Configure CKEditor to include the CKEditor Template Button Configuration -> Content Authoring -> Text formats and editors Now, all the text formats will be listed. Configure any text format that needs the Template insert option (using full HTML in this tutorial) and move the 'Templates' icon from "Available buttons" to "Active Toolbar." Create and Manage Templates in the Template Library Configuration -> Content Authoring -> CKEditor TemplatesNow, the previously created templates will also be listed. Click on the ‘Add CKEditor template’ button to add a new template. It will now redirect to the template page and the following fields must be filled. Title: Title of Template Description: Description of the Template Image path for template: Preview the image of the Template. This can be added manually. Also, it will auto-fill when uploading an image using the next field ‘Image path for this template’. Image path for this template: This will upload the image and the ‘Image path for template’ field will be filled with the path of the uploaded image. Body: The predefined template’s HTML code will be added in this field which will be inserted in CKEditor when this template is added. Inserting the Template into CKEditor Now that you have configured the templates, let's look at how to add them to your CKEditor. Add/Edit any node, block, or other component with CKEditor. Click on the "Template" icon (resembling the attached screenshot) to insert the template. A popup with a list of templates will appear. Click on the template you want to insert in CKEditor, and the template will be added. Make changes in content and save the component. Final Thoughts The Drupal 10 CKEditor Templates module enhances content creation by providing content editors with a user-friendly platform to insert pre-defined templates or snippets effortlessly. This module integrates seamlessly with the CKEditor WYSIWYG editor in Drupal, making the process more efficient and enjoyable. As a focused Drupal development company, our experts continually strive to improve the end-user experience. If you’re looking for any Drupal development services, please do not hesitate to reach out to us. We’re here to assist you. 

Review: Making Money, by Terry Pratchett

Series: Discworld #36 Publisher: Harper Copyright: October 2007 Printing: November 2014 ISBN: 0-06-233499-9 Format: Mass market Pages: 473

Making Money is the 36th Discworld novel, the second Moist von Lipwig book, and a direct sequel to Going Postal. You could start the series with Going Postal, but I would not start here.

The post office is running like a well-oiled machine, Adora Belle is out of town, and Moist von Lipwig is getting bored. It's the sort of boredom that has him picking his own locks, taking up Extreme Sneezing, and climbing buildings at night. He may not realize it, but he needs something more dangerous to do. Vetinari has just the thing.

The Royal Bank of Ankh-Morpork, unlike the post office before Moist got to it, is still working. It is a stolid, boring institution doing stolid, boring things for rich people. It is also the battleground for the Lavish family past-time: suing each other and fighting over money. The Lavishes are old money, the kind of money carefully entangled in trusts and investments designed to ensure the family will always have money regardless of how stupid their children are. Control of the bank is temporarily in the grasp of Joshua Lavish's widow Topsy, who is not a true Lavish, but the vultures are circling.

Meanwhile, Vetinari has grand city infrastructure plans, and to carry them out he needs financing. That means he needs a functional bank, and preferably one that is much less conservative.

Moist is dubious about running a bank, and even more reluctant when Topsy Lavish sees him for exactly the con artist he is. His hand is forced when she dies, and Moist discovers he has inherited her dog, Mr. Fusspot. A dog that now owns 51% of the Royal Bank and therefore is the chairman of the bank's board of directors. A dog whose safety is tied to Moist's own by way of an expensive assassination contract.

Pratchett knew he had a good story with Going Postal, so here he runs the same formula again. And yes, I was happy to read it again. Moist knows very little about banking but quite a lot about pretending something will work until it does, which has more to do with banking than it does with running a post office. The bank employs an expert, Mr. Bent, who is fanatically devoted to the gold standard and the correctness of the books and has very little patience for Moist. There are golem-related hijinks. The best part of this book is Vetinari, who is masterfully manipulating everyone in the story and who gets in some great lines about politics.

"We are not going to have another wretched empire while I am Patrician. We've only just got over the last one."

Also, Vetinari processing dead letters in the post office was an absolute delight.

Making Money does have the recurring Pratchett problem of having a fairly thin plot surrounded by random... stuff. Moist's attempts to reform the city currency while staying ahead of the Lavishes is only vaguely related to Mr. Bent's plot arc. The golems are unrelated to the rest of the plot other than providing a convenient deus ex machina. There is an economist making water models in the bank basement with an Igor, which is a great gag but has essentially nothing to do with the rest of the book. One of the golems has been subjected to well-meaning older ladies and 1950s etiquette manuals, which I thought was considerably less funny (and somewhat creepier) than Pratchett did. There are (sigh) clowns, which continue to be my least favorite Ankh-Morpork world-building element. At least the dog was considerably less annoying than I was afraid it was going to be.

This grab-bag randomness is a shame, since I think there was room here for a more substantial plot that engaged fully with the high weirdness of finance. Unfortunately, this was a bit like the post office in Going Postal: Pratchett dives into the subject just enough to make a few wry observations and a few funny quips, and then resolves the deeper issues off-camera. Moist tries to invent fiat currency, because of course he does, and Pratchett almost takes on the gold standard, only to veer away at the last minute into vigorous hand-waving. I suspect part of the problem is that I know a little bit too much about finance, so I kept expecting Pratchett to take the humorous social commentary a couple of levels deeper.

On a similar note, the villains have great potential that Pratchett undermines by adding too much over-the-top weirdness. I wish Cosmo Lavish had been closer to what he appears to be at the start of the book: a very wealthy and vindictive man (and a reference to Cosimo de Medici) who doesn't have Moist's ability to come up with wildly risky gambits but who knows considerably more than he does about how banking works. Instead, Pratchett gives him a weird obsession that slowly makes him less sinister and more pathetic, which robs the book of a competent antagonist for Moist.

The net result is still a fun book, and a solid Discworld entry, but it lacks the core of the best series entries. It felt more like a skit comedy show than a novel, but it's an excellent skit comedy show with the normal assortment of memorable Pratchettisms. Certainly if you've read this far, or even if you've only read Going Postal, you'll want to read Making Money as well.

Followed by Unseen Academicals. The next Moist von Lipwig book is Raising Steam.

Rating: 8 out of 10

urllib3 is fundraising for HTTP/2 support About • Blog • Newsletter • Links urllib3 is fundraising for HTTP/2 support

Published 2024-01-16 by Seth Larson
Reading time: minutes

TLDR: urllib3 is raising ~$40,000 USD to release HTTP/2 support and ensure long-term sustainable maintenance of the project after a sharp decline in financial support for 2023.

What is urllib3?

urllib3 is an HTTP client library for Python and is depended on by widely used projects like pip, Requests, major cloud and service provider SDKs, and more. urllib3 is one of the most used Python packages overall, installed over 4 billion times in 2023 with 1.5 million dependent repos on GitHub, up 50% from just last year.

Project update

2023 was a transformative year for urllib3, headlined by the first stable release of v2.0 after multiple years of development by our maintainers and community. This major release is only the beginning of our plans to overhaul the library’s capabilities by removing constraints on our HTTP implementation while preserving backwards compatibility.

We’ve been able to accomplish this incredible work in 2023 thanks to financial support from Tidelift, the Spotify 2022 FOSS Fund, and our other sponsors which allowed us to offer bounties on tasks to fairly compensate maintainers and contributors for their time investments with the project.

Unfortunately, compared to past years we’ve experienced a sharp drop in financial support from non-Tidelift sources heading into 2024.

Year Non-Tidelift Funding 2019 $18,580 2020 $100* 2021 $9,950 2022 $14,493 2023 $2,330

* December 2020 was the first time we offered ad-hoc financial support via GitHub Sponsors. Before this we only accepted grants for funding.

Our team has worked hard to set the stage for HTTP/2 support with urllib3 v2.0, and we plan to land HTTP/2 support without compromising on the sustainability of the project. Backwards-compatible HTTP/2 support in urllib3 would immediately benefit millions of users, among them the largest companies in the world, and requires adding more long-term maintenance burden to maintainers. This important work and its maintenance should not be uncompensated.

To ensure timely and sustainable development of HTTP/2 for urllib3 we're launching a fundraiser with a goal of raising our Open Collective balance to $50,000 USD. HTTP/2 support has just started being developed and we're hoping to release stable support once our fundraising goal has been reached. Donations to Open Collective directly or to platforms like GitHub Sponsors or Thanks.dev will all be counted towards this fundraising goal.

Our team has a long track record of using our financial resources to complete larger projects like secure URL parsing, TLS 1.3, modernizing our test suite framework, and finding security issues across multiple projects. All receipts are published publicly on our Open Collective with links to the work items being accomplished and blogged about by our maintainers. If you or your organization has questions about this fundraiser please email sethmichaellarson@gmail.com or ask in our community Discord.

There’s more information below about the work we’ve done so far for HTTP/2 support and what else we plan to do in 2024 during our fundraiser. Thanks for supporting open source software!

Funding update

urllib3 received $17,830 US dollars in financial support in 2023 from all sources and distributed $24,350 to contributors and maintainers. Our primary supporter continues to be Tidelift, who provided $15,500 to core maintainers Seth, Quentin, and Illia.

We distributed $1,800 to community contributors through our bounty program, less than last year but still a sizable amount. We are looking to leverage our bounty program more in 2024 to implement HTTP/2 and WebAssembly features.

Our Open Collective started the year with nearly $19,000 USD and ended the year with $12,179. This statistic clearly shows the gap in funding, comparing this year's fundraising of $2,330 to the average across 4 prior years of over $10,000 per year.

2022 OC Balance → Open Collective: $18,932 Tidelift → Tidelift Lifters: $15,500 Open Collective → 2023 OC Balance: $12,179 Tidelift → Tidelift Partnerships*: $12,000 Tidelift Partnerships* → Seth Larson: $12,000 Tidelift Lifters → Seth Larson: $6,904 Tidelift Lifters → Quentin Pradet: $6,603 Open Collective → Illia Volochii: $3,275 Open Collective → Quentin Pradet: $2,325 Tidelift Lifters → Illia Volochii: $1,993 Open Collective → Bounty Program: $1,800 Open Collective → Seth Larson: $1,450 GitHub Sponsors → Open Collective: $1,346 Sourcegraph → Open Collective: $600 Thanks.dev → Open Collective: $379 Open Collective → OSC Host Fees: $233 Donations → Open Collective: $5 Tidelift: $27,500 Tidelift Partnerships*: $12,000 Seth Larson: $20,354 Tidelift Lifters: $15,500 Quentin Pradet: $8,928 Illia Volochii: $5,268 2022 OC Balance: $18,932 Open Collective: $21,262 GitHub Sponsors: $1,346 Sourcegraph: $600 Thanks.dev: $379 Donations: $5 Bounty Program: $1,800 2023 OC Balance: $12,179 OSC Host Fees: $233 Tidelift$27,500 Tidelift Partnerships*$12,000 Seth Larson$20,354 Tidelift Lifters$15,500 Quentin Pradet$8,928 Illia Volochii$5,268 2022 OC Balance$18,932 Open Collective$21,262 GitHub Sponsors$1,346 Sourcegraph$600 Thanks.dev$379 Donations$5 Bounty Program$1,800 2023 OC Balance$12,179 OSC Host Fees$233

* Seth Larson was also paid $7,000 by Tidelift for a packaging security standards project and $5,000 as a part of their "lifter advocate" program. Neither of these projects are directly related to urllib3 but are listed for completeness.

Maintenance update

2023 marks the 15th anniversary of urllib3 being first published to PyPI! 🥳 Not many open source projects stand the test of time and continue to see the widespread usage that urllib3 does every day. We attribute our longevity to quickly elevating contributors from our community into project maintainers which we believe is a critical property of a sustainable open source project. Financial rewards through our bounty program is a crucial piece of our approach to staying sustainable for the long-term.

This year we welcomed a new core maintainer to our team, Illia Volochii! 🎉 Illia has been putting in high quality and consistent work to get v2.0 out the door. Illia started contributing to urllib3 in 2022 and after landing multiple high-quality pull requests was asked to join the team of collaborators and begin reviewing PRs and issues and helping with the release process.

After adding Illia we now have three core maintainers including Seth Larson and Quentin Pradet, in addition to multiple collaborators and community contributors.

We landed 160 commits from 13 unique contributors during 2023 which is up from ~130 commits during 2022. We published 16 releases to PyPI in 2023, up from 8 in 2022.

From a security perspective, we continue to lead the pack for Python packages in terms of implementing security standards. urllib3 is the highest rated project according to OpenSSF Scorecard with a score of 9.6 out of 10 overall. We also were an early adopter of Trusted Publishers, adopting the new feature days after they were announced during PyCon US 2023.

We remediated two moderate-severity vulnerabilities in 2023 and made the fixes available in both the new v2.0 and security-fix only v1.26.x release streams. Support for the previous major version of urllib3 is provided thanks to funding from Tidelift.

Support for HTTP/2

When you first read this post you might have thought:

“Hasn't HTTP/2 been around for a long time?” 🤔

And you'd be right! HTTP/2 was published in 2015 in RFC 7540 and is now used for the majority of web requests. HTTP/2 and has been around for so long that there's an already HTTP/3!

So why are we only just now starting to add support for HTTP/2 to urllib3? The reason is that the standard library module http.client only supports HTTP/1 and before urllib3 v2.0 was released urllib3 was strongly tied to http.client APIs. By breaking backwards compatibility in a few key ways (while maintaining compatibility where it matters for most users) we've been able to set the stage for adding HTTP/2 to urllib3! 🚀

urllib3 is in good company: many of Python's stable HTTP clients don't support HTTP/2 like Requests (which uses urllib3 under the hood), aiohttp, and httplib2.

Even though we're waiting to release HTTP/2 support until after our fundraiser concludes, we aren't waiting to get started. Our team has already started some of the required prep-work to implement HTTP/2. Want to follow along? We have a top-level tracking issue for HTTP/2 support on GitHub.

Over the past two months Quentin has migrated our test suite from the venerable Tornado web backend to using the Hypercorn server and Quart microframework. Our test application communicates with the server using ASGI, which is perfect for our use-case: low-level enough to satisfy the needs of the test suite and high-level enough to abstract the differences between HTTP/1 and HTTP/2. Now that the test suite runs with both HTTP/1 and HTTP/2, we can start developing HTTP/2 with an extensive initial battery of test cases.

Support for Webassembly and Emscripten

When PyScript was first announced at PyCon US 2022 during a keynote by Peter Wang, Seth was sitting front row to witness Python moving to the web. Later that same day in the PyScript open space there were experiments for making HTTP requests with urllib3 and Pyodide together using a synchronous call to the JavaScript fetch() API. At the time, despite having assistance from PyScript maintainers, there didn't seem to be a way forwards yet.

Fast-forward to today, the pyodide-http project has figured out how to make a synchronous or streaming HTTP exchange using the fetch() and XMLHttpRequest JavaScript APIs along with Web Workers. Now that a synchronous approach to HTTP requests was possible we could add support to urllib3!

Thanks to Joe Marshall, urllib3 now has experimental support for the Emscripten platform, complete with bundling a small JavaScript stub for Web Worker support and testing against Chrome and Firefox in our CI. What's next is to thoroughly test and document the feature. We're aiming to release stable Emscripten support for urllib3 in 2024.

The most exciting part of this is that once a core dependency like urllib3 has been made compatible with Emscripten we'll likely see a wave of other packages that immediately become compatible too, bringing even more of the Python package ecosystem to the web 🥳

Stable release of urllib3 v2.0

urllib3 had its first stable release of v2.0 in April 2023 and later the v2.1.0 release to remove many long-deprecated features like the [secure] extra which had become redundant with new improvements to the ssl standard library module and the urllib3.contrib.securetransport module which was needed on macOS due to unavailability of an OpenSSL library on the platform to perform HTTPS with PyPI.

This release also put the project in a good place for future improvements like those discussed above. The biggest blocker to adopting new HTTP implementations were vestigial APIs from urllib3 primarily subclassing the standard libraries http.client (or for Python 2: httplib) modules.

By removing and discouraging these implicit APIs we're better able to adopt alternate HTTP implementations such as the h2 library for HTTP/2 and JavaScript's fetch API for Emscripten.

Increasing adoption of urllib3 v2.x

The initial adoption of urllib3 v2.x was lower than expected, due to the following factors:

• By default, RedHat Enterprise Linux 7 (RHEL 7), AWS Lambda, Amazon Linux 2 and Read the Docs were all compiling the ssl module with OpenSSL 1.0.2. While botocore still pinned urllib3 to 1.26.x, Amazon Linux 2 was more popular than we expected and many users were not pinning or resolving their dependencies correctly and thus were receiving an incompatible version of urllib3.
• Various third-party packages like dockerpy, request-toolbelt and vcrpy were relying on implementation details of urllib3 that were deprecated or removed in v2.0 so couldn’t upgrade right away.
• And finally, we intentionally removed the strict parameter from HTTPResponse which had no effect since Python 3. This affected only a few users.

After a few weeks, we had around 3 millions daily downloads for v2.0. That's a lot of downloads, but only accounted for 30% of 1.26.x downloads at the time without any obvious upward trend. The only exception was Read the Docs that encouraged users to move to Ubuntu 22.04 and Python 3.11 shortly after the urllib3 2.0 release. To avoid a prolonged split in the ecosystem, we took various actions to help migrating to 2.x:

• Helped some libraries upgrade, including requests, docker-py, vcrpy, and requests-toolbelt.
• We added common migration issues to the v2 migration guide.
• With help from a LibreSSL developer and a Gentoo user, we added back LibreSSL support.
• To allow google-auth-library-python and the rest of the Google ecosystem to upgrade, we added back pyOpenSSL to allow in-memory certificate support.

Our friend and Requests maintainer, Nate Prewitt allowed urllib3 v2.0 for Python 3.10+ users of botocore. This work on Requests inspired snowflake-connector-python to follow suit.

Today, most popular libraries support urllib3 2.0 and later, at least with Python 3.10 and above. And the libraries that don't support it yet get requests from users. urllib3 2.x is reliably above 70% of 1.26.x downloads and growing. Additionally, Python 3.10+ users already download 2.x more than 1.26.x, making us confident that the ecosystem split will eventually disappear in favor of the newest major version of urllib3.

👋 That's all for now, if you want to discuss this article you can join our community Discord. Please share this article to help spread the word of our fundraiser and coming HTTP/2 support.

Thanks for reading! ♡ Did you find this article helpful and want more content like it? Get notified of new posts by subscribing to the RSS feed or the email newsletter.

This work is licensed under CC BY-SA 4.0

As well as the collection and distribution of compromised keys, the pwnedkeys project also matches those pwned keys against issued SSL certificates. I’m excited to announce that, as of the beginning of 2024, all matched certificates are now being published on the Fediverse, thanks to the botsin.space Mastodon server.

Want to know which sites are susceptible to interception and interference, in (near-)real time? Do you have a burning desire to know who is issuing certificates to people that post their private keys in public? Now you can.

How It Works

The process for publishing pwned certs is, roughly, as follows:

1. All the certificates in Certificate Transparency (CT) logs are hoovered up (using my scrape-ct-log tool, the fastest log scraper in the west!), and the fingerprint of the public key of each certificate is stored in an LMDB datafile.

2. As new private keys are identified as having been compromised, the fingerprint of that key is checked against all the LMDB files, which map key fingerprints to certificates (actually to CT log entry IDs, from which the certificates themselves are retrieved).

3. If one or more matches are found, then the certificates using the compromised key are forwarded to the “tooter”, which publishes them for the world to marvel at.

This makes it sound all very straightforward, and it is… in theory. The trick comes in optimising the pipeline so that the five million or so new certificates every day can get indexed on the one slightly middle-aged server I’ve got, without getting backlogged.

Why Don’t You Just Have the Certificates Revoked?

Funny story about that…

I used to notify CAs of certificates they’d issued using compromised keys, which had the effect of requiring them to revoke the associated certificates. However, several CAs disliked having to revoke all those certificates, because it cost them staff time (and hence money) to do so. They went so far as to change their procedures from the standard way of accepting problem reports (emailing a generic attestation of compromise), and instead required CA-specific hoop-jumping to notify them of compromised keys.

Since the effectiveness of revocation in the WebPKI is, shall we say, “homeopathic” at best, I decided I couldn’t be bothered to play whack-a-mole with CAs that just wanted to be difficult, and I stopped sending compromised key notifications to CAs. Instead, now I’m publishing the details of compromised certificates to everyone, so that users can protect themselves directly should they choose to.

Further Work

The astute amongst you may have noticed, in the above “How It Works” description, a bit of a gap in my scanning coverage. CAs can (and do!) issue certificates for keys that are already compromised, including “weak” keys that have been known about for a decade or more (1, 2, 3). However, as currently implemented, the pwnedkeys certificate checker does not automatically find such certificates.

My plan is to augment the CT scraping / cert processing pipeline to check all incoming certificates against the existing (2M+) set of pwned keys. Though, with over five million new certificates to check every day, it’s not necessarily as simple as “just hit the pwnedkeys API for every new cert”. The poor old API server might not like that very much.

Support My Work

If you’d like to see this extra matching happen a bit quicker, I’ve setup a ko-fi supporters page, where you can support my work on pwnedkeys and the other open source software and projects I work on by buying me a refreshing beverage. I would be very appreciative, and your support lets me know I should do more interesting things with the giant database of compromised keys I’ve accumulated.