Feeds

FTP master

This month I accepted 441 and rejected 29 packages. The overall number of packages that got accepted was 448.

I couldn’t believe my eyes, but this month I really accepted the same number of packages as last month.

Debian LTS

This was my hundred-twenty-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. During my allocated time I uploaded or worked on:

• [unstable] libcupsfilters security update to fix one CVE related to validation of IPP attributes obtained from remote printers
• [unstable] cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
• [unstable] cups security update to fix one CVE related to validation of IPP attributes obtained from remote printers
• [DSA 5778-1] prepared package for cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
• [DSA 5779-1] prepared package for cups security update to fix one CVE related to validation of IPP attributes obtained from remote printers
• [DLA 3905-1] cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers
• [DLA 3904-1] cups security update to fix one CVE related to validation of IPP attributes obtained from remote printers
• [DLA 3905-1] cups-filters security update to fix two CVEs related to validation of IPP attributes obtained from remote printers

Despite the announcement the package libppd in Debian is not affected by the CVEs related to CUPS. By pure chance there is an unrelated package with the same name in Debian. I also answered some question about the CUPS related uploads. Due to the CUPS issues, I postponed my work on other packages to October.

Last but not least I did a week of FD this month and attended the monthly LTS/ELTS meeting.

Debian ELTS

This month was the seventy-fourth ELTS month. During my allocated time I uploaded or worked on:

• [ELA-1186-1]cups-filters security update for two CVEs in Stretch and Buster to fix the IPP attribute related CVEs.
• [ELA-1187-1]cups-filters security update for one CVE in Jessie to fix the IPP attribute related CVEs (the version in Jessie was not affected by the other CVE).

I also started to work on updates for cups in Buster, Stretch and Jessie, but their uploads will happen only in October.

I also did a week of FD and attended the monthly LTS/ELTS meeting.

Debian Printing

This month I uploaded …

• … libcupsfilters to also fix a dependency and autopkgtest issue besides the security fix mentioned above.
• … splix for a new upstream version. This package is managed now by OpenPrinting.

Last but not least I tried to prepare an update for hplip. Unfortunately this is a nerve-stretching task and I need some more time.

This work is generously funded by Freexian!

Debian Matomo

This month I even found some time to upload packages that are dependencies of Matomo …

• … php-spyc
• … php-sparkline

This work is generously funded by Freexian!

Debian Astro

This month I uploaded a new upstream or bugfix version of:

• … openvlbi
• … indi-playerone
• … libsbig
• … indi-pentax
• … indi-sbig
• … indi-fishcamp
• … indi-inovaplx
• … libfishcamp
• … libsbig
• … libplayeronecamera
• … libplayerone
• … libahp-gt
• … libahp-xc

Most of the uploads were related to package migration to testing. As some of them are in non-free or contrib, one has to build all binary versions. From my point of view handling packages in non-free or contrib could be very much improved, but well, they are not part of Debian …

Anyway, starting in December there is an Outreachy project that takes care of automatic updates of these packages. So hopefully it will be much easier to keep those package up to date. I will keep you informed.

Debian IoT

This month I uploaded new upstream or bugfix versions of:

• … pywws

Debian Mobcom

This month I did source uploads of all the packages that were prepared last month by Nathan and started the transition. It went rather smooth except for a few packages where the new version did not propagate to the tracker and they got stuck in old failing autopkgtest. Anyway, in the end all packages migrated to testing.

I also uploaded new upstream releases or fixed bugs in:

• … osmo-libasn1c

misc

This month I uploaded new upstream or bugfix versions of:

• … dict-element
• … libcds
• … libctl
• … tntnet
• … gnucobol3
• … gnucobol4
• … meep-mpi-default
• … nuspell
• … c2go

Most of those uploads were needed to help packages to migrate to testing.

#650 – OCTOBER 8, 2024
View in Browser »

Differences Between Python’s Mutable and Immutable Types

In this video course, you’ll learn how Python mutable and immutable data types work internally and how you can take advantage of mutability or immutability to power your code.
REAL PYTHON course

DuckDB in the Browser With Pyodide

Learn how to run DuckDB in an in-browser Python environment to enable simple querying on remote files, interactive documentation, and easy to use training materials.
ALEX MONAHAN

Speech-to-Text With Django

Looking to add new functionality to your Django app? Learn how to integrate Speech-to-Text and build a working app that transcribes audio files—with 100+ free hours to get started →
ASSEMBLY AI sponsor

Free Threaded Python With Asyncio

This post talks about combining the new experimental free threading feature of Python 3.13 with Asyncio.
CHANGS.CO.UK • Shared by Jamie Chang

Python 3.13.0 Released

PYTHON.ORG

Python 3.12.7 Released

CPYTHON DEV BLOG

Quiz: Python 3.13: Cool New Features for You to Try

REAL PYTHON

Quiz: Syntactic Sugar: Why Python Is Sweet and Pythonic

REAL PYTHON

Articles & Tutorials Switching From virtualenvwrapper to direnv

Earlier this week Trey considered whether to switch from virtualenvwrapper to using local .venv managed by direnv. He then also started experimenting with uv and Starship. This post explains why and his new configuration.
TREY HUNNER

Ensuring a Block Is Overridden in a Django Template

Some template blocks are meant to be overloaded and forgetting to do so results in rendering bugs. This post talks about creating a new tag that throws an exception which alerts your tests if you forget to overload.
TOM CARRICK

Build Your Own AI Assistant with Edge AI

Simplify workloads and elevate customer service. Build customized AI assistants that respond to voice prompts with powerful language and comprehension capabilities - all based on your unique needs with Intel’s OpenVINO toolkit.
INTEL CORPORATION sponsor

Arrange, Act, and Assert Pattern in Testing

Learn what the Arrange, Act, and Assert (AAA) pattern is, how it works, the benefits it offers, and its role in unit test automation. Note: sample code is not in Python, but the concepts apply to all unit testing.
ANTONELLO ZANINI

How I Prepare a Technical Talk

This article outlines the system that Rodrigo uses to prepare his Python talks. Steal his ideas and suggestions so that you, too, can start giving talks at your local meetups and at PyCons all over the world.
MATHSPP.COM • Shared by Rodrigo

Implementing a Python Singleton With Decorators

A singleton pattern is one where only one instance of an object type is allowed at a time. One way to implement this concept is through the use of a decorator. This post teaches you how.
PIETER CLAERHOUT

PEP 759: External Wheel Hosting

This Python Enhancement Proposal specifies a mechanism by which projects hosted on pypi.org can safely host wheel artifacts on external sites other than PyPI.
PYTHON.ORG

Let’s Go Easy on PyPI, OK?

The use of containers can mean a lot of calls to PyPI. This post talks about caching properly to reduce the load on our shared community servers.
MICHAEL KENNEDY

Rotating Turn Order With deque

If you need to cycle through values, one way to do that is with deque. This post shows you through an example service for a game engine.
JUHA-MATTI SANTALA

Python 3.13: JIT and GIL Went Up the Hill

All you need to know about the latest Python release’s changes to the Global Interpreter Lock and Just-in-Time compilation.
DREW SILCOCK

A Dinosaur Learns poetry

“Not a real dinosaur and not real poetry.” This post is about Paul changing how what tools he uses for his Python setup.
PAUL COCHRANE

Projects & Code pytest-xflaky: A Flaky-Test Hunter Plugin

GITHUB.COM/TESORIO • Shared by Caio Ariede

foc: A Collection of Python Functions for Somebody’s Sanity

GITHUB.COM/THYEEM

spiderweb: A Small Web Framework

GITHUB.COM/ITSTHEJOKER

pipefunc: DAGs for Scientific Workflows

GITHUB.COM/PIPEFUNC • Shared by Bas Nijholt

django-unique-user-email: Emial Logins With User Model

GITHUB.COM/CARLTONGIBSON

Events Weekly Real Python Office Hours Q&A (Virtual)

October 9, 2024
REALPYTHON.COM

PyCon Uganda 2024

October 9 to October 14, 2024
PYCON.ORG

PyCon NL 2024

October 10 to October 11, 2024
PYCON.ORG

Python Atlanta

October 10 to October 11, 2024
MEETUP.COM

DFW Pythoneers 2nd Saturday Teaching Meeting

October 12, 2024
MEETUP.COM

PyCon MEA & Data Science 2024

October 14 to October 16, 2024
GLOBALDEVSLAM.COM

Python Brasil 2024

October 16 to October 21, 2024
PYTHONBRASIL.ORG.BR

PyCon Panamá 2024

October 16 to October 19, 2024
PYCON.PA

Swiss Python Summit 2024

October 17 to October 19, 2024
PYTHON-SUMMIT.CH

Happy Pythoning!
This was PyCoder’s Weekly Issue #650.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Maps on websites are a useful addition, allowing users to reach, for example, your company headquarters in a simple way. Depending on the requirements, you can implement them in many different ways. This article will show you how to embed Google Maps in a Drupal website.

The Sovol SV08 is a 3D printer which is a semi-assembled clone of Voron 2.4, an open-source design. It's not the cheapest of printers, but for what you get, it's extremely good value for money—as long as you can deal with certain, err, quality issues.

Anyway, I have one, and one of the fun things about an open design is that you can switch out things to your liking. (If you just want a tool, buy something else. Bambu P1S, for instance, if you can live with a rather closed ecosystem. It's a bit like an iPhone in that aspect, really.) So I've put together a spreadsheet with some of the more common choices:

Pimp my SV08

It doesn't contain any of the really difficult mods, and it also doesn't cover pure printables. And none of the dreaded macro stuff that people seem to be obsessing over (it's really like being in the 90s with people's mIRC scripts all over again sometimes :-/), except where needed to make hardware work.

I had the privilege of attending LAS this year. True to my role as a designer, I brought my camera and volunteered during the event to be a photographer. The venue and university of Monterrey were beautiful.

The main hall is a wall-to-wall glass building placed in the middle of campus. The pictures we got from there were so nice!

The day began with a review of OnlyOffice features and capabilities. We then reviewed the progress that Mexico has seen in advancing Open Source initiatives.

The sessions showcased a myriad of topics. They focused on how open source applications can make a difference in many areas. Other sessions focused on design guidelines, application-building logic, publication and efforts to promote Linux in education.

The work done by the organization was great. Internet access at the venue was strong, and allowed the team onsite to broadcast the sessions online. We were in a university setting. A team managed the broadcasting and sound for the venue and online audiences.

The city was beautiful and filled with great food.

During the conference I contributed with images that I will make available to the organizers soon.

Would love to come back!

Join the FSF and friends on Friday, October 11 from 12:00 to 15:00 EDT (16:00 to 19:00 UTC) to help improve the Free Software Directory.

Python 3.13 was published on October 7, 2024. This new version is a major step forward for the language, although several of the biggest changes are happening under the hood and won’t be immediately visible to you.

In a sense, Python 3.13 is laying the groundwork for some future improvements, especially to the language’s performance. As you watch the course, you’ll learn more about the background for this and dive into some new features that are fully available now.

In this video course, you’ll learn about some of the improvements in the new version, including:

• Improvements made to the interactive interpreter (REPL)
• Clearer error messages that can help you fix common mistakes
• Advancements done in removing the global interpreter lock (GIL) and making Python free-threaded
• The implementation of an experimental Just-In-Time (JIT) compiler
• A host of minor upgrades to Python’s static type system

In this video course, you’ll explore these changes and see how this new version of Python can work for you.

If you want to try any of the examples in this video course, then you’ll need to use Python 3.13. The Python 3 Installation & Setup Guide and How Can You Install a Pre-Release Version of Python? walk you through several options for adding a new version of Python to your system.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

As businesses rely more heavily on Open Source software (OSS), the strain on maintainers to provide timely updates and security patches continues to grow – often without fair compensation for their crucial work. Recent high-profile security incidents like XZ and Log4Shell have put a spotlight on the security challenges developers face against a backdrop of burnout that has reached an all-time high. 

To help address this imbalance, the Open Source Initiative (OSI) supports the Open Source Pledge, launched today by Sentry and partners to support maintainers and inspire a shift toward a healthier work-life balance, and more robust software security practices. The Pledge is a commitment from member companies to pay Open Source maintainers and organizations meaningfully in support of a more sustainable maintainer ecosystem and a reduction of flare-ups of high-profile security incidents.

This Pledge is an attempt to address a problem that has long existed within the Open Source ecosystem. Many companies have built their businesses on top of Open Source software, benefiting from the contributions of maintainers taking them for granted. While they’ve reaped the rewards, the burden has been placed on unpaid or underpaid developers.

It is essential that companies recognize their role in sustaining the ecosystem that powers their innovations. By taking the Pledge, companies have one more instrument to commit to supporting an ecosystem of maintainers and organizations, ensuring the long-term health of the Open Source projects they rely on.

In order to qualify, the projects that companies pledge to should meet the Open Source Definition. You can join the Open Source Pledge by donating to the Open Source Initiative or contacting us to become a sponsor.

In this quiz, you’ll test your understanding of Python closures. Closures are a common feature in functional programming languages and are particularly popular in Python because they allow you to create function-based decorators.

Take this quiz after reading our Python Closures: Common Use Cases and Examples tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Today we've issued the 5.1.2 bugfix release.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Natalia Bidart: 2EE82A8D9470983E.

This year we are conducting the eighth iteration of the official Python Developers Survey. The goal is to capture the current state of the language and the ecosystem around it. By comparing the results with last year’s, we can identify and share with everyone the hottest trends in the Python community and the key insights into it.

We encourage you to contribute to our community’s knowledge by sharing your experience and perspective. Your participation is valued! The survey should only take you about 10-15 minutes to complete.

Contribute to the Python Developers Survey 2024!

This year we aim to reach even more of our community and ensure accurate global representation by highlighting our localization efforts:

• The survey is translated into Spanish, Portuguese, Chinese, Korean, Japanese, German, French and Russian. It has been translated in years past, as well, but we plan to be louder about the translations available this year!
• To assist individuals in promoting the survey and encouraging their local communities and professional networks we have created a Promotion Kit with images and social media posts translated into a variety of languages. We hope this promotion kit empowers folks to spread the invitation to respond to the survey within their local communities.
• We’d love it if you’d share one or more of the posts below to your social media or any community accounts you manage, as well as share the information in discords, mailing lists, or chats you participate in.
• If you would like to help out with translations you see are missing, please request edit access to the doc and share what language you will be translating to. Translation into languages the survey may not be translated to is also welcome.
• If you have ideas about what else we can do to get the word out and encourage a diversity of responses, please comment on the corresponding Discuss thread.
  

The survey is organized in partnership between the Python Software Foundation and JetBrains. After the survey is over, we will publish the aggregated results and randomly choose 20 winners (among those who complete the survey in its entirety), who will each receive a $100 Amazon Gift Card or a local equivalent.

Testing the feed in English and check If it's going to Debian Planet.

Sorry the noise :-)

Things you can get/do as a Python core developer

I’ve been a Python core developer since January 27, 2017.

Being a Python core developer comes with perks, privileges, and also responsibilities.

Sometimes I can’t tell whether something is a perk, or a privilege, or a responsibility. I think depends on who you’re talking to, they might see it as an optional nice thing they could get/do, but the same thing might be seen as burden responsibility to others.

Plasma 6 has come into its own over the last two releases. The wrinkles that always come with a major migration have been ironed out, and it’s time to start delivering on the promises of the new Qt 6 and Wayland technology platforms that Plasma is built on top of.

One of the outstanding issues has been to make Plasma a more artist-friendly environment by providing full support for the hardware that creative people need to get their work done.

So let’s start there…

What’s New For Digital Artists

Plasma 6.2 includes a smorgasbord of new features for users of drawing tablets. Open System Settings and look for Drawing Tablet to see various tools for configuring drawing tablets.

New in Plasma 6.2: a tablet calibration wizard and test mode; a feature to define the area of the screen that your tablet covers (the whole screen or a section); and the option to re-bind pen buttons to different kinds of mouse clicks.

All this is built into Plasma; there’s no need to install new drivers or software from device manufacturers.

And if your tablet is not yet supported, “We care about your Input” is a community-wide project that aims to provide support for unusual input devices. Let us know about your device so we can add it to the list!

Color Management

Related to the above — and to ensure consistent colors across monitors — we’ve implemented more complete support for the Wayland color management protocol, and enabled it by default.

We have also improved brightness handling for HDR and ICC profiles, as well as HDR performance. This will improve your experience when designing graphics, playing games, and watching videos.

A new tone mapping feature built into Plasma’s KWin compositor will help improve the look of images with a brightness or set of colors greater than what the screen can display, thus reducing the “blown out” look such images can otherwise exhibit.

Before After Power Management

Managing how much energy your system consumes and when are not only important for preserving its resources for when you need them, but also for using it in an environmentally responsible way.

You can now override misbehaving applications that block the system from going to sleep or locking the screen (and thus prevent saving power), and you can also adjust the brightness of each connected monitor machine separately.

As for the Power and Battery widget, it not only shows how much power is remaining, but also allows you to adjust power profiles for different scenarios. New in Plasma 6.2: hold down the Meta (Windows) key and press B to cycle through the different options one at a time. A little badge of a leaf will show up on the battery icon to indicate when the system is in power save mode, and a rocket for performance mode.

Discover and System Updating

Another thing we put you in complete control of is your software.

Plasma’s built-in app store and software management tool, Discover, now supports PostmarketOS packages for your mobile devices, helps you write better reviews of apps, and presents apps’ license information more accurately.

You can also now choose to shut down the system after applying an offline system update, in addition to the existing option to restart afterwards.

Accessibility

Since we made improving accessibility a community-wide project, we have increased the ways in which Plasma is easy to use for everyone.

In Plasma 6.2, we overhauled System Settings’ Accessibility page and added colorblindness filters. We also added support for the full “sticky keys” feature on Wayland.

UI/Visual Design

And of course, improving the look and feel of Plasma is always a high priority from one release to the next.

In Plasma 6.2, we tweaked accent colors and the System Tray, reworked the Widget Explorer, and unified the look of dialogs and pop-ups. Finally, we improved the Welcome Center, sound effects, and actions.

Many of these changes are subtle, but will provide a smoother and more enjoyable experience.

And All This Too…

• The Weather Report widget now shows “feels like” temperatures, adds more information for BBC weather forecasts, and more.
• You can turn off window borders in the Pager widget.
• The Minimize All widget now minimizes only windows on the current virtual desktop and activity.
• You can now give custom names to your custom shortcuts.
• There's now an integrated cropping tool when setting a new user avatar.
• We’ve added a once-a-year donation request notification — please consider using it to show your love for Plasma by donating!

…and there’s much more going on as well: numerous stability and performance improvements, smaller visual changes, more Wayland protocols, and more! Check out the complete changelog for Plasma 6.2.

I just installed a new FreeBSD desktop machine. For this one, I wanted to have KDE Plasma6, since that’s already running on my Linux laptop and gaming machine – and it’s time for me to dogfood on FreeBSD. Here’s some notes.

Base Install

FreeBSD has a Live ISO and a graphical installer. For licensing reasons, it isn’t Calamares (which is GPLv3). So I use the text-based installer. I downloaded the FreeBSD 14.1 memstick image. From boot to reboot into an installed system takes less than five minutes, but then you have an old-school UNIX system: a login: prompt.

From there:

• log in as root
• switch the package repository to latest
• pkg install pkg, to get the package manager
• pkg install git cmake, because you’re a developer
• pkg install plasma6-plasma, which is an 800MiB download
• pkg install sddm, because you’ll want a login-manager
• pkg install drm-kmod, because you need a graphics driver

I have an AMD RX550 video card (cheapest I could get this year) in the machine, so then to set that graphics driver:

• sysrc kld_list=amdgpu

Make sure DBus will run as a system service:

• sysrc dbus_enable=yes

Create a regular user and add them to the video group.

Then reboot.

KDE Plasma6 X11 by Hand

At this point, the system has KDE Plasma6 installed, but it won’t come up (the login manager isn’t enabled, for instance) so we need some convenience things from The Before Times to do a little testing. (Installed as “automatic” so they are easy to remove later).

• pkg install -A xinit xterm

As a normal user, create ~/.xinitrc and put this in it:

#! /bin/sh exec /usr/local/bin/startplasma-x11

Then make it executable and start X11 (like it’s 1994):

• chmod 755 .xinitrc
• startx

Voila. KDE Plasma6. Note that, in this state, there are no applications besides xterm and KInfoCenter (and a handful of other KDE Plasma internal things). There’s no configuration applied to the system. The window manager does not honor alt-tab or alt-F4. Use ctrl-Q to quit KInfoCenter.

But X11 is soooo passé. Let’s move on to the Future!

Any Wayland by Hand

First, let’s install some convenience things that will help in debugging.

• pkg install -A river foot

Like I wrote 3 years ago about river, it just works. You’ll need the example configuration file so that super-shift-E exits the compositor and window manager. super-shift-enter gets you a terminal window.

I started river as a regular user with

• ck-launch-session river

KDE Plasma6 Wayland by Hand

Yeah, right.

So there was a brief time in 2021 that it worked. Since then, not so much – certainly not for me, not from the regular ports tree. So this post is a start of “ok, let’s give it another shot”.

I know that Wayland can work on FreeBSD with hardware rendering – that’s why that river section is there.

Here is a very short script that can launch KDE Plasma6 with software rendering. The resulting desktop experience is rather slow.

#! /bin/sh export KWIN_COMPOSE=Q exec /usr/local/bin/ck-launch-session \ /usr/local/lib/libexec/plasma-dbus-run-session-if-needed \ /usr/local/bin/startplasma-wayland

So the TODO part of this post is: figure out why opening dri/card0 fails when kwin_wayland is not using the software renderer.

KDE Applications5 and KDE Plasma6

Yeah, right.

Most (maybe even all) of the KDE Applications – for instance, konsole, or kmail – are still KDE Frameworks 5 based. Unfortunately, there are KDE Frameworks that have file-collisions between versions 5 and 6. As an example, package kf6-baloo and kf5-baloo both want to install a libbalooplugin.so. I mentioned the co-installability problem a half-year ago, but we haven’t fixed it since.

Edit 2024-10-08: this is entirely a packaging problem, where we could install the things to separate prefixes. That hasn’t happened, because of a lack of person-time to actually do it (and test it).

On this front I think we’re at a chicken-and-egg place: we would like to switch wholesale to newer applications releases and just drop the existing KDE Applications 5, but are so bogged down with Other Stuff that it’s not happening. On the Linux side of things KDE Applications 6 are doing fine.

All that said, there’s the KDE-FreeBSD ports development fork with a branch where newer KDE-FreeBSD packages are prepared. That already has a KF6-based KMail. So the TODO part of this section is: I need to double-check what’s holding that up.

Takeaways

• for a nice KDE Plasma6 experience on a BSD, why not try OpenBSD?
• KDE-FreeBSD has a nasty amount of hardware-testing to do.
• The future is here, just not in the ports tree.

At work-work the system uses, for historical reasons, a lot of SystemV shared memory. The SysV shared memory API has C functions like shmat(2). There is also a different shared memory API, POSIX shared memory, which has functions like shm_open(3). For reasons, on some work-work systems we’re constrained to Python 3.7 and no additional libraries. I wanted to mess with the shared memory on such a system, from Python for convenience, so I wrote some very simple wrappers. Here’s a recap.

As usual, corrections are welcome, or tips (by email). I write these notes as much for future me as anyone else.

Here is the core of the story (I have also added this to my personal GitHub repository, which I won’t link because it’s not future-proof storage).

import ctypes lib = ctypes.cdll.LoadLibrary(None) shmget = lib.shmget shmget.argtypes = [ctypes.c_int, ctypes.c_size_t, ctypes.c_int] shmget.restype = ctypes.c_int shmget.__doc__ = """See shmget(2)"""

This works on FreeBSD, where SysV shmem is in the core libraries. On Linux, I think you need to call LoadLibrary("librt"). Anyway, wrapping the library-loading to be safe isn’t the point here.

Once ctypes has loaded a library, you can extract function pointers from the library. By adding annotations, you can give the Python function the same prototype as the C manpage for shmget.

Note that the manpage points to some special flag values. For those, you need to dig into the C headers. On FreeBSD, the special value IPC_PRIVATE is equal to 0, so that’s easy enough to write in Python. The following snippet is then sufficient to create a shared memory segment (one that is 1024 bytes large and world-readable) and print out its ID. The returned value is -1 on error.

print(shmget(0, 1024, 0o644))

The ID can be cross-checked with command ipcs -m (it’s installed by default on FreeBSD and in my KDE Neon machine, so seems like a common tool). To get rid of the segment, ipcrm -m <id> does the trick.

Similar wrappers are there for shmat, shmdt and shmctl – but those wrangle void * in C, and how does that work with Python?

The void pointer

CTypes has a c_void_p type, which can be created from None (a null pointer, seems reasonable) and returned from C functions. It can cast to-and-fro (in classic C style, the thing in memory is what I say is in memory) to other pointer types, and without a typed-pointer type at the machine level that just works (but don’t ask me how).

So the C function int shmctl(int shmid, int cmd, struct shmid_ds *buf) gets these types in Python: shmctl.argtypes = [ctypes.c_int, ctypes.c_int, ctypes.c_void_p], which presents the struct-pointer as a void-pointer.

The function void *shmat(int shmid, const void *addr, int flag) works similarly. When calling it, unless you have specific address needs, parameter addr can be nullptr (er .. ok, this is C, so NULL and in Python None). The pointer it returns is where the shared memory is attached.

Actually doing something with a void * takes work in C, it also takes work in Python with ctypes. You can cast to an int * for instance, with iaddr = ctypes.cast(addr, ctypes.POINTER(ctypes.c_int)) (a cast to char * is also readily available).

A special case is when you need to provide a void * to some C function. Where do they come from? In C you would just declare a (character) buffer of some size and pass it in. In Python, ctypes.create_string_buffer() does the job. Give it a size and get a memory-managed buffer.

Wrangling shared-memory segment destruction

There’s shmget() to create a segment, shmat() to attach (map it into memory of a process) to it, shmdt() to detach from a segment, but destroying a shared-memory segment does not have a simple C call to do it. There is shmctl() which does special-control-actions on a shared-memory segement, and destruction is one of them.

I ended up writing this little wrapper.

def shmrm(shmid : int) -> int: return shmctl(shmid, 0, None) Sending messages

As an experiment, I wrote a program that can create, read, write and destroy a shared-memory segment. By writing (from one invocation) and then reading (from another invocation) I can “send” messages from the past! Or to the future! It is nearly as convenient as writing the messages to a file.

Here’s the write function. It attaches the shared-memory segment and then writes a Pascal-style string to that memory (Pascal-style in the sense of “starts with a length, followed by the actual data, no NUL-termination”). For bloggy purposes I have removed error-handling.

def write(shmid : int, v : str): addr = shmat(shmid, ctypes.c_void_p(None), 0) iaddr = ctypes.cast(addr, ctypes.POINTER(ctypes.c_int)) caddr = ctypes.cast(addr, ctypes.POINTER(ctypes.c_char)) ustring = v.encode("utf-8") iaddr[0] = len(ustring) for i in range(len(ustring)): caddr[4+i] = ustring[i] return shmdt(addr)

Here, shmat() returns a void * and I cast that to two typed pointers to the segment. I haven’t figured out how to do pointer arithmetic, so on the assumption there are 32-bit integers, the integer goes first and then the message goes starting at byte (char) number 4.

Takeaways

CTypes is really cool! It makes wrangling C APIs in Python .. well, let’s call it “acceptable”.

Starting with Python 3.8, everything I’ve written above is unnecessary because there is a good shared-memory abstraction in the standard Python library, but for my work-work purposes in a very restricted environment, this particular tool has turned out to be really useful.

Welcome to the September 2024 report from the Reproducible Builds project!

Our reports attempt to outline what we’ve been up to over the past month, highlighting news items from elsewhere in tech where they are related. As ever, if you are interested in contributing to the project, please visit our Contribute page on our website.

Table of contents:

1. New binsider tool to analyse ELF binaries
2. Unreproducibility of GHC Haskell compiler “95% fixed”
3. Mailing list summary
4. Towards a 100% bit-for-bit reproducible OS…
5. Two new reproducibility-related academic papers
6. Distribution work
7. diffoscope
8. Other software development
9. Android toolchain core count issue reported
10. New Gradle plugin for reproducibility
11. Website updates
12. Upstream patches
13. Reproducibility testing framework

New binsider tool to analyse ELF binaries

Reproducible Builds developer Orhun Parmaksız has announced a fantastic new tool to analyse the contents of ELF binaries. According to the project’s README page:

Binsider can perform static and dynamic analysis, inspect strings, examine linked libraries, and perform hexdumps, all within a user-friendly terminal user interface!

More information about Binsider’s features and how it works can be found within Binsider’s documentation pages.

Unreproducibility of GHC Haskell compiler “95% fixed”

A seven-year-old bug about the nondeterminism of object code generated by the Glasgow Haskell Compiler (GHC) received a recent update, consisting of Rodrigo Mesquita noting that the issue is:

95% fixed by [merge request] !12680 when -fobject-determinism is enabled. […]

The linked merge request has since been merged, and Rodrigo goes on to say that:

After that patch is merged, there are some rarer bugs in both interface file determinism (eg. #25170) and in object determinism (eg. #25269) that need to be taken care of, but the great majority of the work needed to get there should have been merged already. When merged, I think we should close this one in favour of the more specific determinism issues like the two linked above.

Mailing list summary

On our mailing list this month:

• Fay Stegerman let everyone know that she started a thread on the Fediverse about the problems caused by unreproducible zlib/deflate compression in .zip and .apk files and later followed up with the results of her subsequent investigation.

• Long-time developer kpcyrd wrote that “there has been a recent public discussion on the Arch Linux GitLab [instance] about the challenges and possible opportunities for making the Linux kernel package reproducible”, all relating to the CONFIG_MODULE_SIG flag. […]

• Bernhard M. Wiedemann followed-up to an in-person conversation at our recent Hamburg 2024 summit on the potential presence for Reproducible Builds in recognised standards. […]

• Fay Stegerman also wrote about her worry about the “possible repercussions for RB tooling of Debian migrating from zlib to zlib-ng” as reproducibility requires identical compressed data streams. […]

• Martin Monperrus wrote the list announcing the latest release of maven-lockfile that is designed aid “building Maven projects with integrity”. […]

• Lastly, Bernhard M. Wiedemann wrote about potential role of reproducible builds in combatting silent data corruption, as detailed in a recent Tweet and scholarly paper on faulty CPU cores. […]

Towards a 100% bit-for-bit reproducible OS…

Bernhard M. Wiedemann began writing on journey towards a 100% bit-for-bit reproducible operating system on the openSUSE wiki:

This is a report of Part 1 of my journey: building 100% bit-reproducible packages for every package that makes up [openSUSE’s] minimalVM image. This target was chosen as the smallest useful result/artifact. The larger package-sets get, the more disk-space and build-power is required to build/verify all of them.

This work was sponsored by NLnet’s NGI Zero fund.

Two new reproducibility-related academic papers

Marvin Strangfeld published his bachelor thesis, “Reproducibility of Computational Environments for Software Development” from RWTH Aachen University. The author offers a more precise theoretical definition of computational environments compared to previous definitions, which can be applied to describe real-world computational environments. Additionally, Marvin provide a definition of reproducibility in computational environments, enabling discussions about the extent to which an environment can be made reproducible. The thesis is available to browse or download in PDF format.

In addition, Shenyu Zheng, Bram Adams and Ahmed E. Hassan of Queen’s University, ON, Canada have published an article on “hermeticity” in Bazel-based build systems:

A hermetic build system manages its own build dependencies, isolated from the host file system, thereby securing the build process. Although, in recent years, new artifact-based build technologies like Bazel offer build hermeticity as a core functionality, no empirical study has evaluated how effectively these new build technologies achieve build hermeticity. This paper studies 2,439 non-hermetic build dependency packages of 70 Bazel-using open-source projects by analyzing 150 million Linux system file calls collected in their build processes. We found that none of the studied projects has a completely hermetic build process, largely due to the use of non-hermetic top-level toolchains. […]

Distribution work

In Debian this month, 14 reviews of Debian packages were added, 12 were updated and 20 were removed, all adding to our knowledge about identified issues. A number of issue types were updated as well. […][…]

In addition, Holger opened 4 bugs against the debrebuild component of the devscripts suite of tools. In particular:

• #1081047: Fails to download .dsc file.
• #1081048: Does not work with a proxy.
• #1081050: Fails to create a debrebuild.tar.
• #1081839: Fails with E: mmdebstrap failed to run error.

Last month, an issue was filed to update the Salsa CI pipeline (used by 1,000s of Debian packages) to no longer test for reproducibility with reprotest’s build_path variation. Holger Levsen provided a rationale for this change in the issue, which has already been made to the tests being performed by tests.reproducible-builds.org. This month, this issue was closed by Santiago R. R., nicely explaining that build path variation is no longer the default, and, if desired, how developers may enable it again.

In openSUSE news, Bernhard M. Wiedemann published another report for that distribution.

diffoscope

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made the following changes, including preparing and uploading version 278 to Debian:

• New features:

  • Add a helpful contextual message to the output if comparing Debian .orig tarballs within .dsc files without the ability to “fuzzy-match” away the leading directory.  […]

• Bug fixes:

  • Drop removal of calculated os.path.basename from GNU readelf output. […]
  • Correctly invert “X% similar” value and do not emit “100% similar”. […]

• Misc:

  • Temporarily remove procyon-decompiler from Build-Depends as it was removed from testing (via #1057532). (#1082636)
  • Update copyright years. […]

For trydiffoscope, the command-line client for the web-based version of diffoscope, Chris Lamb also:

• Added an explicit python3-setuptools dependency. (#1080825)
• Bumped the Standards-Version to 4.7.0. […]

Other software development

disorderfs is our FUSE-based filesystem that deliberately introduces non-determinism into system calls to reliably flush out reproducibility issues. This month, version 0.5.11-4 was uploaded to Debian unstable by Holger Levsen making the following changes:

• Replace build-dependency on the obsolete pkg-config package with one on pkgconf, following a Lintian check. […]
• Bump Standards-Version field to 4.7.0, with no related changes needed. […]

In addition, reprotest is our tool for building the same source code twice in different environments and then checking the binaries produced by each build for any differences. This month, version 0.7.28 was uploaded to Debian unstable by Holger Levsen including a change by Jelle van der Waa to move away from the pipes Python module to shlex, as the former will be removed in Python version 3.13 […].

Android toolchain core count issue reported

Fay Stegerman reported an issue with the Android toolchain where a part of the build system generates a different classes.dex file (and thus a different .apk) depending on the number of cores available during the build, thereby breaking Reproducible Builds:

We’ve rebuilt [tag v3.6.1] multiple times (each time in a fresh container): with 2, 4, 6, 8, and 16 cores available, respectively:

• With 2 and 4 cores we always get an unsigned APK with SHA-256 14763d682c9286ef….
• With 6, 8, and 16 cores we get an unsigned APK with SHA-256 35324ba4c492760… instead.

New Gradle plugin for reproducibility

A new plugin for the Gradle build tool for Java has been released. This easily-enabled plugin results in:

reproducibility settings [being] applied to some of Gradle’s built-in tasks that should really be the default. Compatible with Java 8 and Gradle 8.3 or later.

Website updates

There were a rather substantial number of improvements made to our website this month, including:

• Chris Lamb:

  • Attempt to use GitLab CI to ‘artifact’ the website; hopefully useful for testing branches. […]
  • Correct the linting rule whilst building the website. […]
  • Make a number of small changes to Kees’ post written by Vagrant. […][…][…]
  • Add the Civil Infrastructure Platform to the Projects page. […]
  • Miscellaneous administration of misfiled images. […][…]

• Evangelos Tzaras made a huge number of changes related to the recent Hamburg 2024 summit […][…][…][…][…] as well as proposed an infographic about which question Reproducible Builds is trying to answer.

• Holger Levsen added his two presentations (Reproducible Builds: The First Eleven Years and Preserving *other* build artifacts) to the website. […]

• Jelle van der Waa completely modernised the System Images documentation, noting that “a lot has changed since 2017(!); ext4, erofs and FAT filesystems can now be made reproducible”. […]

• Developer RyanSquared replaced the continuous integration test link for Arch Linux on our Projects page with an external instance […][…] as well as updated the documentation to reflect the dependencies required to build the website […].

• Vagrant Cascadian pushed a lengthy interview with Linux developer Kees Cook. […][…][…][…]

Upstream patches

The Reproducible Builds project detects, dissects and attempts to fix as many currently-unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:

  • agama-integration-tests (random)
  • contrast (FTBFS-nocheck)
  • cpython (FTBFS-2038)
  • crash (parallelism, race)
  • ghostscript (toolchain date)
  • glycin-loaders (FTBFS -j1)
  • gstreamer-plugins-rs (date, other)
  • kernel-doc/Sphinx (toolchain bug, parallelism/race)
  • kernel (parallelism in BTF)
  • libcamera (random key)
  • libgtop (uname -r)
  • libsamplerate (random temporary directory)
  • lua-luarepl (FTBFS)
  • meson (toolchain)
  • netty (modification time in .a)
  • nvidia-persistenced (date)
  • nvidia-xconfig (date-related issue)
  • obs-build (build-tooling corruption)
  • perl (Perl records kernel version)
  • pinentry (make efl droppable)
  • python-PyGithub (FTBFS 2024-11-25)
  • python-Sphinx (parallelism/race)
  • python-chroma-hnswlib (CPU)
  • python-libcst
  • python-pygraphviz (random timing)
  • python312 (.pyc embeds modification time)
  • python312 (drop .pyc from documentation time)
  • scap-security-guide (date)
  • seahorse (parallelism)
  • subversion (minor Java .jar modification times)
  • xen/acpica (date-related issue in toolchain)
  • xmvn (random)
• Fridrich Strba:
  • ant (jar mtime)
  • xmvn (various fixes)

• Chris Lamb:

  • #1082702 filed against magic-wormhole-transit-relay.
  • #1082706 filed against python-sphobjinv.
  • #1082707 filed against lomiri-content-hub.
  • #1082796 filed against python-mt-940.
  • #1082806 filed against tree-puzzle.
  • #1083053 filed against muon-meson.

• James Addison:

  • guake (fix a parallelism/timing-race build bug)
  • sphobjinv (duplicates fix from Debian bug #1082706)

Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework running primarily at tests.reproducible-builds.org in order to check packages and other artifacts for reproducibility. In September, a number of changes were made by Holger Levsen, including:

• Debian-related changes:

  • Upgrade the osuosl4 node to Debian trixie in anticipation of running debrebuild and rebuilderd there. […][…][…]
  • Temporarily mark the osuosl4 node as offline due to ongoing xfs_repair filesystem maintenance. […][…]
  • Do not warn about (very old) broken nodes. […]
  • Add the risc64 architecture to the multiarch version skew tests for Debian trixie and sid. […][…][…]
  • Mark the virt{32,64}b nodes as down. […]

• Misc changes:

  • Add support for powercycling OpenStack instances. […]
  • Update the fail2ban to ban hosts for 4 weeks in total […][…] and take care to never ban our own Jenkins instance. […]

In addition, Vagrant Cascadian recorded a disk failure for the virt32b and virt64b nodes […], performed some maintenance of the cbxi4a node […][…] and marked most armhf architecture systems as being back online.

Finally, If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.

• Mastodon: @reproducible_builds@fosstodon.org

• Mailing list: rb-general@lists.reproducible-builds.org

• Twitter: @ReproBuilds

Today we are talking about Creating Recipes, What Recipes already exist, and helpful tips and tricks with guest Jim Birch. We’ll also cover Features as our module of the week.

For show notes visit: https://www.talkingDrupal.com/470

Topics

• What are recipes
• How do you recommend someone get started writing recipes
• Where can people find recipes
• Can you include sub recipes
• How should you test recipes
• Any tools that make writing recipes easier
• What recipes are needed that do not exist
• How can people move recipes forward

Resources

• Recipe Author Guide
• Drupal Core Recipes
• Preconditions for recipes
• Drupal Recipes Cookbook
• Recipes Packagist
• Recipe type
• Phase 2 roadmap
• Umami profile recipes
• Minimal profile recipes

Guests

Jim Birch - linkedin.com/in/jimbirch thejimbirch

Hosts

Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Aubrey Sambor - star-shaped.org starshaped

MOTW Correspondent

Martin Anderson-Clutz - mandclu.com mandclu

• Brief description:
  • Have you ever wanted an admin UI to manage sets of configuration, to version and share across Drupal sites? There’s a module for that.
• Module name/project name:
  • Features
• Brief history
  • How old: created in Mar 2009 by yhahn, though recent releases are by Dave Reid
  • Versions available: 7.x-2.15 and 8.x-3.14, the latter of which works with Drupal 9.4 and 10
• Maintainership
  • Minimally maintained
  • Security coverage
  • Test coverage
  • Documentation: Has a documentation guide and probably hundreds if not thousands of of tutorials available
  • Number of open issues: 610 open issues, 54 of which are bugs against the 8.x branch
• Usage stats:
  • Almost 117,000 sites, though the majority are using the D7 version
• Module features and usage
  • Many listeners will remember Features as the de facto solution for configuration management in Drupal 7 and earlier
  • As the name implies, it was really intended to share common capabilities across different Drupal sites
  • Unlike recipes, Features can have version numbers, because there is a path to sync configuration updates across sites using a Feature, though this is where a lot of teams found Features could be complex to use
  • We did previously cover Features as MOTW all the way back in episode #147, but I thought it was relevant to today’s discussion because of the way it provides a UI for organizing and exporting specific sets of configuration
  • There is an open issue for Features to directly export recipes, because it already does a lot of the time-consuming work of collecting together necessary config files, including dependencies
  • Even its current state, it could be a time saver for anyone wanting to start creating their own recipes

Python 3.13 comes with a brand new REPL and improvements to virtual environments and the Python debugger.

Table of contents

1. Important but not my favorite
2. The New Python REPL
3. Git-friendly virtual environments
4. Python Debugger improvements
5. Try out Python 3.13

Important but not my favorite

First, I'd like to note that I'm not going to talk about the experimental free-threaded mode, the experimental just-in-time compilation option, or many other features that don't affect most Python developers today.

Instead, let's focus on some of the more fun things.

The New Python REPL

My favorite feature by far …

Read the full article: https://www.pythonmorsels.com/python-313-whats-new/

Maybe, you want to experiment small stuffs in wish (the tcl/tk) interpreter because of a post claiming that direct python tcl/tk is simpler in some simple cases than tkinter.

As a convinced tkinter/FreeSimpleGUI user, I see this as an extreme claim that requires solid evidences.

When all is said and done, wish interpreter is not interactive, and for testing simple stuff it can get annoying very fast. Thus, it would be nice to add readline to the interface.

So here is a less than 100 line of code exercice of soing exactly so while having fun with : readline and multiprocessing (I would have taken multithreading if threads were easy to terminate).

Readline I quote The readline module defines a number of functions to facilitate completion and reading/writing of history files from the Python interpreter. Basically, it adds arrow navigation in history, back search with Ctrl+R, Ctrl+K for cuting on the right, Ctrl+Y for yanking ... all the facilities of interaction you have in bash or ipython for instance.

We are gonna use multiprocessing because tcl/tl is event oriented, hence, asynchronuous hence, we may have string coming from the tcl stdout while we do nothing and we would like to print them.

We also introduce like in ipython some magic prefixed with # (comment in tcl) like #? for the help. A session should look like this : # pack [ button .c -text that -command { puts "hello" } ] # tcl output> hello # here we pressed the button "that" tcl output> hello # here we pressed the button "that" # set name 32 # puts $name tcl output> 32 # #? #l print current recorded session #? print current help #! calls python code like #!save(name="temp") which saves the current session in current dir in "temp" file bye exit quit quit the current session # #l pack [ button .c -text that -command { puts "hello" } ] set name 32 puts $name # #!save("my_test.tcl") # quit The code in itself is fairly easy to read the only catch is that wish accepts multiline input. I can't because I don't know how to parse tcl. As a result I « eval in tcl » every line to know if there is an error and ask politely tcl to do the job of signaling the error with a « catch/error » (the equivalent of python try + raise an exception). #!/usr/bin/env python3 # -*- coding: utf8 -*- from subprocess import Popen, PIPE, STDOUT from multiprocessing import Process import sys, os import atexit import os import readline from select import select from time import sleep ### interactive session with history with readline histfile = os.path.join(os.path.expanduser("~"), ".wish_history") try: readline.read_history_file(histfile) # default history len is -1 (infinite), which may grow unruly readline.set_history_length(-1) except FileNotFoundError: pass ### saving history at the end of the session atexit.register(readline.write_history_file, histfile) ### opening wish wish = Popen(['wish'], stdin=PIPE, stdout=PIPE, stderr=PIPE, bufsize=-1, ) os.set_blocking(wish.stdout.fileno(), False) os.set_blocking(wish.stderr.fileno(), False) os.set_blocking(wish.stdin.fileno(), False) def puts(s): out = f"""set code [ catch {{ {s} }} p ] if {{$code}} {{ error $p }} """ select([], [wish.stdin], []) wish.stdin.write(out.encode()) def gets(): while True: wish.stdout.flush() tin = wish.stdout.read() if tin: print("\ntcl output> " + tin.decode()) sleep(.1) def save(fn="temp"): with open(fn,"wt") as f: f.write(session) session=s="" def load(fn="temp"): global session with open(fn, "rt") as f: while l:= f.readline(): session+=l + "\n" puts(l) # async io in tcl requires a background process to read the input t =Process(target=gets, arwish=()) t.start() while True: s = input("# ") if s in { "bye", "quit", "exit" }: t.terminate() wish.stdin.write("destroy .".encode()) break elif s == "#l": print(session) elif s == "#?": print(""" #l print current recorded session #? print current help #! calls python code like #!save(name="temp") which saves the current session in current dir in "temp" file #!load(name="temp") which load the session stored in current dir in "temp" file bye exit quit quit the current session """ ) continue elif s.startswith("#!"): print(eval(s[2:])) continue else: puts(s) if err:=wish.stderr.readline(): sys.stderr.write(err.decode()) else: if s and not s.startswith("#"): session += s + "\n" This code is available on pypi as iwish (interactive wish) and the git link is in the README.