Feeds
Seth Michael Larson: PyCon US 2024 as Security Developer-in-Residence
Published 2024-06-13 by Seth Larson
Reading time: minutes
This was my first PyCon US as the Security Developer-in-Residence. I accepted the offer shortly after PyCon US 2023, where Deb Nicholson provided an exciting cliffhanger that the PSF was close to making a hiring decision for the role during the PSF update.
The timing of my hiring to coincide with PyCon US is quite the unique opportunity, it means I get to share all the things we've been able to accomplish together in the past year all-together in person. This also means I get to share the road ahead and learn from many people I don't get to see on a day-to-day, like maintainers of projects large and small, software users at organizations and companies, and users whose work extends beyond software like scientists and industrial control systems operators.
Overall I think this yearly cadence couldn't be better, I'm already looking forward to what I'll be able to share at PyCon US 2025 back in Pittsburgh. As I've said in the past, this role wouldn't be effective without the support of the community, so I want to thank you all for the parts you've played in me being successful.
State of Python Supply Chain Security with Alpha OmegaAlpha Omega is a sponsor of the Python Software Foundation, specifically for my role! Alpha Omega's goal is to "improve global software supply chain security by partnering with open source". I presented alongside Alpha Omega's co-founder Michael Winser. The slides for our presentation are available online.
Alpha Omega uses a two-pronged approach, and it's right there in the name: Alpha and Omega:
- Alpha: Improve the security posture of the most critical projects through staffing.
- Omega: Automated security analysis, metrics, and remediation for wider range of projects.
My role is full-time staffing, so is in the "Alpha" bucket of funding.
Speaking for Alpha Omega on the state of Python supply chain security
- Python Software Foundation as a CVE Numbering Authority (and how we helped Linux, curl, and others with a guide).
- Joining Python Security Response Team and working on process
- Working with Python release managers to move the release process to GitHub Actions as an isolated build environment.
- Build reproducibility for the Python release process.
- Software Bill-of-Materials for Python release artifacts.
- Coordinated cross-ecosystem response to libwebp and xz-utils vulnerabilities.
- Tons of community work: PEP reviewer, talks, blogs, and more.
- Enabling Build Provenance and Software Bill-of-Materials for Python packages.
- Adoption of security features, hardening, and best practices for Python packages.
- Special focus on Python packaging tools and workflows.
I finished the talk by describing the unique nature of this role as being flexible and how that's a boon for the rapidly changing space of software supply chain security. This role has a fairly wide scoping, which means that when things come up (like xz-utils and novel social-engineering techniques) it's in this role's scope to think about how to respond for the Python ecosystem.
“Vuln Together”, an open space on vulnerability management“Got vulns? Let's talk!” -- I co-hosted this open space with GitHub Security champion and CVE board member Madison Oliver. This open space focused on the soft-side of vulnerability management for open source projects: people!
Managing vulnerabilities for open source projects is a non-trivial and effort intensive process, because maintainers need to create and publish a security policy, accept private vulnerability reports, and then know what reporters need, request a CVE ID, know how to estimate severity and write an advisory text, and then how to publish an advisory alongside fixed versions. Phew!
Madison and I pictured holding the open space card for “Vuln Together”
The open space was a forum to discuss difficulties with managing vulnerabilities, answering questions, and pointing folks at resources. We shared recommendations like how maintainers can easily request CVEs from GitHub, enable Private Vulnerability Reporting on GitHub and Confidential Issues on GitLab to make reporting easier, and showed resources like the Guide for Vulnerability Disclosure Process from the OpenSSF Vulnerability Disclosures working group.
The open space was well attended!
My bingo space for "xz" was almost immediately filled at PyCon US when I attended the Python Language Summit as the blogger. Release manager and core developer Pablo Galindo Salgado spoke about the Python contribution security model in "the wake of xz-utils backdoor".
The complete language summit blog posts are coming soon to the Python Software Foundation blog (I should know, I authored them!) so if you're interested in this topic you can stay tuned over there. I won't spoil any of the contents of the blog post here, but it was a great discussion all around. Lots of interesting ideas and thoughts from core developers that'll help with figuring process improvements that work for CPython.
Meeting all the peopleMike Fiedler and I were on the stage briefly after Kate Chapman's keynote for a "Meet the Python Software Foundation Security Engineers" segment where we went over our plans for the upcoming year and recommended folks to follow the PSF and PyPI blogs for future updates.
Of course, I got to talk to so many people, too many to name individually. I chatted with the upcoming Release Manager for CPython, Hugo van Kemenade, on some ideas to further improve the CPython release process, specifically around signatures. I chatted with folks from specific sub-ecosystems like Jannis Leidel from Conda and Jazzband, David Lord from Flask, and Jarek Potiuk from Airflow.
Also I handed out stickers! I went through over a hundred, it brought me a lot of joy seeing how much folks liked the derpy snake knight design.
... and beyond!PyCon US 2024 is the start of conferences this year for me. Shortly after PyCon US it was announced that I will be keynoting PyCon Taiwan in September and also speaking at All Things Open 2024 in Raleigh, North Carolina. If you're attending either of these events get in contact with me (and I promise to bring stickers).
That's all for this post! 👋 If you're interested in more you can read the previous report.
Thanks for reading! ♡ Did you find this article helpful and want more content like it?
Get notified of new posts by subscribing to the RSS feed or the email newsletter.
This work is licensed under CC BY-SA 4.0
KDE Gear 24.05.1
Over 180 individual programs plus dozens of programmer libraries and feature plugins are released simultaneously as part of KDE Gear.
Today they all get new bugfix source releases with updated translations, including:
- elisa: Fix DBus service activation (Commit, fixes bug #487905)
- kcalc: Handle certain expressions correctly (Commit, fixes bug #487614)
- kcolorpicker: Allow picking the screen color with Wayland (Commit, fixes bug #479406)
Distro and app store packagers should update their application packages.
- 24.05 release notes for information on tarballs and known issues.
- Package download wiki page
- 24.05.1 source info page
- 24.05.1 full changelog
Serving up git trees
For FreeBSD packaging (writing “ports”, as it is called), I usually just build on my host system – either in the actual host, or with poudriere – and call it a day. If it compiles on amd64 then it should work everywhere, right? (After all, “all the world’s a VAX”)
No exciting software engineering here, BTW, just a “here’s how I set up some infrastructure to help me build ports locally in various environments.”
I have recently finally fixed the build of Sayonara, an audio player that I quite like, on amd64. It still fails on i386, though, because of a mismatch between GStreamer integer sizes and host integer sizes, where the code on the Sayonara size assumes they are the same. Gladly, 32-bit code is rarely built and tested these days, but it keeps spamming me with build failures. So I set out to fix it with an i386 VM running FreeBSD 13.2 i386.
What I wanted to do was use the ports tree from my host – it is a git checkout of the upstream FreeBSD ports tree – and clone it into my VM. Then I can do packaging fixes on the host, commit them locally, update the checkout in the VM and test the packaging fixes there, too. I want to keep those commits local so that I don’t pollute the global upstream repo with a ton of “maybe this works” kind of commits. Once I’m done, I can squash it down to one commit and push, fixing the build for once and for all (ha!).
So, serve up a git checkout to some remote. It’s remarkably simple (as long as StackOverflow answers still show up in web search; to guard against future enshittification, these notes are my own copy):
ports$ git daemon --reuseaddr --base-path=. --export-all --verboseIn my ports/ directory, run the simple and insecure git server that comes with it. Export the ports tree. On the receiving (that is, in the VM) side, clone with git:// protocol.
# git clone git://host/ /usr/portsAnd voila! That enables a quick turn-around for port fifxes that need to be done on a different machine from where I have my usual ports tree checkout (and my GPG keys, and …).
GNU Taler news: Cashless to e-Cash
Redfin Solutions: DDEV, You're Still the One!
C.J. Collier: IPv6 Multicast support on libvirt guests using macvtap
Wim Leers: XB week 4: annotated data model test
My Monday started with a pleasant surprise that is only possible in cross-timezone collaborations: Ted “tedbow” 1 had made a huge leap forward on #3450957: Prevent modules from being uninstalled if they provide field types used in an Experience Builder (XB) field, where he’s working on the first aspect where XB’s JSON blobs in the database must be queried.
Drupal 11 requires versions of databases that support JSON querying. Ted rather quickly discovered that JSON querying support is not consistent across different databases … but fortunately Brad “bradjones1 is working on #3343634 to add explicit JSON support to Drupal’s database abstraction layer, which is one of the endeavors sponsored by Pitch-Burgh! So: database nerds, unite!
(That means reviewing Brad’s work in that issue is yet another way of contributing to XB — in addition to helping Single Directory Components (SDC) move forward.)
To ensure we are aware of what works on which database from day one, I updated XB’s CI pipeline to run the test suite against MariaDB+MySQL+PostgreSQL+SQLite.
Missed a prior week? See all posts tagged Experience Builder.
Goal: make it possible to follow high-level progress by reading ~5 minutes/week. I hope this empowers more people to contribute when their unique skills can best be put to use!
For more detail, join the #experience-builder Slack channel. Check out the pinned items at the top!
Later on Monday, Ben “bnjmnm” and Jesse landed an MR that starts to connect the UI that you saw two weeks ago to the back end, and brought it alive: there’s now a tree view to provide a sense of place on the canvas, and much more:
Try it yourself locally if you like, but there’s not much you can do yet.
Install the 0.x branch — the “Experience Builder PoC” toolbar item takes you there!
A day later, the first set of unit tests landed: those for StructuredDataPropExpressions (these express where to retrieve values mapped for SDC props).
On Thursday, Harumi “hooromoo”’s MR to provide the foundations for undo/redo support landed, after the most enthusiastic approval I think I’ve ever seen Lee “larowlan” give! :D
That same day, his colleague Griffyn “griffynh” ran the second asynchronous XB meeting in Drupal Slack (archive on d.o: #3453097), this time with 21 participants of the ~250 people in the #experience-builder channel (~100 more compared to 2 weeks ago).
The most heavily debated topics were the data model/persistence layer and Cypress vs vitest for tests.
By Friday, I managed to finish up the work that I already shared a video of a week prior (see week 3), where a PropSource (which itself uses a PropExpression) can be chosen for each component prop and edited using Drupal’s existing Field Widgets. Crucially, it now includes a fully annotated end-to-end test that verifies the demo in detail:
- an article using an XB field can be created
- expected data is stored for the XB field, with a handful of placed components, and all prop source types: static, dynamic and adapter
- each of these prop sources can be evaluated/resolved to a concrete value to pass into the corresponding SDC prop
- each of the placed components renders into the expected HTML
In other words: it’s making the concrete implementation of #3440578: JSON-based data storage proposal for component-based page building tangible and debuggable in a way that required reading dozens of files previously! 2
Thanks to Ted & Lauri for reviewing this!
-
Forever infamous for his shining sunglasses moment on stage during DrupalCon Pittsburgh — captured it in exquisite detail by Mike “mherchel”! ↩︎
-
Hence it’ll serve as a canary for the data model/storage back-end infrastructure in the short–medium term. ↩︎
Real Python: Python Mappings: A Comprehensive Guide
One of the main data structures you learn about early in your Python learning journey is the dictionary. Dictionaries are the most common and well-known of Python’s mappings. However, there are other mappings in Python’s standard library and third-party modules. Mappings share common characteristics, and understanding these shared traits will help you use them more effectively.
In this tutorial, you’ll learn about:
- Basic characteristics of a mapping
- Operations that are common to most mappings
- Abstract base classes Mapping and MutableMapping
- User-defined mutable and immutable mappings and how to create them
This tutorial assumes that you’re familiar with Python’s built-in data types, especially dictionaries, and with the basics of object-oriented programming.
Get Your Code: Click here to download the free sample code that you’ll use to learn about mappings in Python.
Take the Quiz: Test your knowledge with our interactive “Python Mappings” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Python MappingsIn this quiz, you'll test your understanding of the basic characteristics and operations of Python mappings. By working through this quiz, you'll revisit the key concepts and techniques of creating a custom mapping.
Understanding the Main Characteristics of Python MappingsA mapping is a collection that allows you to look up a key and retrieve its value. The keys in mappings can be objects of a broad range of types. However, in most mappings, there are object types that can’t be used as keys, as you’ll learn later in this tutorial.
The previous paragraph described mappings as collections. A collection is an iterable container that has a defined size. However, mappings also have additional features. You’ll explore each of these mapping characteristics with examples from Python’s main mapping types.
The feature that’s most characteristic of mappings is the ability to retrieve a value using a key. You can use a dictionary to demonstrate this operation:
Python >>> points = { ... "Denise": 3, ... "Igor": 2, ... "Sarah": 3, ... "Trevor": 1, ... } >>> points["Sarah"] 3 >>> points["Matt"] Traceback (most recent call last): ... KeyError: 'Matt' Copied!The dictionary points contains four items, each with a key and a value. You can use the key within the square brackets to fetch the value associated with that key. However, if the key doesn’t exist in the dictionary, the code raises a KeyError.
You can use one of the mappings in the standard-library collections module to assign a default value for keys that aren’t present in the collection. The defaultdict type includes a callable that’s called each time you try to access a key that doesn’t exist. If you want the default value to be zero, you can use a lambda function that returns 0 as the first argument in defaultdict:
Python >>> from collections import defaultdict >>> points_default = defaultdict( ... lambda: 0, ... points, ... ) >>> points_default defaultdict(<function <lambda> at 0x104a95da0>, {'Denise': 3, 'Igor': 2, 'Sarah': 3, 'Trevor': 1}) >>> points_default["Sarah"] 3 >>> points_default["Matt"] 0 >>> points_default defaultdict(<function <lambda> at 0x103e6c700>, {'Denise': 3, 'Igor': 2, 'Sarah': 3, 'Trevor': 1, 'Matt': 0}) Copied!The defaultdict constructor has two arguments in this example. The first argument is the callable that’s used when a default value is needed. The second argument is the dictionary you created earlier. You can use any valid argument when you call dict() as the second argument in defaultdict() or omit this argument to create an empty defaultdict.
When you access a key that’s missing from the dictionary, the key is added, and the default value is assigned to it. You can also create the same points_default object using the callable int as the first argument since calling int() with no arguments returns 0.
All mappings are also collections, which means they’re iterable containers with a defined length. You can explore these characteristics with another mapping in Python’s standard library, collections.Counter:
Python >>> from collections import Counter >>> letters = Counter("learning python") >>> letters Counter({'n': 3, 'l': 1, 'e': 1, 'a': 1, 'r': 1, 'i': 1, 'g': 1, ' ': 1, 'p': 1, 'y': 1, 't': 1, 'h': 1, 'o': 1}) Copied!The letters in the string "learning python" are converted into keys in Counter, and the number of occurrences of each letter is used as the value corresponding to each key.
You can confirm that this mapping is iterable, has a defined length, and is a container:
Python >>> for letter in letters: ... print(letter) ... l e a r n i g p y t h o >>> len(letters) 13 >>> "n" in letters True >>> "x" in letters False Copied!You can use the Counter object letters in a for loop, which confirms it’s iterable. All mappings are iterable. However, the iteration loops through the keys and not the values. You’ll see how to iterate through the values or through both keys and values later in this tutorial.
The built-in len() function returns the number of items in the mapping. This is equal to the number of unique characters in the original string, including the space character. The object is sized since len() returns a value.
You can use the in keyword to confirm which elements are in the mapping. This check alone isn’t sufficient to confirm that the mapping is a container. However, you can also access the object’s .__contains__() special method directly:
Python >>> letters.__contains__("n") True Copied! Read the full article at https://realpython.com/python-mappings/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Golems GABB: 2024 Trends: What's New for Drupal
People always want to predict the future. They say that is not good, but this doesn't apply to the Drupal 2024 trends. Undoubtedly, Generative Artificial Intelligence will shape tech trends and stimulate further progress in the website development industry.
However, there are other matters worth paying attention to. Mintz, World Wildlife Fund, Chupa Chups, Mattel, and other prominent Drupal websites won’t sit idly by viewing this chaos of innovations and advancements.
Now is the time for unique insights with Golems web development agency about what Drupal 2024 will be like. The more aware you are of prospective game-changing rules, the more up in arms you will act in SEO, marketing, and business growth strategies. Stay tuned to take a sneak peek into the future!
Palantir: From Finance to Palantir
Taking a dive into the deep, dark quagmire of life
Some people have career aspirations from a young age, but for the rest of us, discovering a career-worthy passion, or even learning how to distinguish a job from a career, can take some time. I was nearly ten years into a job in finance (managing insurance/retirement/wealth-building products) when I finally admitted to myself that I couldn’t do it anymore. I felt like I should have a career, not just a job. I wanted to feel proud of the way I spent my days.
Diving inAfter some soul-searching, I signed up for a Full-Stack Web Development bootcamp. I was never the kid who tinkered with code or tried to see how my AOL Instant Messenger worked (yes, I was a kid of the 90s) - but I suddenly found myself fascinated with the combination of language, math, and puzzle that was hiding in my browser’s Inspect tool. The bootcamp’s cost made this a risky prospect, but it also made me more likely to commit fully. I can’t say I loved every moment of the grind; working full-time, raising three kids (whom I had adopted just two days into the bootcamp), attending evening classes, keeping up with homework, and maintaining sanity was not always the easiest, but I tend to thrive in chaos. I learned the MERN Stack (MongoDB, Express.js, React, and Node), and after six months I left the bootcamp with one of the highest scores in my class and even higher hopes.
Cue the mass tech layoffs of late 2022, pitting my emerging knowledge against developers with years of experience at tech giants such as the site formerly known as Twitter. My understanding of code was evident, but my ability to explain code was underdeveloped, leaving me flailing in interviews and unable to find anyone willing to take a chance on me. Anyone, that is, until my local Women Who Code chapter posted about Palantir.net’s fellowship program. I applied, spent three months learning Drupal through DrupalEasy's Drupal Career Online program, and started an internship with Palantir.net. That internship turned into a full-time engineering role, and now I’m creating innovative tech and building professional websites from scratch.
Going deepHow could I transition from finance to React/JavaScript to Drupal/PHP in such a short time? Same way I managed to learn French — its proximity to English based on its Latin roots gave me a leg up. Both JavaScript and PHP are Object-Oriented languages, meaning they derive from making real-world objects into code. Object-Oriented Programming, or OOP, is a paradigm that encompasses several coding languages, and since I already had a base in OOP via JavaScript, I mainly needed to learn the syntax of PHP as the concepts were mostly the same. Not only are the languages similar, but the overall folder structure and architecture of both Drupal and React can be set up using the same patterns. I quickly noticed my new Palantiri teammates using the MVC (Model-View-Controller) architecture, which I was already comfortable with.
Another benefit of that MVC architecture is its collaborative nature. By separating sections of the code based on functionality, different people could work simultaneously on building one piece of a site without risking overwriting one another. React most prominently offers collaboration through its separation of interests via Components, delivering an end product to users, while Drupal is set up to not only be collaboratively built, but to allow site builders to delegate content creation, and even site administration, to users.
Surfacing from murky depthsPerhaps the coolest thing I discovered through this journey is the ability to create a site harnessing the powers of both Drupal and React! Drupal’s Content Management System structure creates a powerful backend for React’s simple and extensive front-end library. This just leaves me with the accessibility question - how can we make Drupal a more popular and attainable technology for entry-level developers? According to Statista’s data analytic survey, Drupal ranks 29th among the most-used frameworks among developers worldwide as of 2023. In the same survey, React ranked second only to Node, which is another JavaScript framework.
I think we can find a path for learners of all types, we just need to harness the power of our community to get there. We are not in competition with React or Wordpress or any other framework, Drupal is just another tool that people should have an opportunity to explore at any level. Who knows? Maybe they’ll fall in love like so many others have.
Calm watersI love the glimmer of awe in people’s eyes now when I tell them I’m a software engineer. I can’t say the path was easy, but I’ve never regretted making the career change, and I never tire of continuing to suck up every ounce of knowledge I can find. I love the curiosity and creativity that I can bring to work with me every day, bouncing from front-end to back-end work, developing brand new technologies, and exploring every little thing that interests me. I’m happy to say I have found my career.
Culture Drupal PeopleReal Python: Quiz: Python Mappings
In this quiz, you’ll test your understanding of the basic characteristics and operations of Python mappings. By working through this quiz, you’ll revisit the key concepts and techniques of creating a custom mapping.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
The Drop Times: Why 1xINTERNET Rushed to Support the Starshot Initiative: Insights from Baddý Sonja
Matthew Garrett: SSH agent extensions as an arbitrary RPC mechanism
It turns out that it was far easier than I expected. The ssh agent protocol is documented here, and the interesting part is the extension support extension mechanism. Basically, you can declare an extension and then just tunnel whatever you want over it. As before, my goto was the go ssh agent package which conveniently implements both the client and server side of this. Implementing the local agent is trivial - look up SSH_AUTH_SOCK, connect to it, create a new agent client that can communicate with that by calling NewClient, and then implement the ExtendedAgent interface, create a new socket, and call ServeAgent against that. Most of the ExtendedAgent functions should simply call through to the original agent, with the exception of Extension(). Just add a case statement against extensionType, define some reasonably namespaced extension, and you're done.
Now you need to use this agent. You probably don't want to use this for arbitrary hosts (agent forwarding should only be enabled for remote systems you trust, not arbitrary machines you connect to - if you enabled agent forwarding for github and github got compromised, github would be able to use any private keys loaded into your agent, and you probably don't want that). So the right approach is to add a Host entry to the ssh config with a ForwardAgent stanza pointing at the socket you created in your new agent. This way the configured subset of remote hosts will automatically talk to this new custom agent, while forwarding for anything else will still be at the user's discretion.
For the remote end things are even easier. Look up SSH_AUTH_SOCK and call NewClient as before, and then simply call client.Extension(). Whatever you stick in the contents argument will simply end up being received at the client end. You now have a communication channel between a the remote system and the local client, and what you do with that is up to you. I'm using it to allow a remote system to obtain auth tokens from Okta and forward WebAuthn challenges that can either be satisfied via a local WebAuthn token or by passing the query off to Mac TouchID, but there's fundamentally no constraints whatsoever on what can be done here.
(If you want to do this on Windows and still have everything work with existing clients you'll need to take this into account - Windows didn't really do Unix sockets until recently so everything there is awful)
comments
Freexian Collaborators: Monthly report about Debian Long Term Support, May 2024 (by Roberto C. Sánchez)
Like each month, have a look at the work funded by Freexian’s Debian LTS offering.
Debian LTS contributorsIn May, 17 contributors have been paid to work on Debian LTS, their reports are available:
- Adrian Bunk did 34.25h (out of 24.0h assigned and 22.0h from previous period), thus carrying over 11.75h to the next month.
- Bastien Roucariès did 20.0h (out of 20.0h assigned).
- Ben Hutchings did 16.0h (out of 24.0h assigned), thus carrying over 8.0h to the next month.
- Chris Lamb did 18.0h (out of 18.0h assigned).
- Daniel Leidert did 8.0h (out of 10.0h assigned), thus carrying over 2.0h to the next month.
- Emilio Pozuelo Monfort did 35.5h (out of 46.0h assigned), thus carrying over 10.5h to the next month.
- Guilhem Moulin did 13.0h (out of 14.75h assigned and 5.25h from previous period), thus carrying over 7.0h to the next month.
- Lee Garrett did 11.0h (out of 37.25h assigned and 8.75h from previous period), thus carrying over 35.0h to the next month.
- Lucas Kanashiro did 10.0h (out of 20.0h assigned), thus carrying over 10.0h to the next month.
- Markus Koschany did 40.0h (out of 40.0h assigned).
- Ola Lundqvist did 6.5h (out of 22.5h assigned and 1.5h from previous period), thus carrying over 17.5h to the next month.
- Roberto C. Sánchez did 7.75h (out of 11.0h assigned and 1.0h from previous period), thus carrying over 4.25h to the next month.
- Santiago Ruano Rincón did 8.0h (out of 16.0h assigned), thus carrying over 8.0h to the next month.
- Sean Whitton did 5.5h (out of 5.5h assigned and 0.5h from previous period), thus carrying over 0.5h to the next month.
- Sylvain Beucler did 10.5h (out of 0.75h assigned and 45.25h from previous period), thus carrying over 35.5h to the next month.
- Thorsten Alteholz did 14.0h (out of 14.0h assigned).
- Tobias Frost did 7.75h (out of 10.0h assigned and 2.0h from previous period), thus carrying over 4.25h to the next month.
In May, we have released 20 DLAs.
Notable security updates in May included:
- apache2: multiple vulnerabilities which may result in HTTP response splitting, denial of service, or authorization bypass (by Bastien Roucariès, in collaboration with apache2 maintainer Yadd)
- bind9: two vulnerabilities, called KeyTrap and NSEC3, which may result in denial of service (by Santiago Ruano Rincón)
- python-pymysql: potential SQL injection attack (by Chris Lamb)
The aforementioned apache2 was prepared by its Debian maintainer Yadd. This update also involved work on the package test suite in buster, which contributor Bastien Roucariès then forwarded to the apache2 package in unstable. More importantly, a regression in fossil was reported, and Bastien prepared a fix for it. Bastien coordinated the upload of both packages to minimize the introduction of regressions.
Contributor Daniel Leidert also prepared an upload of runc to Debian 11 in order fix a number of CVEs still affecting that package. Finally, contributor Thorsten Alteholz prepared uploads for qtbase-opensource-src, libjwt, and libmicrohttpd in Debian 11. Note that Debian 11 will pass into the LTS phase of support in August and these updates will improve the state and long-term supportability of Debian 11.
Debian 10 is presently in its final month of LTS support (as announced on the debian-lts-announce mailing list, support will end on June 30th), after which no new security updates will be made available on security.debian.org.
However, Freexian and its team of paid Debian contributors will continue to maintain Debian 10 going forward for the customers of the Extended LTS offer. Subscribe right away if you sill have Debian 10 which must be kept secure (and which cannot yet be upgraded).
Thanks to our sponsorsSponsors that joined recently are in bold.
- Platinum sponsors:
- TOSHIBA (for 105 months)
- Civil Infrastructure Platform (CIP) (for 73 months)
- VyOS Inc (for 37 months)
- Gold sponsors:
- Roche Diagnostics International AG (for 116 months)
- Linode (for 110 months)
- Babiel GmbH (for 99 months)
- Plat’Home (for 99 months)
- CINECA (for 73 months)
- University of Oxford (for 55 months)
- Deveryware (for 42 months)
- EDF SA (for 26 months)
- CERN
- Silver sponsors:
- Domeneshop AS (for 120 months)
- Nantes Métropole (for 115 months)
- Univention GmbH (for 106 months)
- Université Jean Monnet de St Etienne (for 106 months)
- Ribbon Communications, Inc. (for 100 months)
- Exonet B.V. (for 90 months)
- Leibniz Rechenzentrum (for 84 months)
- Ministère de l’Europe et des Affaires Étrangères (for 67 months)
- Cloudways by DigitalOcean (for 57 months)
- Dinahosting SL (for 55 months)
- Bauer Xcel Media Deutschland KG (for 49 months)
- Platform.sh SAS (for 49 months)
- Moxa Inc. (for 43 months)
- sipgate GmbH (for 40 months)
- OVH US LLC (for 38 months)
- Tilburg University (for 38 months)
- GSI Helmholtzzentrum für Schwerionenforschung GmbH (for 30 months)
- Soliton Systems K.K. (for 27 months)
- THINline s.r.o.
- Bronze sponsors:
- Evolix (for 121 months)
- Seznam.cz, a.s. (for 121 months)
- Intevation GmbH (for 118 months)
- Linuxhotel GmbH (for 118 months)
- Daevel SARL (for 116 months)
- Bitfolk LTD (for 115 months)
- Megaspace Internet Services GmbH (for 115 months)
- NUMLOG (for 115 months)
- Greenbone AG (for 114 months)
- WinGo AG (for 114 months)
- Ecole Centrale de Nantes - LHEEA (for 110 months)
- Entr’ouvert (for 105 months)
- Adfinis AG (for 102 months)
- GNI MEDIA (for 97 months)
- Laboratoire LEGI - UMR 5519 / CNRS (for 97 months)
- Tesorion (for 97 months)
- Bearstech (for 88 months)
- LiHAS (for 88 months)
- Catalyst IT Ltd (for 83 months)
- Supagro (for 78 months)
- Demarcq SAS (for 77 months)
- Université Grenoble Alpes (for 63 months)
- TouchWeb SAS (for 55 months)
- SPiN AG (for 52 months)
- CoreFiling (for 47 months)
- Institut des sciences cognitives Marc Jeannerod (for 42 months)
- Observatoire des Sciences de l’Univers de Grenoble (for 39 months)
- Tem Innovations GmbH (for 34 months)
- WordFinder.pro (for 33 months)
- CNRS DT INSU Résif (for 32 months)
- Alter Way (for 25 months)
- Institut Camille Jordan (for 14 months)
Kay Hayen: Nuitka Release 2.3
This is to inform you about the new stable release of Nuitka. It is the extremely compatible Python compiler, “download now”.
This release bumps the long-awaited 3.12 support to a complete level. Now, Nuitka behaves identically to CPython 3.12 for the most part.
In terms of bug fixes, it’s also huge. Especially for Unicode paths and software with Unicode extension module names and Unicode program names, and even non-UTF8 code names, there have been massive amounts of improvements.
Table of Contents
Bug FixesStandalone: Added support for python-magic-bin package. Fixed in 2.2.1 already.
Fix: The cache directory creation could fail when multiple compilations started simultaneously. Fixed in 2.2.1 already.
macOS: For arm64 builds, DLLs can also have an architecture dependent suffix; check that as well. Makes the soundfile dependency scan work. Fixed in 2.2.1 already.
Fix: Modules where lazy loaders handling adds hard imports when a module is first processed did not affect the current module, potentially causing it not to resolve hidden imports. Fixed in 2.2.1 already.
macOS: The use of libomp in numba needs to cause the extension module not to be included and not to look elsewhere. Fixed in 2.2.1 already.
Python3.6+: Fix, added support for keyword arguments of ModuleNotFoundError. Fixed in 2.2.1 already.
macOS: Detect more versioned DLLs and arm64 specific filenames. Fixed in 2.2.1 already.
Fix, was not annotating exception exit when converting an import to a hard submodule import. Fixed in 2.2.2 already.
Fix, branches that became empty can still have traces that need to be merged.
Otherwise, usages outside the branch will not see propagated assignment statements. As a result, these falsely became unassigned instead. Fixed in 2.2.2 already.
Windows: Fix, uninstalled self-compiled Python didn’t have proper installation prefix added for DLL scan, resulting in runtime DLLs not picked up from there. Fixed in 2.2.2 already.
Standalone: Added support for newer PySide6 version 6.7. It needed correction on macOS and has a new data file type. Fixed in 2.2.3 already.
Standalone: Complete support for pyocd package. Fixed in 2.2.3 already.
Module: Fix, the created .pyi files were incomplete.
The list of imported modules created in the finalization step was incomplete, we now go over the actual done modules and mark all non-included modules as dependencies.
Scons: Fix, need to avoid using Unicode paths towards the linker on Windows. Instead, use a temporary output filename and rename it to the actual filename after Scons has completed.
Windows: Avoid passing Unicode paths to the dependency walker on Windows, as it cannot handle those. Also, the temporary filenames in the build folder must be in short paths, as it cannot handle them in case that is a Unicode path.
Scons: For ccache on Windows, the log filename must be a short path too, if the build folder is a Unicode path.
Windows: Make sure the Scons build executes inside a short path as well, so that a potential Unicode path is visible to the C compiler when resolving the current directory.
Windows: The encoding of Unicode paths for accelerated mode values of __file__ was not making sure that hex sequences were correctly terminated, so in some cases, it produced ambiguous C literals.
Windows: Execute binaries created with --windows-uac-admin with and --run options with proper UAC prompt.
Fix, need to allow for non-UTF8 Unicode in variable names, function names, class names, and method names.
Python3.10+: Fix, match statements that captured the rest of mapping checks were not working yet.
match value: case {"key1": 5, **rest}: ... # rest was not assigned hereWindows: When deleting build folders, make sure the retries leading to a complete deletion always.
Python2: Fix, could crash with non-unicode program paths on Windows.
Avoid giving SyntaxWarning from reading source code
For example, the standard site module of Python 3.12 gives warnings about illegal escape sequences that nobody cares about apparently.
Fix, the matplotlib warnings by options-nanny were still given even if the no-qt plugin was used, since the variable name referenced there was not actually set yet by that plugin.
Windows: Fix, when using the uninstalled self-compiled Python, we need python.exe to find DLL dependencies. Otherwise it doesn’t locate the MSVC runtime and Python DLL properly.
Standalone: Added support for freetype package.
Support for Python 3.12 is finally there. We focused on scalability first and because we did things the correct way immediately, rather than rushing to get it working and improving only later.
As a result, the correctness and performance of Nuitka with previous Python releases are improved as well.
Some things got delayed, though. We need to do more work to take advantage of other core changes. Concerning exceptions normalized at creation time, the created module code doesn’t yet take advantage. Also, more efficient two-digit long handling is possible with Python 3.12, but not implemented. It will take more time before we have these changes completed.
Experimental support for Python 3.13 beta 1 is also there, and potentially surprising, but we will try and follow its release cycle closely and aim to support it at the time of release.
Nuitka has followed all of its core changes so far, and basic tests are passing; the accelerated, module, standalone, and onefile modes all work as expected. The only thing delayed is the uncompiled generator integration, where we need to replicate the exact CPython behavior. We need to have perfect integration only for working with the asyncio loop, so we wait with it until release candidates appear.
Plugins: Added support to include directories entirely unchanged by adding raw_dir values for data-files section, see Nuitka Package Configuration.
UI: The new command line option --include-raw-dir was added to allow including directories entirely unchanged.
Module: Added support for creating modules with Unicode names. Needs a different DLL entry function name and to make use of two-phase initialization for the created extension module.
Added support for OpenBSD standalone mode.
Python3: Avoid API calls for allocators
Most effective with Python 3.11 or higher but also many other types like bytes, dict keys, float, and list objects are faster to create with all Python3 versions.
Python3.5+: Directly use the Python allocator functions for object creation, avoiding the DLL API calls. The coverage is complete with Python3.11 or higher, but many object types like float, dict, list, bytes benefit even before that version.
Python3: Faster creation of StopIteration objects.
With Python 3.12, the object is created directly and set as the current exception without normalization checks.
We also added a new specialized function to create the exception object and populate it directly, avoiding the overhead of calling of the StopIteration type.
Python3.10+: When accessing freelists, we were not passing for tstate but locally getting the interpreter object, which can be slower by a few percent in some configurations. We now use the free lists more efficient with tuple, list, and dict objects.
Python3.8+: Call uncompiled functions via vector calls.
We avoid an API call that ends up being slower than using the same function via the vector call directly.
Python3.4+: Avoid using _PyObject_LengthHint API calls in list.extend and have our variant that is faster to call.
Added specialization for os.path.normpath. We might benefit from compile time analysis of it once we want to detect file accesses.
Avoid using module constants accessor for global constant values
For example, with (), we used the module-level accessor for no reason, as it is already available as a global value. As a result, constant blobs shrink, and the compiled code becomes slightly smaller , too.
Anti-Bloat: Avoid using dask from the sparse module. Added in 2.2.2 already.
UI: Major change in console handling.
Compiled programs on Windows now have a third mode, besides console or not. You can now create GUI applications that attach to an available console and output there.
The new option --console controls this and allows to enforce console with the force value and disable using it with the disable value, the attach value activates the new behavior.
Note
Redirection of outputs to a file in attach mode only works if it is launched correctly, for example, interactively in a shell, but some forms of invocation will not work; prominently, subprocess.call without inheritable outputs will still output to a terminal.
On macOS, the distinction doesn’t exist anymore; technically it wasn’t valid for a while already; you need to use bundles for non-console applications, though, by default otherwise a console is forced by macOS itself.
Detect patchelf usage in buggy version 0.18.0 and ask the user to upgrade or downgrade it, as this specific version is known to be broken.
UI: Make clear that the --nofollow-import-to option accepts patters.
UI: Added warning for module mode and usage of the options to force outputs as they don’t have any effect.
UI: Check the success of Scons in creating the expected binary immediately after running it and not only once we reach post-processing.
UI: Detect empty user package configuration files
UI: Do not output module ast when a plugin reports an error for the module, for example, a forbidden import.
Actions: Update from deprecated action versions to the latest versions.
Use Nuitka Project Options for the user plugin test rather than passing by environment variables to the test runner.
- Added a new search mode, skip, `` to complement ``resume which resumes right
after the last test resume stopped on. We can use that while support for a Python version is not complete.
Solved a TODO about using unified code for setting the StopIteration, coroutines, generators, and asyncgen used to be different.
Unified how the binary result filename is passed to Scons for modules and executables to use the same result_exe key.
This release marks a huge step in catching up with compatibility of Python. After being late with 3.12 support, we will now be early with 3.13 support if all goes well.
The many Unicode support related changes also enhanced Nuitka to generate 2 phase loading extension modules, which also will be needed for sub-interpreter support later on.
From here on, we need to re-visit compatibility. A few more obscured 3.10 features are missing, the 3.11 compatibility is not yet complete, and we need to take advantage of the new caching possibilities to enhance performance for example with attribute lookups to where it can be with the core changes there.
For the coming releases until 3.13 is released, we hope to focus on scalability a lot more and get a much needed big improvement there, and complete these other tasks on the side.
Open Source AI Definition – Weekly update June 10
- With many different discussions happening at once, here are the main points:
- On the issue of training data
- @mark is concerned with openness of AI not being meaningful if there is not a focus on the training data.” Model weights are the most inscrutable component of current generative AI, and providers that release only [the weights] should not get a free ‘openness’ pass.”
- @stefano agrees with all of that but questions the criteria used to assign green marks in Mark’s paper, pointing out inconsistencies. They use the example of Pythia-Chat-Base-7, which relies on a dataset from OpenDataHub with potential issues like non-versioned data and stale links, failing to meet stringent requirements required by @juliaferraioli. Similar concerns are raised for other models like OLMo 7B Instruct, which lack specific data versioning details. Maffulli also highlights the case of Pythia-7B, which once may have been compliant but it’s now problematic due to the unavailability of its foundational dataset, the Pile, illustrating the complexities in maintaining an “open source” status over time, if the stringent proposal suggested by @juliaferraioli and the AWS team is adopted.
- @shujisado adds that while he sympathizes with @juliaferraioli‘s request for datasets, @stefano‘s arguments in support of the concept of “Data information” are aligned with the OSI principles and are reasonable.
- @spotaws stresses that “data information” alone is insufficient if the data itself is too vague.
- @juliaferraioli adds that while replicating AI systems like OLMo or Pythia may seem impractical due to costs and statistical nature, the capability is crucial for broader adoption and consistency. She finds the current definition to be unclear and subjective.
- @zack recommends to review StarCoder2, recognizing that it would be in the same category of BLOOM: a system with lots of transparency and a dataset made available but released with a restrictive license.
- @Ezequiel_Lanza joined the conversation in support of the concept of Data information, claiming, with technical arguments that “sharing the dataset is not necessarily required and may not justify the potential risks associated with making it mandatory.”
- Partially open / restrictive licenses
- Continuing @marks points regarding restrictive licenses (like the ethical licenses), @stefano has added a link to an article highlighting some reasons why OSI is staying away from these licenses.
- @pchestek further adds that a partially open license would create even more opportunities for open washing, as “open source AI” could have many meanings.
- @mark clarified that rather than proposing a variety of meanings, they are seeking to highlight the dimensions of openness in their paper, exploring the broader landscape.
- @stefano adds that in the 26 years of OSI, it has contended with numerous organizations claiming varying degrees of openness as “open source. This issue is now mirrored in AI, as companies seek the market value of being labeled Open Source. Open Source is binary: either users have full rights or they don’t, and any system that falls short is not Open Source AI, regardless of how “almost” open it is.
- Field of use/restriction
- @juliaferraioli believes that OSAID should include prohibitions against field-of-use restrictions.
- @shujisado adds that OSAID specifies four freedoms as requirements for being considered open source and that this should be understood as the same since “freedom” is the same as “non-restricted”. The 10 clauses of the OSD have been replaced by the checklist in draft v0.0.8.
- @juliaferraioli adds that individual components may be covered by their individual licenses, but the overall system may be subject to additional terms, which is why we need this to be explicit.
- On the issue of training data
- @Mer has added how far we are regarding our system analysis compared to our current draft definition. Some points that remain incomplete have been highlighted.
- Mistral (Mixtral 8x7B) is considered not in alignment with the OSAID because its data pre-processing code is not released under an OSI-approved license.
- @tarek_ziade shares his experience fine-tuning a “small” model (200M parameters) for a Firefox feature to describe images, using a base model for image encoding and text decoding. Despite not having 100% traceability of upstream data, Tarek argues that intentional fine-tuning and transparency make the new fine-tuned model open source. Any issues arising from downstream data can be addressed by the project maintainers, maintaining the model’s open source status.
- We held our 10th town hall meeting a week and a half ago. You can access the recording here if you missed it.
- A new town hall meeting is scheduled for this Friday, June 14.
Brett Cannon: Saying thanks to open source maintainers
After signing up for GitHub Sponsors, I had a nagging feeling that somehow asking for money from other people to support my open source work was inappropriate. But after much reflection, I realized that phrasing the use of GitHub Sponsors as a way to express patronage/support and appreciation for my work instead of sponsorship stopped me feeling bad about it. It also led me to reflect on to what degree people can express thanks to open source maintainers.
⚠️This blog post is entirely from my personal perspective and thus will not necessarily apply to every open source developer out there.Be niceThe absolutely easiest way to show thanks is to simply not be mean. It sounds simple, but plenty of people fail at even this basic level of civility. This isn&apost to say you can&apost say that a project didn&apost work for you or you disagree with something, but there&aposs a massive difference between saying "I tried the project and it didn&apost meet my needs" and "this project is trash".
People failing to support this basic level of civility is what leads to burnout.
Be an advocateIt&aposs rather indirect, but saying nice things about a project is a way of showing thanks. As an example, I have seen various people talk positively about pyproject.toml online, but not directly at me. That still feels nice due to how much effort I put into helping make that file exist and creating the [project] table.
Or put another way, you never know who is reading your public communications.
Produce your own open sourceAnother indirect way to show thanks is by sharing your own open source code. By maintaining your own code, you&aposll increase the likelihood I myself will become a user of your project. That then becomes a circuitous cycle of open source support between us.
Say thanksDirectly saying "thank you" actually goes a really long way. It takes a lot of positive interactions to counteract a single negative interaction. You might be surprised how much it might brighten someone&aposs day when someone takes the time and effort to reach out and say "thank you", whether that&aposs by DM, email, in-person at a conference, etc.
Fiscal supportAs I said in the opening of this post, I set up GitHub Sponsors for myself as a way for people to show fiscal support for my open source work if that&aposs how they prefer to express their thanks (including businesses). Now I&aposm purposefully not saying "sponsor" as to me that implies that giving money leads to some benefit (e.g. getting a shout-out somewhere) which is totally reasonable for people to do. But for me, since every commit is a gift, I&aposm financially secure, and I&aposm not trying to make a living from my volunteer open source work or put in the effort to make sponsorship worth it, I have chosen to treat fiscal support as a way of showing reciprocity for the gift of sharing my code that you&aposve already received. This means I fully support all open source maintainers setting up fiscal support at a minimum, and if they want to put in the effort to go the sponsorship route then they definitely should.
Producing open source also isn&apost financially free. For instance, I pay for:
- The hosting of this blog via Ghost(Pro)
- Obsidian Sync to keep my open source notes available on all my devices so when I have an idea I can write it down
- Obsidian Publish to share my open source notes
- Computer upgrades (including ergonomic upgrades like keyboards)
- My personal time away from my wife and child, family and friends (which my open source journal exists to try and point out for those who don&apost realize how much time I put into my volunteer work)
So while open source is "free" for you as the consumer, the producer very likely has concrete financial costs in producing that open source on top of the intangible costs like volunteering their personal time.
But as I listed earlier, there are plenty of other ways to show thanks without having to spend money that can be equally valuable to a maintainer.
I also specifically didn&apost mention contributing. I have said before that contributions are like giving someone a puppy: it seems like a lovely gift at the time, but the recipient is now being "gifted" daily walks involving scooping 💩 and vet bills. As such, contributions from others can be a blessing and a curse all at the same time depending on the contribution itself, the attitude of the person making the contribution, etc. So I wouldn&apost always assume my contribution is as welcomed and desired as much as a "thank you" note.
ImageX: Easy Third-Party Integration in Drupal Forms: Dynamically Pulling Data From Other Sources
Authored by Nadiia Nykolaichuk, João Paulo Constantino, Ana Carolina, and Gabriel Passarelli.
ImageX: Easy Third-Party Integration in Drupal Forms: Dynamically Pulling Data From Other Sources
Authored by Nadiia Nykolaichuk, João Paulo Constantino, Ana Carolina, and Gabriel Passarelli.
PyCoder’s Weekly: Issue #633 (June 11, 2024)
#633 – JUNE 11, 2024
View in Browser »
In this tutorial, you’ll learn about the different tools that Python provides for performing string interpolation. String interpolation allows you to create new strings by inserting different objects into a string template.
REAL PYTHON
This is a collection of Python Notebooks for teaching and learning the fundamentals of music processing. Examples include illustrations, sound samples, math, and more.
INTERNATIONAL AUDIO LABS
Stop wasting 30% of your team’s sprint on maintaining legacy codebases. Automatically migrate and keep up-to-date on Python versions, so that you can focus on being productive while staying secure, without the risk of breaking changes - Get a code assessment today →
ACTIVESTATE sponsor
This article describes every command-line tool included with Python, each of which can be run with python -m module_name.
TREY HUNNER
Take this quiz to test your understanding of the available tools for string interpolation in Python, as well as their strengths and weaknesses. These tools include f-strings, the .format() method, and the modulo operator.
REAL PYTHON
See the full list of changes in this release
CPYTHON DEV BLOG
This Python Enhancement Proposal “Adding a ‘converter’ parameter to dataclasses.field” was determined to have an insufficient number of use cases.
PYTHON
CRUD operations are the cornerstone of application functionality. Whether you access a database or interact with a REST API, you usually want to create, retrieve, update, and delete data. In this tutorial, you’ll explore how CRUD operations work in practice.
REAL PYTHON
Mahesh talks about the rules he has encountered when doing research on designing large systems. Guidelines include late-binding on the design, focusing on the problem rather than existing systems, talking about other applications, and more.
MAHESH BALAKRISHNAN
Introducing “Database Mind” - a ready-to-use AI system designed for easy integration into your projects. As part of the “Minds Endpoints” AI platform, it offers a simple plug-and-play API service, enabling developers to effortlessly incorporate advanced AI capabilities into their solutions →
MINDSDB sponsor
This detailed article looks at how to use the match statement along with Python’s typing mechanism to write functional programs similar in style to Kotlin.
OSKAR WICKSTROM
The Matplotlib package is great for visualizing data. One of its many features is the ability to annotate points on your graph. This article shows you how.
MIKE DRISCOLL
The bytes data type looks a bit like a string, but it isn’t a string. This article explores it and also looks at the main Unicode encoding, UTF-8
STEPHEN GRUPPETTA
“Being a manager is a focus change from code to people, from output to outcomes and from being productive to making most of everyone’s time.” Read more of Victor’s reflecting on his first year as a manager.
VICTOR STOJANOV
This article is on using mock in your Python testing and is part of a larger series on testing in general.
BITECODE
GITHUB.COM/HEALTHLANE-TECHNOLOGIES
gloe: Library for Flow-Oriented Code Events Weekly Real Python Office Hours Q&A (Virtual) June 12, 2024
REALPYTHON.COM
June 12 to June 15, 2024
WAGTAIL.SPACE
June 13 to June 15, 2024
DJANGOGIRLS.ORG
June 13 to June 14, 2024
MEETUP.COM
June 14 to June 17, 2024
PYDATA.ORG
June 15 to June 17, 2024
BARCAMPS.EU
June 20 to June 23, 2024
WAGTAIL.SPACE
Happy Pythoning!
This was PyCoder’s Weekly Issue #633.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]