Feeds

This month, I had the privilege of representing the Open Source Initiative at the Digital Public Goods Alliance (DPGA) Annual Members Meeting. Held in Singapore, this event marked our second year participating as members, following our first participation in Ethiopia. It was an inspiring gathering of innovators, developers and advocates working to create a thriving ecosystem for Digital Public Goods (DPGs) as part of UNICEF’s initiative to advance the United Nations’ sustainable development goals (SDGs).

Keynote insights

The conference began with an inspiring keynote by Liv Marte Nordhaug, CEO of the DPGA, who made a call for governments and organizations to incorporate:

1. Open Source first principles
2. Open data at scale
3. Interoperable Digital Public Infrastructure (DPI)
4. DPGs as catalysts for climate change action

These priorities underscored the critical role of DPGs in fostering transparency, accountability and innovation across sectors.

DPGs and Open Source AI

A standout feature of the first day of the event was the Open Source AI track led by Amreen Taneja, DPGA Standards Lead, which encompassed three dynamic sessions:

1. Toward AI Democratization with Digital Public Goods: This session explored the role of DPGs in contributing to the democratization of AI technologies, including AI use, development and governance, to ensure that everyone benefits from AI technology equally.
   
2. Fully Open Public Interest AI with Open Data: This session highlighted the need for open AI infrastructure supported by accessible, high-quality datasets, especially in the global majority countries. Discussions evolved over how open training data sets ought to be licensed to ensure open, public interest AI.
   
3. Creating Public Value with Public AI: This session examined real-world applications of generative AI in public services. Governments and NGOs showcased how AI-enabled tools can effectively tackle social challenges, leveraging Open Source solutions within the AI stack.

DPG Product Fair: showcasing innovation

The second day of the event was marked by the DPG Product Fair, which provided a science-fair-style platform for showcasing DPGs. Notable examples included:

• India’s eGov DIGIT Open Source platform, serving over 1 billion citizens with robust digital infrastructure.
• Singapore’s Open Government Products, which leverages Open Source and enables open collaboration to allow this small nation to expand their impact together with other southeast Asian nations.

One particularly engaging session was Sarah Espaldon’s “Improving Government Services with Legos.” This presentation from the Singapore government highlighted the benefits of modular DPGs in enhancing service delivery and building flexible DPI capabilities.

Privacy best practices for DPGs

A highlight from the third and final day of the event was the privacy-focused workshop co-hosted by the DPGA and the Open Knowledge Foundation. As privacy becomes a central concern for DPGs, the DPGA introduced its Standard Expert Group to refine privacy requirements and develop best practice guidelines. The interactive session provided invaluable feedback, driving forward the development of robust privacy standards for DPGs.

Looking ahead

The event reaffirmed the potential of Open Source technologies to transform global public goods. As we move forward, the Open Source Initiative is committed to advancing these conversations, including around the Open Source AI Definition and fostering a more inclusive digital ecosystem. A special thanks to the dedicated DPGA team—Liv Marte Nordhaug, Lucy Harris, Max Kintisch, Ricardo Miron, Luciana Amighini, Bolaji Ayodeji, Lea Gimpel, Pelin Hizal Smines, Jon Lloyd, Carol Matos, Amreen Taneja, and Jameson Voisin—whose efforts made this conference a success. We look forward to another year of impactful collaboration and to the Annual Members Meeting next year in Brazil!

There is more than one way to do it, but it seems that The Right Way to build Debian packages today is using sbuild with the unshare backend. The most common backend before the rise of unshare was schroot.

The official Debian Build Daemons have recently transitioned to using sbuild with unshare, providing a strong motivation to consider making the switch. Additionally the new approach means: (1) no need to configure schroot, and (2) no need to run the build as root.

Here are my notes about moving to the new setup, for future reference and in case they may be useful to others.

First I installed the required packages:

apt install sbuild mmdebstrap uidmap

Then I created the following script to update my chroots every night:

#!/bin/bash for arch in arm64 armhf armel; do HOME=/tmp mmdebstrap --quiet --arch=$arch --include=ca-certificates --variant=buildd unstable \ ~/.cache/sbuild/unstable-$arch.tar http://127.0.0.1:3142/debian done

In the script, I’m calling mmdebstrap with --quiet because I don’t want to get any output on succesful execution. The script is running in cron with email notifications, and I only want to get a message if something goes south. I’m setting HOME=/tmp for a similar reason: the unshare user does not have access to my actual home directory, and by default dpkg tries to use $HOME/.dpkg.cfg as the configuration file. By overriding HOME I avoid the following message to standard error:

dpkg: warning: failed to open configuration file '/home/ema/.dpkg.cfg' for reading: Permission denied

Then I added the following to my sbuild configuration file (~/.sbuildrc):

$chroot_mode = 'unshare'; $unshare_tmpdir_template = '/dev/shm/tmp.sbuild.XXXXXXXXXX';

The first option sets the sbuild backend to unshare, whereas unshare_tmpdir_template is needed on Bookworm to ensure that the build process runs in memory rather than on disk for performance reasons. Starting with Trixie, /tmp is by default a tmpfs so the setting won’t be needed anymore.

Packages for different architectures can now be built as follows:

# Tarball used: ~/.cache/sbuild/unstable-arm64.tar $ sbuild --dist=unstable hello # Tarball used: ~/.cache/sbuild/unstable-armhf.tar $ sbuild --dist=unstable --arch=armhf hello

If you have any comments or suggestions about any of this, please let me know.

We know you love FAQs! We'll answer your questions you have about the all-new Drupal CMS (yep, 13 of them). From features to migration, find out everything you need to know about this groundbreaking new CMS.

In order to prepare for the Akademy I started some days before to give my Librem 5 ( an Open Hardware Phone) another try and ended up with a non starting Plasma 6. Actually this issue was known already, but hasn't been addressed. In the end I reached the Akademy with my Librem 5 having phosh installed (which is Gnome based), in order to have something working.

I met Bushan and Bart who took care and the issue was fixed two days later I could finally install Plasma 6 on it. The last time I tested my Librem 5 with Plasma 5 it felt sluggish and not well working. But this time I was impressed how well the system reacts. Sure there are some things here and there, but in the bigger picture it is quite useable. One annoying issue is that the camera is only working with one app and the other issue is the battery capacity, you have to charge it once a day. Because of missing a QR reader that can use the camera, getting data to the phone was quite challenging. Unfortunately the conference Wifi separated the devices and I couldn't use KDE Connect to transfer data. In the end the only way to import data was taking five photos from the QR Code to import my D-Ticket to Itinerary.

With a device with Plasma Mobile, it directly was used for a experiment: How well does Dolphin works on a Plasma Mobile device. Together with Felix Ernst we tried it out and were quite impressed, that Dolphin does work very well on Plasma Mobile, after some simple modifications on the UI. That resulted in a patch to add a mobile UI for Dolphin !826.

With more time to play with my Librem 5 I also found an bug in KWeather, that is missing a Refresh option, when used in a Plasma Mobile environment #493656.

Akademy is a good place to identify and solve some issues. It is always like that, you chat with someone and they can tell you who to ask to answer the concrete question and in the end you can solve things, that seems unsolvable in the beginning.

There was also time to look into the travelling app Itinerary. A lot people are faced with a lot of real world issues, when not in their home town. Itinerary is the best traveling apps I know about. It can import nearly every ticket you have and can get location information from restaurant websites and allow routing to that place. It does add many useful information, while traveling like current delays, platform changes, live updates for elevator, weather information at the destination, a station map and all those features with strong focus on privacy.

In detail I found some small things to improve:

• If you search for a bus ride and enter the correct name for the bus stop, it will still add some walk from and to the station. The issue here is that we use different backends and not all backends share the same geo coordinate. That's why Itinerary needs to add some heuristics to delete those paths.

• Instead of displaying just a small station map of one bus stop in the inner city, it showed complete Würzburg inner city, as there is one big park around the inner city (named "Ringpark").

• Würzburg has a quite big bus station but the platform information were missing in the map, so we tweaked the CSS to display the platform. To be sure, that we don't fix only Würzburg, we also looked at Greifswald and Aix-en-Provence if they are following the same name scheme.

I additionally learned that it has a lot of details that helps people who have special needs. That is the reason why Daniel Kraut wants to get Itinerary available for iOS. As spoken out, that Daniel wants to reach this goal, others already started to implement the first steps to build apps for iOS.

This year I was volunteering in helping out at Akademy. For me it was a lot of fun to meet everyone at the infodesk or help the speakers setup the beamer and microphone. It is also a good opportunity to meet many new faces and get in contact with them. I see also room for improvement. As we were quite busy at the Welcome Event to get out the badges to everyone, I couldn't answer the questions from newcomers, as the queue was too long. I propose that some people volunteer to be available for questions from newcomers. Often it is hard for newcomers to get their first contact(s) in a new community. There is a lot of space for improvement to make it easier for newcomers to join. Some ideas in my head are: Make an event for the newcomers to get them some links into the community and show that everyone is friendly. The tables at the BoFs should make a circle, so everyone can see each other. It was also hard for me to understand everyone as they mostly spoken towards the front. And then BoFs are sometimes full of very specific words and if you are not already deep in the topic you are lost. I can see the problem, on the one side BoFs are also the place where the person that knows the topic already wants to get things done. On the other side new comers join BoFs, are overwhelmed by to many new words get frustrated and think, that they are not welcome. Maybe at least everyone should present itself with name and ask new faces, why they joined the BoF to help them joining.

I'm happy, that the food provided for the attendees was very delicious and that I'm not the only one mostly vegetarian with a big amount to be vegan.

At the conference the KDE Eco initiation really caught me, as I see a lot of new possibilities in giving more reasons to switch to an Open Source system. The talk from Natalie was great to see how pupils get excited about Open Source and also help their grandparents to move to a Linux system. As I also will start to work as a teacher, I really got ideas what I can do at school. Together with Joseph and Nicole, we finally started to think about how to drive an exploration on what kind of old hardware is still KDE software running. The ones with the oldest hardware will get an old KDE shirt. For more information see #40.

The conference was very motivating for me, I also had still energy at the evening to do some Debian packaging and finally pushed kweathercore to Debian and started to work on KWeather. Now I'm even more interested in the KDE apps focusing the mobile world, as I now have some hardware that can actually use those apps.

I really enjoyed the workshop how to contribute to Qt by Volker Hilsheimer, especially the way how Volker explained things in a very friendly way, answered every question, sometime postponed some questions but came back to them later. All in all I now have a good overview how Qt is doing development and how I can fix bugs.

The daytrip to Rothenburg ob der Tauber was very interesting for me. It was the first time I visited the village. But in my memory it feels like I know the village already. I grew up with reading a lot of comic albums including the good SiFi comic album series "Yoku Tsuno" created by the Belgian writer Roger Leloup. Yoku Tsuno is an electronics engineer, raised in Japan but now living in Belgium. In "On the edge of life" she helps her friend Ingard, who actually lives in Rothenburg. Leloup invested a lot of time to travel to make the make his drawings as accurate as possible.

In order to not have a hard cut from Akademy to normal life, I had a lunch with Carlos, to discuss KDE Neon and how we can improve the interaction with Debian. In the future this should have less friction and make both communities work together more smoothly. Additionally as I used to develop on KDEPIM with the help of Docker images based on Neon I ask for a meta kf6 dev meta package. That should help to get rid of most hand written lists of dev packages in the Docker file in order to make it more simple for new contributors to start hacking on KDEPIM.

The rest of the day I finally found time to do the normal tourist stuff: Going to the Wine bridge and having a walk to the castle of Würzburg. Unfortunately you hear a lot of car noises up there, but I could finally relaxe in a Japanese designed garden.

Finally at Saturday I started my trip back. The trains towards Eberswalde are broken and I needed to find alternative routing. I got a little bit nervous, as it was the first time I travelled with my Librem 5 and Itinerary only and needed to reach the next train in less than two mins. With the indoor maps provided, I could prepare my run through the train station so I reached successfully my next train.

By the way, also if you only only use KDE software, I would recommend everyone to join Akademy ;)

In order to prepare for the Akademy I started some days before to give my Librem 5 ( an Open Hardware Phone) another try and ended up with a non starting Plasma 6. Actually this issue was known already, but hasn't been addressed. In the end I reached the Akademy with my Librem 5 having phosh installed (which is Gnome based), in order to have something working.

I met Bushan and Bart who took care and the issue was fixed two days later I could finally install Plasma 6 on it. The last time I tested my Librem 5 with Plasma 5 it felt sluggish and not well working. But this time I was impressed how well the system reacts. Sure there are some things here and there, but in the bigger picture it is quite useable. One annoying issue is that the camera is only working with one app and the other issue is the battery capacity, you have to charge it once a day. Because of missing a QR reader that can use the camera, getting data to the phone was quite challenging. Unfortunately the conference Wifi separated the devices and I couldn't use KDE Connect to transfer data. In the end the only way to import data was taking five photos from the QR Code to import my D-Ticket to Itinerary.

With a device with Plasma Mobile, it directly was used for a experiment: How well does Dolphin works on a Plasma Mobile device. Together with Felix Ernst we tried it out and were quite impressed, that Dolphin does work very well on Plasma Mobile, after some simple modifications on the UI. That resulted in a patch to add a mobile UI for Dolphin !826.

With more time to play with my Librem 5 I also found an bug in KWeather, that is missing a Refresh option, when used in a Plasma Mobile environment #493656.

Akademy is a good place to identify and solve some issues. It is always like that, you chat with someone and they can tell you who to ask to answer the concrete question and in the end you can solve things, that seems unsolvable in the beginning.

There was also time to look into the travelling app Itinerary. A lot people are faced with a lot of real world issues, when not in their home town. Itinerary is the best traveling apps I know about. It can import nearly every ticket you have and can get location information from restaurant websites and allow routing to that place. It does add many useful information, while traveling like current delays, platform changes, live updates for elevator, weather information at the destination, a station map and all those features with strong focus on privacy.

In detail I found some small things to improve:

• If you search for a bus ride and enter the correct name for the bus stop, it will still add some walk from and to the station. The issue here is that we use different backends and not all backends share the same geo coordinate. That's why Itinerary needs to add some heuristics to delete those paths.

• Instead of displaying just a small station map of one bus stop in the inner city, it showed complete Würzburg inner city, as there is one big park around the inner city (named "Ringpark").

• Würzburg has a quite big bus station but the platform information were missing in the map, so we tweaked the CSS to display the platform. To be sure, that we don't fix only Würzburg, we also looked at Greifswald and Aix-en-Provence if they are following the same name scheme.

I additionally learned that it has a lot of details that helps people who have special needs. That is the reason why Daniel Kraut wants to get Itinerary available for iOS. As spoken out, that Daniel wants to reach this goal, others already started to implement the first steps to build apps for iOS.

This year I was volunteering in helping out at Akademy. For me it was a lot of fun to meet everyone at the infodesk or help the speakers setup the beamer and microphone. It is also a good opportunity to meet many new faces and get in contact with them. I see also room for improvement. As we were quite busy at the Welcome Event to get out the badges to everyone, I couldn't answer the questions from newcomers, as the queue was too long. I propose that some people volunteer to be available for questions from newcomers. Often it is hard for newcomers to get their first contact(s) in a new community. There is a lot of space for improvement to make it easier for newcomers to join. Some ideas in my head are: Make an event for the newcomers to get them some links into the community and show that everyone is friendly. The tables at the BoFs should make a circle, so everyone can see each other. It was also hard for me to understand everyone as they mostly spoken towards the front. And then BoFs are sometimes full of very specific words and if you are not already deep in the topic you are lost. I can see the problem, on the one side BoFs are also the place where the person that knows the topic already wants to get things done. On the other side new comers join BoFs, are overwhelmed by to many new words get frustrated and think, that they are not welcome. Maybe at least everyone should present itself with name and ask new faces, why they joined the BoF to help them joining.

I'm happy, that the food provided for the attendees was very delicious and that I'm not the only one mostly vegetarian with a big amount to be vegan.

At the conference the KDE Eco initiation really caught me, as I see a lot of new possibilities in giving more reasons to switch to an Open Source system. The talk from Natalie was great to see how pupils get excited about Open Source and also help their grandparents to move to a Linux system. As I also will start to work as a teacher, I really got ideas what I can do at school. Together with Joseph and Nicole, we finally started to think about how to drive an exploration on what kind of old hardware is still KDE software running. The ones with the oldest hardware will get an old KDE shirt. For more information see #40.

The conference was very motivating for me, I also had still energy at the evening to do some Debian packaging and finally pushed kweathercore to Debian and started to work on KWeather. Now I'm even more interested in the KDE apps focusing the mobile world, as I now have some hardware that can actually use those apps.

I really enjoyed the workshop how to contribute to Qt by Volker Hilsheimer, especially the way how Volker explained things in a very friendly way, answered every question, sometime postponed some questions but came back to them later. All in all I now have a good overview how Qt is doing development and how I can fix bugs.

The daytrip to Rothenburg ob der Tauber was very interesting for me. It was the first time I visited the village. But in my memory it feels like I know the village already. I grew up with reading a lot of comic albums including the good SiFi comic album series "Yoku Tsuno" created by the Belgian writer Roger Leloup. Yoku Tsuno is an electronics engineer, raised in Japan but now living in Belgium. In "On the edge of life" she helps her friend Ingard, who actually lives in Rothenburg. Leloup invested a lot of time to travel to make the make his drawings as accurate as possible.

In order to not have a hard cut from Akademy to normal life, I had a lunch with Carlos, to discuss KDE Neon and how we can improve the interaction with Debian. In the future this should have less friction and make both communities work together more smoothly. Additionally as I used to develop on KDEPIM with the help of Docker images based on Neon I ask for a meta kf6 dev meta package. That should help to get rid of most hand written lists of dev packages in the Docker file in order to make it more simple for new contributors to start hacking on KDEPIM.

The rest of the day I finally found time to do the normal tourist stuff: Going to the Wine bridge and having a walk to the castle of Würzburg. Unfortunately you hear a lot of car noises up there, but I could finally relaxe in a Japanese designed garden.

Finally at Saturday I started my trip back. The trains towards Eberswalde are broken and I needed to find alternative routing. I got a little bit nervous, as it was the first time I travelled with my Librem 5 and Itinerary only and needed to reach the next train in less than two mins. With the indoor maps provided, I could prepare my run through the train station so I reached successfully my next train.

By the way, also if you only only use KDE software, I would recommend everyone to join Akademy ;)

Tuesday, 26 November 2024. Today KDE releases a bugfix update to KDE Plasma 6, versioned 6.2.4.

Plasma 6.2 was released in October 2024 with many feature refinements and new modules to complete the desktop experience.

This release adds three weeks' worth of new translations and fixes from KDE's contributors. The bugfixes are typically small but important and include:

• libkscreen Doctor: clarify the meaning of max. brightness zero. Commit. Fixes bug #495557
• Plasma Workspace: Battery Icon, Add headphone icon. Commit.
• Plasma Audio Volume Control: Fix speaker test layout for Pro-Audio profile. Commit. Fixes bug #495752

View full changelog

Ruqola 2.3.2 is a feature and bugfix release of the Rocket.chat app.

It includes many fixes for RocketChat 7.0.

New features:

• Fix administrator refresh user list
• Fix menu when we select video conference message
• Fix RocketChat 7.0 server support
• Fix create video message
• Fix update cache when we change video/attachment description
• Fix export message job
• Fix show userOffline when we have a group
• Fix enable/disable ok button when search room in team dialog
• Fix crash when we remove room in team dialog
• Fix update channel selection when we reconnect server

URL: https://download.kde.org/stable/ruqola/
Source: ruqola-2.3.2.tar.xz
SHA256: 57c8ff6fdeb4aba286425a1bc915db98ff786544a3ada9dec39056ca4b587837
Signed by: E0A3EB202F8E57528E13E72FD7574483BB57B18D Jonathan Riddell jr@jriddell.org
https://jriddell.org/jriddell.pgp

Today we are talking about being the CTO of the Drupal Association, How the job has changed, and How its impacted Drupal with guests Josh Mitchell & Tim Lehnen. We’ll also cover Automatic Anchors as our module of the week.

For show notes visit: https://www.talkingDrupal.com/477

Topics

• How long ago were you CTO Josh
• Tim when did you take over
• DA infrastructure
• Drupal Credit System
• Josh's proudest moment
• Tim's proudest moment
• Growth
• Josh if you could do one thing differently
• Tim if you could make one change
• Future of the CTO job

Resources

• OOP Hook conversion
• Oregon State University Open Source Lab
• Whuffie: Cory doctorow Down and Out in the Magic Kingdom
• Rethink weighing of contrib projects and credits

Guests

Tim Lehnen - aspenthornpress.com hestenet

Hosts

Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Joshua "Josh" Mitchell - joshuami.com joshuami

MOTW Correspondent

Martin Anderson-Clutz - mandclu.com mandclu

• Brief description:
  • Have you ever wanted headings on your Drupal site to have unique id values, so links can be created to take users to specific parts of any page? There’s a module for that.
• Module name/project name:
  • Automatic Anchors
• Brief history
  • How old: created in Jun 2020 by Chris Komlenic (komlenic) of Penn State
  • Versions available: 2.1.1-beta1, which supports Drupal 8.8, 9, and 10
• Maintainership
  • Test coverage
  • Number of open issues: x open issues, y of which are bugs against the current branch
• Usage stats:
  • 137 sites
• Module features and usage
  • By default, the module automatically generates ids on , , , , and elements within the page content
  • Even if two headings have the same content, the module will make sure their ids are unique, as well as making sure they are i18n-friendly, use hyphens instead of spaces, and are short enough to be useful
  • The module won’t interfere with or change manually-added or already-existing HTML ids
  • There’s a permission to view helpful links on each heading that the ids obvious and easy to copy
  • Configuration options include the root element it should look within (defaults to the body tag), which elements should get ids, what content to use for the displayed links, and whether or not generate ids on admin pages

Black Friday and Cyber Monday are nearly here, so it’s time to do a Python sale! All my books and courses are 35% off until December 4th if you use this code: BF24 at checkout.

You can learn about any of the following topics in my books and courses:

• Basic Python (Python 101)
• PDF Processing (ReportLab book)
• Excel and Python
• Image processing (Pillow book)
• Python Logging (NEW this year!)
• Jupyter Notebook
• JupyterLab 101 (NEW this month!)
• and more

Check out my books and courses today!

Other Python Sales

• Sundeep Agarwal: ~50% off Sundeep’s all book and Python bundles with code FestiveOffer
• Python Jumpstart with Python Morsels: 50% off Trey Hunner’s brand new Python course, an introduction to Python that’s very hands-on ($99 instead of $199)
• Rodrigo 50% off Rodrigo’s all books bundle with code BF24
• The Python Coding Place: 40% off The Python Coding Book and 40% off a lifetime membership to The Python Coding Place with code black2024
• Adam Johnson’s Django-related Deals for Black Friday 2024

The post Black Friday Python Deals 2024 appeared first on Mouse Vs Python.

I’ve just recently launched a self-paced introduction to Python that is extremely hands-on. It’s called Python Jumpstart and it’s based on introductory Python curriculum that I have been iterating on for years.

Learn Python by writing Python code ✍

We do not learn by putting information into the brain. Our brains simply don’t retain knowledge that way.

Learning happens from repeatedly attempting to retrieve information from the brain. When it comes to Python, that means writing Python code.

So unlike most Python courses, Python Jumpstart is not focused on watching videos and rewriting code seen within videos. Instead, this new Python course is based around learning Python by writing Python code.

How Python Jumpstart is structured 🔬

Python Jumpstart includes learning Python by solving 46 short Python exercises.

Before each exercise, you’ll watch a 5 minute video explaining a new Python topic. You’ll then attempt the exercise to put one or more Python topics into practice.

You won’t solve many of the exercises on your first try and that’s okay. These exercises are designed to be revisited a few times over the course of weeks, until you’re satisfied with your solution.

This is not a sprint 📆

This structured path to Python proficiency, includes:

• Carefully crafted exercises that build real understanding
• Short, detailed explanations that won’t waste your time
• A proven teaching approach that focuses on active learning
• Spaced repetition to help concepts stick

This is not a course for impatient or passive learners. Learning takes repetitive effort spaced over many days and this course is structured to embrace that fact.

If you spend 30 minutes on Python Jumpstart each day, I estimate that you’ll complete this course in about 7 weeks. You’ll spend the large majority of that time writing Python code.

This course will take time, but it will be time well-spent.

Launch week special: 50% off until December 2 ⏰

Through Monday December 2, 2024, you can get lifetime access to Python Jumpstart for $99. After this launch week, Python Jumpstart will be $199.

Ready to jumpstart your Python learning journey?

Get Python Jumpstart for $99

Concurrency refers to the ability of a program to manage multiple tasks at once, improving performance and responsiveness. It encompasses different models like threading, asynchronous tasks, and multiprocessing, each offering unique benefits and trade-offs. In Python, threads and asynchronous tasks facilitate concurrency on a single processor, while multiprocessing allows for true parallelism by utilizing multiple CPU cores.

Understanding concurrency is crucial for optimizing programs, especially those that are I/O-bound or CPU-bound. Efficient concurrency management can significantly enhance a program’s performance by reducing wait times and better utilizing system resources.

In this tutorial, you’ll learn how to:

• Understand the different forms of concurrency in Python
• Implement multi-threaded and asynchronous solutions for I/O-bound tasks
• Leverage multiprocessing for CPU-bound tasks to achieve true parallelism
• Choose the appropriate concurrency model based on your program’s needs

To get the most out of this tutorial, you should be familiar with Python basics, including functions and loops. A rudimentary understanding of system processes and CPU operations will also be helpful. You can download the sample code for this tutorial by clicking the link below:

Get Your Code: Click here to download the free sample code that you’ll use to learn about speeding up your Python program with concurrency.

Take the Quiz: Test your knowledge with our interactive “Python Concurrency” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Python Concurrency

In this quiz, you'll test your understanding of Python concurrency. You'll revisit the different forms of concurrency in Python, how to implement multi-threaded and asynchronous solutions for I/O-bound tasks, and how to achieve true parallelism for CPU-bound tasks.

Exploring Concurrency in Python

In this section, you’ll get familiar with the terminology surrounding concurrency. You’ll also learn that concurrency can take different forms depending on the problem it aims to solve. Finally, you’ll discover how the different concurrency models translate to Python.

What Is Concurrency?

The dictionary definition of concurrency is simultaneous occurrence. In Python, the things that are occurring simultaneously are called by different names, including these:

• Thread
• Task
• Process

At a high level, they all refer to a sequence of instructions that run in order. You can think of them as different trains of thought. Each one can be stopped at certain points, and the CPU or brain that’s processing them can switch to a different one. The state of each train of thought is saved so it can be restored right where it was interrupted.

You might wonder why Python uses different words for the same concept. It turns out that threads, tasks, and processes are only the same if you view them from a high-level perspective. Once you start digging into the details, you’ll find that they all represent slightly different things. You’ll see more of how they’re different as you progress through the examples.

Now, you’ll consider the simultaneous part of that definition. You have to be a little careful because, when you get down to the details, you’ll discover that only multiple system processes can enable Python to run these trains of thought at literally the same time.

In contrast, threads and asynchronous tasks always run on a single processor, which means they can only run one at a time. They just cleverly find ways to take turns to speed up the overall process. Even though they don’t run different trains of thought simultaneously, they still fall under the concept of concurrency.

Note: Threads in most other programming languages often run in parallel. To learn why Python threads can’t, check out What Is the Python Global Interpreter Lock (GIL)?

If you’re curious about even more details, then you can also read about Bypassing the GIL for Parallel Processing in Python or check out the experimental free threading introduced in Python 3.13.

The way the threads, tasks, or processes take turns differs. In a multi-threaded approach, the operating system actually knows about each thread and can interrupt it at any time to start running a different thread. This mechanism is also true for processes. It’s called preemptive multitasking since the operating system can preempt your thread or process to make the switch.

Preemptive multitasking is handy in that the code in the thread doesn’t need to do anything special to make the switch. It can also be difficult because of that at any time phrase. The context switch can happen in the middle of a single Python statement, even a trivial one like x = x + 1. This is because Python statements typically consist of several low-level bytecode instructions.

On the other hand, asynchronous tasks use cooperative multitasking. The tasks must cooperate with each other by announcing when they’re ready to be switched out without the operating system’s involvement. This means that the code in the task has to change slightly to make it happen.

The benefit of doing this extra work upfront is that you always know where your task will be swapped out, making it easier to reason about the flow of execution. A task won’t be swapped out in the middle of a Python statement unless that statement is appropriately marked. You’ll see later how this can simplify parts of your design.

What Is Parallelism?

So far, you’ve looked at concurrency that happens on a single processor. What about all of those CPU cores your cool, new laptop has? How can you make use of them in Python? The answer is to execute separate processes!

A process can be thought of as almost a completely different program, though technically, it’s usually defined as a collection of resources including memory, file handles, and things like that. One way to think about it is that each process runs in its own Python interpreter.

Because they’re different processes, each of your trains of thought in a program leveraging multiprocessing can run on a different CPU core. Running on a different core means that they can actually run at the same time, which is fabulous. There are some complications that arise from doing this, but Python does a pretty good job of smoothing them over most of the time.

Read the full article at https://realpython.com/python-concurrency/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

With the release of Qt 6.8.1 and 6.5.8, we are updating our documentation to clarify Qt’s support for newly released Apple operating system versions. 

Welcome to a new issue of "This Week in KDE Apps"! Every week we cover as much as possible of what's happening in the world of KDE apps.

This week, we are continuing to prepare for the KDE Gear 24.12.0 release, with a focus on bugfixing now that we've entered the feature freeze period.

Meanwhile, and as part of the 2024 end-of-year fundraiser, you can "Adopt an App" in a symbolic effort to support your favorite KDE app. This week, we are particularly grateful to mdPlusPlus and txemaq for supporting Dolphin; mdPlusPlus, Greg Helding and Archie Lamb for Okular; Henning Lammert and Thibault Molleman for Filelight; Nithanim, Dominik Perfler, and Thibault Molleman for Spectacle; Vladimir Solomatin, Akseli Lahtinen, Haakon Johannes Tjelta Meihack, and Nithanim for Kate; Henning Lammert and Marco Ryll for Kasts; GhulDev, Anders Lund Tobias Junghans, and William Wojciechowski for Konsole; Piwix for KWrite; Gabriel Klavans for Tokodon; Matthew Lamont for Kontact; and Gabriel Karlsson for Itinerary.

Any monetary contribution, however small, will help us cover operational costs, salaries, travel expenses for contributors and in general just keep KDE bringing Free Software to the world. So consider donating today!

Getting back to all that's new in the KDE App scene, let's dig in!

Global Changes

The KIO implementation for SFTP used by many KDE applications like Dolphin, Gwenview, and many others, now correctly closes the network connection when if a fatal error occurs, which means it is now possible to reconnect immediately without having to wait a few minutes. (Harald Sitter, 24.12.0. Link)

Many apps received some small bug fixes to ensure they work correctly on Haiku OS. (Luc Schrijvers, Link 1, link 2 and many more)

Dolphin Manage your files

Dolphin now sorts more naturally when comparing filenames by excluding extensions. So now "a.txt" appears before "a 2.txt". (Eren Karakas, 24.12.0. Link)

Haruna Media player

Left-clicking on a video now plays or pauses it by default. (Nate Graham, 25.04.0. Link)

Karp KDE arranger for PDFs

Karp is a new PDF arranger and editor by Tomasz Bojczuk which has just finished the incubation phase.

Tomasz was active this week and added an option to select the PDF version of the resulting PDF (Link) and made it possible to move multiple pages at the same time (Link).

Kate Advanced Text Editor

The build plugin — which allows you to trigger a rebuild from Kate's interface — now supports multiple projects being open at the same time without having to constantly reload the list of targets every time you switch projects. (Waqar Ahmed, 25.04.0. Link)

The ctag indexing doesn't happen anymore on the root and home folder as it makes no sense and just wastes CPU cycles. (Waqar Ahmed, 24.12.0. Link)

Fix getting a PATH when launching Kate outside of the console on macOS. (Waqar Ahmed, 24.12.0. Link)

KMyMoney Personal finance manager based on double-entry bookkeeping

It is no longer possible to apply category filters on the "Net Worth report" of KMyMoney as this was resulting in erroneous results. (Thomas Baumgart, 5.2, Link)

KRDC Connect with RDP or VNC to another computer

Fix loading the gateway server address in the settings. (Fabio Bas, 24.12.0. Link)

Fix a crash when scrolling; the app was previously sending empty scroll events which were rejected by the remote server. (Fabio Bas, 24.12.0. Link)

Konqueror KDE File Manager & Web Browser

Stefano fixed various issues with the Plasma Activities integration inside Konqueror. We now, for example, wait for the Plasma Activity service to be ready, before restoring activities when starting Konqueror. (Stefano Crocco, 24.12.0. Link)

Konsole Use the command line interface

We added the Campbell color scheme from Microsoft. (Mingcong Bai, 25.04.0. Link)

We fixed a few font rendering issues in Konsole. (Matan Ziv-Av, 24.12.0. Link)

Okular View and annotate documents

Changed the default value of the "scroll overlap" feature from 0% to 10%, which means that when you scroll down in a document using Page Down or Space Bar, the bottom 10% of the previous page will remain visible at the top of the view. This helps you retain your spatial awareness when quickly navigating. (David Cerenius, 25.04.0. Link)

Merkuro Calendar Manage your tasks and events with speed and ease

Claudio fixed many issues with the day and month views. Now, clicking on a day in the month view will open the day view on the selected day and not just a random one, the current day will be correctly highlighted, some sizing issues are fixed, and the month view won't appear as disabled anymore in some situations. (Claudio Cambra, 24.12.0. Link 1, link 2 and link 3)

When double-clicking on an empty space in the month view, the incidence editor will use the selected date as its start date. (Claudio Cambra, 24.12.0. Link)

NeoChat Chat on Matrix

We fixed the sed-edit feature in NeoChat, which allows you to type a sed expression like s/foo/bar to edit your previous message. (James Graham, 24.12.0. Link)

On mobile devices, NeoChat won't open the space homepage when trying to just switch the selected space. (James Graham, 24.12.0. Link)

Implemented MSC4228: Search Redirection to harmlessly redirect searches for harmful and potentially illegal content.

OptiImage Image optimizer to reduce the size of images

It is now possible to remove an image from the list of images to optimize. (Soumyadeep Ghosh, Link)

Soumyadeep also fixed an issue where it was possible to add the same image multiple times (Soumyadeep Ghosh, Link)

Skanpage Scan multi-page documents and images

Ported the export dialog in Skanpage to use Kirigami.Dialog, giving it a nicer and more consistent appearance. (Thomas Duckworth, 25.04.0. Link)

Ported Skanpage to use KIO, which allows saving scanned documents to remote folders. (Alexander Stippich, 25.04.0. Link)

Telly Skout A convergent Kirigami TV guide

Fix the list of favorite TV channels being empty after opening a different page. (Plata Hill, 24.12.0. Link)

Tokodon Browse the Fediverse

Joshua added titles to the profile pages, so that it is not empty anymore. (Joshua Goins, 24.12.0. Link)

Quote post are now better detected. (Joshua Goins, 24.12.0. Link)

It is now possible to start a new chat from the conversation page. (Joshua Goins, 24.12.0. Link)

…And Everything Else

This blog only covers the tip of the iceberg! If you’re hungry for more, check out Nate's blog about Plasma and be sure not to miss his This Week in Plasma series, where every Saturday he covers all the work being put into KDE's Plasma desktop environment.

For a complete overview of what's going on, visit KDE's Planet, where you can find all KDE news unfiltered directly from our contributors.

Get Involved

The KDE organization has become important in the world, and your time and contributions have helped us get there. As we grow, we're going to need your support for KDE to become sustainable.

You can help KDE by becoming an active community member and getting involved. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine! You don’t have to be a programmer either. There are many things you can do: you can help hunt and confirm bugs, even maybe solve them; contribute designs for wallpapers, web pages, icons and app interfaces; translate messages and menu items into your own language; promote KDE in your local community; and a ton more things.

You can also help us by donating. Any monetary contribution, however small, will help us cover operational costs, salaries, travel expenses for contributors and in general just keep KDE bringing Free Software to the world.

To get your application mentioned here, please ping us in invent or in Matrix.

Twig & PHP Templating in Drupal 11 Editor Mon, 11/25/2024 - 10:11

Welcome here! This is your complete guide to Twig and PHP templating in Drupal 11. As you understand,  Twig and PHP are important for the frontend and backend development in Drupal. If you know how they work, you can create beautiful, effective, and easy-to-maintain websites. 
Today, Golems Drupal company explores Twig's smooth template engine and PHP's strong backend logic. Our blog will be helpful for every kind of person, whether you are a skilled Drupal developer, someone who owns a website or business owner, or simply starting your path in this field. We will dive into the details of Twig and PHP in Drupal to help you better understand how they work together so that your digital experiences can be crafted more effectively.

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong><a href="https://talkpython.fm/blog/posts/talk-python-rewritten-in-quart-async-flask/?featured_on=pythonbytes">Talk Python rewritten in Quart</a></strong></li> <li><strong><a href="https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-digital-attestations/?featured_on=pythonbytes">PyPI now supports digital attestations</a></strong></li> <li><strong><a href="https://github.com/LilyFoote/django-rusty-templates?featured_on=pythonbytes">Django Rusty Templates</a></strong></li> <li><a href="https://discuss.python.org/t/pep-639-round-3-improving-license-clarity-with-better-package-metadata/53020/128?featured_on=pythonbytes"><strong>PEP 639 is now supported by PYPI</strong></a></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=Qk1b0AiZCYo' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="411">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/?featured_on=pythonbytes"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://courses.pythontest.com/p/the-complete-pytest-course?featured_on=pythonbytes"><strong>The Complete pytest Course</strong></a></li> <li><a href="https://www.patreon.com/pythonbytes"><strong>Patreon Supporters</strong></a></li> </ul> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a> <strong>/</strong> <a href="https://bsky.app/profile/mkennedy.codes?featured_on=pythonbytes"><strong>@mkennedy.codes</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a> <strong>/</strong> <a href="https://bsky.app/profile/brianokken.bsky.social?featured_on=pythonbytes"><strong>@brianokken.bsky.social</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a> <strong>/</strong> <a href="https://bsky.app/profile/pythonbytes.bsky.social"><strong>@pythonbytes.bsky.social</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Monday</strong> at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</p> <p><strong>Michael #1:</strong> <a href="https://talkpython.fm/blog/posts/talk-python-rewritten-in-quart-async-flask/?featured_on=pythonbytes">Talk Python rewritten in Quart</a></p> <ul> <li>Rewrote all of <a href="https://talkpython.fm/?featured_on=pythonbytes">talkpython.fm</a> in <a href="https://quart.palletsprojects.com/en/latest/?featured_on=pythonbytes">Quart</a> (10k lines of code total, 4k changed)</li> <li>Considered <ul> <li>FastAPI</li> <li>Litestar</li> <li>Django</li> <li>Hugo Static Site + Python</li> <li>Flask</li> </ul></li> <li>Discussed the multistage upgrade / conversion process</li> <li>Automating tests for all 1,000 pages</li> </ul> <p><strong>Brian #2:</strong> <a href="https://blog.pypi.org/posts/2024-11-14-pypi-now-supports-digital-attestations/?featured_on=pythonbytes">PyPI now supports digital attestations</a></p> <ul> <li>Dustin Ingram</li> <li>“<strong>Attestations provide a verifiable link to an upstream source repository:</strong> By signing with the identity of the upstream source repository, such as in the case of an upload of a project built with GitHub Actions, PyPI's support for digital attestations defines a strong and verifiable association between a file on PyPI and the source repository, workflow, and even the commit hash that produced and uploaded the file. Additionally, publishing attestations to a transparency log helps mitigate against both compromise of PyPI and compromise of the projects themselves.”</li> <li>For maintainers <ul> <li>If using GH Actions and Trusted Publishing <ul> <li>make sure you use pypa/gh-action-pypi-publish, version v1.11.0 or newer</li> <li>that’s it</li> </ul></li> <li>If not <ul> <li>“Support for automatic attestation generation and publication from other Trusted Publisher environments <a href="https://github.com/pypi/warehouse/issues/17001?featured_on=pythonbytes">is planned</a>.”</li> <li>“While not recommended, maintainers can also <a href="https://docs.pypi.org/attestations/producing-attestations/#the-manual-way">manually</a> <a href="https://docs.pypi.org/attestations/producing-attestations/#the-manual-way">generate and publish attestations</a>.”</li> </ul></li> </ul></li> <li>See also <ul> <li><a href="https://socket.dev/blog/pypi-introduces-digital-attestations?featured_on=pythonbytes">PyPI Introduces Digital Attestations to Strengthen Python Package Security</a> <ul> <li>by Sarah Gooding</li> </ul></li> <li><a href="https://trailofbits.github.io/are-we-pep740-yet/?featured_on=pythonbytes">Are we PEP 740 yet?</a></li> </ul></li> </ul> <p><strong>Michael #3:</strong> <a href="https://github.com/LilyFoote/django-rusty-templates?featured_on=pythonbytes">Django Rusty Templates</a></p> <ul> <li>by Lily Foote</li> <li>An experimental reimplementation of Django's templating language in Rust.</li> <li>Goals <ul> <li>100% compatibility of rendered output.</li> <li>Error reporting that is at least as useful as Django's errors.</li> <li>Improved performance over Django's pure Python implementation.</li> </ul></li> </ul> <p><strong>Brian #4:</strong> <a href="https://discuss.python.org/t/pep-639-round-3-improving-license-clarity-with-better-package-metadata/53020/128?featured_on=pythonbytes"><strong>PEP 639 is now supported by PYPI</strong></a></p> <ul> <li><a href="https://bsky.app/profile/snarky.ca/post/3laudpvnabc27?featured_on=pythonbytes">from Brett Cannon</a></li> <li><a href="https://peps.python.org/pep-0639/?featured_on=pythonbytes">PEP 639 – Improving License Clarity with Better Package Metadata</a></li> <li>For project metadata, use these fields: license and license-files:</li> <li><p>Examples license field</p> <pre><code>[project] license = "MIT" [project] license = "MIT AND (Apache-2.0 OR BSD-2-clause)" [project] license = "MIT OR GPL-2.0-or-later OR (FSFUL AND BSD-2-Clause)" [project] license = "LicenseRef-Proprietary" </code></pre></li> <li><p>Examples of <code>license-files</code>:</p> <pre><code>[project] license-files = ["LICEN[CS]E*", "AUTHORS*"] [project] license-files = ["licenses/LICENSE.MIT", "licenses/LICENSE.CC0"] [project] license-files = ["LICENSE.txt", "licenses/*"] [project] license-files = [] </code></pre></li> </ul> <p>Extras </p> <p>Brian:</p> <ul> <li>Playground Wisdom: Threads Beat Async/Await - interesting read from Armin Ronacher about different language abstractions around concurrency.</li> <li>PythonTest.com Discord community is now live <ul> <li>Launched last week, as of this morning we’ve got 89 members</li> <li>Anyone already a pythontest community member has received an invite</li> <li>Anyone can join through courses.pythontest.com</li> </ul></li> <li>Everything at pythontest.com is 20% off through Dec with code turkeysale2024</li> <li>“Python Testing with pytest” eBook 40% off through Dec 2, use code turkeysale2024</li> </ul> <p>Michael:</p> <ul> <li>Python 3.14.0a2 released</li> <li>Starter packs: <ul> <li>Michael’s Python people: https://bsky.app/starter-pack/mkennedy.codes/3lbdnupl26e2x </li> <li>Directory: https://blueskydirectory.com/starter-packs/all</li> </ul></li> </ul> <p>Joke: <a href="https://www.youtube.com/watch?v=atcqMWqB3hw">curl - heavy metal style</a>!</p>

SSH commands as API microservices 2024-11-25, by Dariusz Suchojad

This is a quick guide on how to turn SSH commands into a REST API service. The use-case may be remote administration of devices or equipment that does not offer a REST interface or making sure that access to SSH commands is restricted to selected external REST-based API clients only.

Python

The first thing needed is code of the service that will connect to SSH servers. Below is a service doing just that - it receives name of the command to execute and host to run in on, translating stdout and stderr of SSH commands into response documents which Zato in turn serializes to JSON.

# -*- coding: utf-8 -*- # stdlib from traceback import format_exc # Zato from zato.server.service import Service class SSHInvoker(Service): """ Accepts an SSH command to run on a remote host and returns its output to caller. """ # A list of elements that we expect on input input = 'host', 'command' # A list of elements that our responses will contain output = 'is_ok', 'cid', '-stdout', '-stderr' def handle(self): # Local aliases host = self.request.input.host command = self.request.input.command # Correlation ID is always returned self.response.payload.cid = self.cid try: # Build the full command full_command = f'ssh {host} {command}' # Run the command and collect output output = self.commands.invoke(full_command) # Assign both stdout and stderr to response self.response.payload.stdout = output.stdout self.response.payload.stderr = output.stderr except Exception: # Catch any exception and log it self.logger.warn('Exception caught (%s), e:`%s', self.cid, format_exc()) # Indicate an error self.response.payload.is_ok = False else: # Everything went fine self.response.payload.is_ok = True Dashboard

In the Zato Dashboard, let's go ahead and create an HTTP Basic Auth definition that a remote API client will authenticate against:

Now, the SSH service can be mounted on a newly created REST channel - note the security definition used and that data format is set to JSON. We can skip all the other details such as caching or rate limiting, for illustration purposes, this is not needed.

Usage

At this point, everything is ready to use. We could make it accessible to external API clients but, for testing purposes, let's simply invoke our SSH API gateway service from the command line:

$ curl "api:password@localhost:11223/api/ssh" -d \ '{"host":"localhost", "command":"uptime"}' { "is_ok": true, "cid": "27406f29c66c2ab6296bc0c0", "stdout": " 09:45:42 up 37 min, 1 user, load average: 0.14, 0.27, 0.18\n"} $ Note that, at this stage, the service should be used in trusted environments only, e.g. it will run any command that it is given on input which means that in the next iteration it could be changed to only allow commands from an allow-list, rejecting anything that is not recognized.

And this completes it - the service is deployed and made accessible via a REST channel that can be invoked using JSON. Any command can be sent to any host and their output will be returned to API callers in JSON responses.

More resources

➤ Python API integration tutorial
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

I've just released version 1.1.23 of plocate, almost a year after 1.1.22. The changes are mostly around the systemd unit this time, but perhaps more interestingly is that this is the first release where I don't have the majority of patches; in fact, I don't have any patches at all. All of them came from contributors, many of them through the “just do git push to send me a patch email” interface.

I guess this means that I'll need to actually start streamlining my “git am” workflow… it gets me every time. :-)

You can get download numbers for PyPI packages (or projects) from a Google BigQuery dataset. You need a Google account and credentials, and Google gives 1 TiB of free quota per month.

Each month, I have automation to fetch the download numbers for the 8,000 most popular packages over the past 30 days, and make it available as more accessible JSON and CSV files at Top PyPI Packages. This data is widely used for research in academia and industry.

However, as more packages and releases are uploaded to PyPI, and there are more and more downloads logged, the amount of billed data increases too.

This chart shows the amount of data billed per month.

At first, I was only collecting downloads data for 4,000 packages, and it was fetched for two queries: downloads over 365 days and over 30 days. But as time passed, it started using up too much quota to download data for 365 days.

So I ditched the 365-day data, and increased the 30-day data from 4,000 to 5,000 packages. Later, I checked how much quota was being used and increased from 5,000 packages to 8,000 packages.

But then I exceeded the BigQuery monthly quota of 1 TiB fetching data for July 2024.

To fetch the missing data and investigate what's going in, I started Google Cloud's 90-day, $300 (€277.46) free-trial 💸

Here's what I found!

Finding: it costs more to get data for downloads from only pip than from all installers

I use the pypinfo client to help query BigQuery. By default, it only fetches downloads for pip.

Only pip

This command gets one day's download data for the top 10 packages, for pip only:

$ pypinfo --limit 10 --days 1 "" project Served from cache: False Data processed: 58.21 GiB Data billed: 58.21 GiB Estimated cost: $0.29

Results:

project download count boto3 37,251,744 aiobotocore 16,252,824 urllib3 16,243,278 botocore 15,687,125 requests 13,271,314 s3fs 12,865,055 s3transfer 12,014,278 fsspec 11,982,305 charset-normalizer 11,684,740 certifi 11,639,584 Total 158,892,247 All installers

Adding the --all flag gets one day's download data for the top 10 packages, for all installers:

$ pypinfo --all --limit 10 --days 1 "" project Served from cache: False Data processed: 46.63 GiB Data billed: 46.63 GiB Estimated cost: $0.23 project download count boto3 39,495,624 botocore 17,281,187 urllib3 17,225,121 aiobotocore 16,430,826 requests 14,287,965 s3fs 12,958,516 charset-normalizer 12,781,405 certifi 12,647,098 setuptools 12,608,120 idna 12,510,335 Total 168,226,197

So we can see the default pip-only costs an extra 25% data processed and data billed, and costs an extra 25% in dollars.

Unsurprisingly, the actual download counts are higher for all installers. The ranking has changed a bit, but I expect we're still getting more-or-less the same packages in the top thousands of results.

Queries

It sends a query like this to BigQuery for only pip:

SELECT file.project as project, COUNT(*) as download_count, FROM `bigquery-public-data.pypi.file_downloads` WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY) AND details.installer.name = "pip" GROUP BY project ORDER BY download_count DESC LIMIT 10

And for all installers:

SELECT file.project as project, COUNT(*) as download_count, FROM `bigquery-public-data.pypi.file_downloads` WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY) GROUP BY project ORDER BY download_count DESC LIMIT 10

These queries are the same, except the default has an extra AND details.installer.name = "pip" condition. It seems reasonable it would cost more to do extra filtering work.

Installers

Let's look at the installers:

$ pypinfo --all --limit 100 --days 1 "" installer Served from cache: False Data processed: 29.49 GiB Data billed: 29.49 GiB Estimated cost: $0.15 installer name download count pip 1,121,198,711 uv 117,194,833 requests 29,828,272 poetry 23,009,454 None 8,916,745 bandersnatch 6,171,555 setuptools 1,362,797 Bazel 1,280,271 Browser 1,096,328 Nexus 593,230 Homebrew 510,247 Artifactory 69,063 pdm 62,904 OS 13,108 devpi 9,530 conda 2,272 pex 194 Total 1,311,319,514

pip still by far the most popular, and unsurprising uv is up there too, with about 10% of pip's downloads.

The others are about 25% or less of uv. A lot of them are mirroring services that we wanted to exclude before.

I think given uv's importance, and my expectation that it will continue to take a bigger share of the pie, plus especially the extra cost for filtering by just pip, means that we should switch to fetching data for all downloaders. Plus the others don't account for that much of the pie.

Finding: the number of packages doesn't affect the cost

This was the biggest surprise. Earlier I'd been increasing or decreasing the number to try and remain under quota. But it turns out it makes no difference how many packages you query!

I fetched data for just one day and all installers for different package limits: 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000. Sample query:

SELECT file.project as project, COUNT(*) as download_count, FROM `bigquery-public-data.pypi.file_downloads` WHERE timestamp BETWEEN TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -2 DAY) AND TIMESTAMP_ADD(CURRENT_TIMESTAMP(), INTERVAL -1 DAY) GROUP BY project ORDER BY download_count DESC LIMIT 8000

Result: Interestingly, the cost is the same for all limits (1000-8000): $0.31.

Repeating with one day but filtering for pip only:

Result: Cost increased to $0.39 but again the same for all limits.

Let's repeat with all installers, but for 30 days, and this time query in decreasing limits, in case we were only paying for incremental changes: 8000, 7000, 6000, 5000, 4000, 3000, 2000, 1000:

Result: Again, the cost is the same regardless of package limit: $4.89 per query.

Well then, let's repeat with the limit increasing by powers of ten, up to 1,000,000! This last one fetches data for all 531,022 packages on PyPI:

limit projects count estimated cost bytes billed bytes processed 1 1 0.20 43,447,746,560 43,447,720,943 10 10 0.20 43,447,746,560 43,447,720,943 100 100 0.20 43,447,746,560 43,447,720,943 1000 1,000 0.20 43,447,746,560 43,447,720,943 8000 8,000 0.20 43,447,746,560 43,447,720,943 10000 10,000 0.20 43,447,746,560 43,447,720,943 100000 100,000 0.20 43,447,746,560 43,447,720,943 1000000 531,022 0.20 43,447,746,560 43,447,720,943

Result: Again, same cost, whether for 1 package or 531,022 packages!

Finding: the number of days affects the cost

No surprise. I'd earlier noticed 365 days too took much quota, and I could continue with 30 days.

Here's the estimated cost and bytes billed (for one package, all installers) between one and 30 days (f"pypinfo --all --json --indent 0 --days {days} --limit 1 '' project"), showing a roughly linear increase:

Conclusion

• It doesn't matter how many packages I fetch data for, I might as well fetch all and make it available to everyone, depending on the size of the data file. It will make sense to still offer a smaller file with 8,000 or so packages: often you just need a large-ish yet manageable number.

• It costs more to filter for only downloads from pip, so I've switched to fetching data for all installers.

• The number of days affects the cost, so I will need to decrease this in the future to stay within quota. For example, at some point I may need to switch from 30 to 25 days, and later from 25 to 20 days.

More details from the investigation, the scripts and data files can be found at
hugovk/top-pypi-packages#36.

And let me know if you know any tricks to reduce costs!

Header photo: "The Balancing Rock, Stonehenge, Near Glen Innes, NSW" by the Royal Australian Historical Society, with no known copyright restrictions.