Feeds

Exciting news! Tag1 Consulting is proud to be a module sponsor at DrupalCon Barcelona 2024. Join us from September 24-27 for four days of Drupal innovation, collaboration, and community spirit. Our team will be presenting on Gander, Drupal Core development, LMS, DDEV, and more.

Read more Hank Wed, 08/28/2024 - 10:14

In this article, we delve into the process of migrating Drupal fields, building on the knowledge from previous discussions about Drupal fields and their database structures. We begin by addressing the two key components of field migrations: storage and instance settings. This is the first step in a multi-stage migration process that will ultimately involve four different migrations.

Read more mauricio Wed, 08/28/2024 - 07:41

Scrapy is a robust Python web scraping framework that can manage requests asynchronously, follow links, and parse site content. To store scraped data, you can use MongoDB, a scalable NoSQL database, that stores data in a JSON-like format. Combining Scrapy with MongoDB offers a powerful solution for web scraping projects, leveraging Scrapy’s efficiency and MongoDB’s flexible data storage.

In this tutorial, you’ll learn how to:

• Set up and configure a Scrapy project
• Build a functional web scraper with Scrapy
• Extract data from websites using selectors
• Store scraped data in a MongoDB database
• Test and debug your Scrapy web scraper

If you’re new to web scraping and you’re looking for flexible and scalable tooling, then this is the right tutorial for you. You’ll also benefit from learning this tool kit if you’ve scraped sites before, but the complexity of your project has outgrown using Beautiful Soup and Requests.

To get the most out of this tutorial, you should have basic Python programming knowledge, understand object-oriented programming, comfortably work with third-party packages, and be familiar with HTML and CSS.

By the end, you’ll know how to get, parse, and store static data from the Internet, and you’ll be familiar with several useful tools that allow you to go much deeper.

Get Your Code: Click here to download the free code that shows you how to gather Web data with Scrapy and MongoDB.

Take the Quiz: Test your knowledge with our interactive “Web Scraping With Scrapy and MongoDB” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Web Scraping With Scrapy and MongoDB

In this quiz, you'll test your understanding of web scraping with Scrapy and MongoDB. You'll revisit how to set up a Scrapy project, build a functional web scraper, extract data from websites, store scraped data in MongoDB, and test and debug your Scrapy web scraper.

Prepare the Scraper Scaffolding

You’ll start by setting up the necessary tools and creating a basic project structure that will serve as the backbone for your scraping tasks.

While working through the tutorial, you’ll build a complete web scraping project, approaching it as an ETL (Extract, Transform, Load) process:

• Extract data from the website using a Scrapy spider as your web crawler.
• Transform this data, for example by cleaning or validating it, using an item pipeline.
• Load the transformed data into a storage system like MongoDB with an item pipeline.

Scrapy provides scaffolding for all of these processes, and you’ll tap into that scaffolding to learn web scraping following the robust structure that Scrapy provides and that numerous enterprise-scale web scraping projects rely on.

Note: In a Scrapy web scraping project, a spider is a Python class that defines how to crawl a specific website or a group of websites. It contains the logic for making requests, parsing responses, and extracting the desired data.

First, you’ll install Scrapy and create a new Scrapy project, then explore the auto-generated project structure to ensure that you’re well-equipped to proceed with building a performant web scraper.

Install the Scrapy Package

To get started with Scrapy, you first need to install it using pip. Create and activate a virtual environment to keep the installation separate from your global Python installation. Then, you can install Scrapy:

Shell (venv) $ python -m pip install scrapy Copied!

After the installation is complete, you can verify it by running the scrapy command and viewing the output:

Shell (venv) $ scrapy Scrapy 2.11.2 - no active project Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [ more ] More commands available when run from project directory Use "scrapy <command> -h" to see more info about a command Copied!

The command-line (CLI) program should display the help text of Scrapy. This confirms that you installed the package correctly. You’ll next run the highlighted startproject command to create a project.

Create a Scrapy Project

Scrapy is built around projects. Generally, you’ll create a new project for each web scraping project that you’re working on. In this tutorial, you’ll work on scraping a website called Books to Scrape, so you can call your project books.

As you may have already identified in the help text, the framework provides a command to create a new project:

Shell (venv) $ scrapy startproject books Copied! Read the full article at https://realpython.com/web-scraping-with-scrapy-and-mongodb/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

The Open Source Initiative (OSI) is running a blog series to introduce some of the people who have been actively involved in the Open Source AI Definition (OSAID) co-design process. The co-design methodology allows for the integration of diverging perspectives into one just, cohesive and feasible standard. Support and contribution from a significant and broad group of stakeholders is imperative to the Open Source process and is proven to bring diverse issues to light, deliver swift outputs and garner community buy-in.

This series features the voices of the volunteers who have helped shape and are shaping the Definition.

Meet Ezequiel Lanza

What’s your background related to Open Source and AI?

I’ve been working in AI for more than 10 years (Yes, before ChatGPT!). With a background in engineering, I’ve consistently focused on building and supporting AI applications, particularly in machine learning and data science. Over the years, I’ve contributed to and collaborated on various projects. A few years ago, I decided to pursue a master’s in data science to deepen my theoretical knowledge and further enhance my skills. Open Source has also been a significant part of my work; the frameworks, tools and community have continually drawn me in, making me an active participant in this evolving conversation for years.

What motivated you to join this co-design process to define Open Source AI?

AI owes much of its progress to Open Source, and it’s essential for continued innovation. My experience in both AI and Open Source spans many years, and I believe this co-design process offers a unique chance to contribute meaningfully. It’s not just about sharing my insights but also about learning from other professionals across AI and different disciplines. This collective knowledge and diverse perspectives make this initiative truly powerful and enriching, to shape the future of Open Source AI together.

Can you describe your experience participating in this process? What did you most enjoy about it, and what were some of the challenges you faced?

Participating in this process has been both rewarding and challenging. I’ve particularly enjoyed engaging with diverse groups and hearing different perspectives. The in-person events, such as All Things Open in Raleigh in 2023, have been valuable for fostering direct collaboration and building relationships. However, balancing these meetings with my work duties has been challenging. Coordinating schedules and managing time effectively to attend all the relevant discussions can be demanding. Despite these challenges, the insights and progress have made the effort worthwhile.

Why do you think AI should be Open Source?

We often say AI is everywhere, and while that’s partially true, I believe AI will be everywhere, significantly impacting our lives. However, AI’s full potential can only be realized if it is open and accessible to everyone. Open Source AI should also foster innovation by enabling developers and researchers from all backgrounds to contribute to and improve existing models, frameworks and tools, allowing freedom of expression. Without open access, involvement in AI can be costly, limiting participation to only a few large companies. Open Source AI should aim to democratize access, allowing small businesses, startups and individuals to leverage powerful tools that might otherwise be out of reach due to cost or proprietary barriers.

What do you think is the role of data in Open Source AI?

Data is essential for any AI system. Initially, from my ML bias perspective, open and accessible datasets were crucial for effective ML development. However, I’ve reevaluated this perspective, considering how to adapt the system while staying true to Open Source principles. As AI models, particularly GenAI like LLMs, become increasingly complex, I’ve come to value the models themselves. For example, Generative AI requires vast amounts of data, and gaining access to this data can be a significant challenge.

This insight has led me to consider what I—whether as a researcher, developer or user—truly need from a model to use/investigate it effectively. While understanding the data used in training is important, having access to specific datasets may not always be necessary. In approaches like federated learning, the model itself can be highly valuable while keeping data private, though understanding the nature of the data remains important. For LLMs, techniques such as fine-tuning, RAG and RAFT emphasize the benefits of accessing the model rather than the original dataset, providing substantial advantages to the community.

Sharing model architecture and weights is crucial, and data security can be maintained through methods like model introspection and fine-tuning, reducing the need for extensive dataset sharing.

Data is undoubtedly a critical component. However, the essence of Open Source AI lies in ensuring transparency, then the focus should be on how data is used in training models. Documenting which datasets were used and the data handling processes is essential. This transparency helps the community understand the origins of the data, assess potential biases and ensure the responsible use of data in model development. While sharing the exact datasets may not always be necessary, providing clear information about data sources and usage practices is crucial for maintaining trust and integrity in Open Source AI.

Has your personal definition of Open Source AI changed along the way? What new perspectives or ideas did you encounter while participating in the co-design process?

Of course, it changed and evolved – that’s what a thought process is about. I’d be stubborn if I never changed my perspective along the way. I’ve often questioned even the most fundamental concepts I’ve relied on for years, avoiding easy or lazy assumptions. This thorough process has been essential in refining my understanding of Open Source AI. Engaging in meaningful exchanges with others has shown me the importance of practical definitions that can be implemented in real-world scenarios. While striving for an ideal, flawless definition is tempting, I’ve found that embracing a pragmatic approach is ultimately more beneficial.

What do you think the primary benefit will be once there is a clear definition of Open Source AI?

As I see it, the Open Source AI Definition will support the growth, and it will be the first big step. The primary benefit of having a clear definition of Open Source AI will be increased clarity and consistency in the field. This will enhance collaboration by setting clear standards and expectations for researchers, developers and organizations. It will also improve transparency by ensuring that AI models and tools genuinely follow Open Source principles, fostering trust in their development and sharing.

A clear definition will create standardized practices and guidelines, making it easier to evaluate and compare different Open Source AI projects.

What do you think are the next steps for the community involved in Open Source AI?

The next steps for the community should start with setting up a certification process for AI models to ensure they meet certain standards. This could include tools to help automate the process. After that, it would be helpful to offer templates and best practice guides for AI models. This will support model designers in creating high-quality, compliant systems and make the development process smoother and more consistent.

How to get involved

The OSAID co-design process is open to everyone interested in collaborating. There are many ways to get involved:

• Join the forum: share your comment on the drafts.
• Leave comment on the latest draft: provide precise feedback on the text of the latest draft.
• Follow the weekly recaps: subscribe to our monthly newsletter and blog to be kept up-to-date.
• Join the town hall meetings: we’re increasing the frequency to weekly meetings where you can learn more, ask questions and share your thoughts.
• Join the workshops and scheduled conferences: meet the OSI and other participants at in-person events around the world.

Drupal GovCon wrapped up with actionable insights from Rod Martin on non-coding solutions for site builders and Bree Benesh on navigating leadership challenges. Explore their sessions for practical strategies to apply in your Drupal projects.

We are looking for the next group of organizers to own and lead the 2026 DjangoCon Europe conference. Could your town - or your football stadium, circus tent, private island or city hall - host this wonderful community event?

DjangoCon Europe is a major pillar of the Django community, as people from across the world meet and share. This includes many qualities that make it a unique event - unconventional and conventional venues, creative happenings, a feast of talks and a dedication to inclusion and diversity.

Hosting a DjangoCon is an ambitious undertaking. It's hard work, but each year it has been successfully run by a team of community volunteers, not all of whom have had previous experience - more important is enthusiasm, organizational skills, the ability to plan and manage budgets, time and people - and plenty of time to invest in the project.

For 2026, we want to kickstart the organization much earlier than in previous years to allow more flexibility for the organizing team, and open up more opportunities for support from our DjangoCon Europe support working group.

Step 1: Submit your expression of interest

If you’re considering organizing DjangoCon Europe (🙌 great!), fill in our DjangoCon Europe 2026 expression of interest form with your contact details. No need to fill in all the information at this stage if you don’t have it all already, we’ll reach out and help you figure it out.

Express your interest in organizing

Step 2: We’re here to help!

We've set up a DjangoCon Europe support working group of previous organizers that you can reach out to with questions about organizing and running a DjangoCon Europe.

The group will be in touch with everyone submitting the expression of interest form, or you can reach out to them directly: european-organizers-support@djangoproject.com

We'd love to hear from you as soon as possible, so your proposal can be finalized and sent to the DSF board by October 6th 2024. The selected hosts will be publicly announced at DjangoCon Europe 2025 by the current organizers.

Step 3: Submitting the proposal

The more detailed and complete your final proposal is, the better. Basic details include:

• Organizing committee members: You won’t have a full team yet, probably, naming just some core team members is enough.
• The legal entity that is intended to run the conference: Even if the entity does not exist yet, please share how you are planning to set it up.
• Dates: See “What dates are possible in 2026?” below. We must avoid conflicts with major holidays, EuroPython, DjangoCon US, and PyCon US.
• Venue(s), including size, number of possible attendees, pictures, accessibility concerns, catering, etc.
• Transport links and accommodation: Can your venue be reached by international travelers?
• Budgets and ticket prices: Talk to the DjangoCon Europe Support group to get help with this, including information on past event budgets.

We also like to see:

• Timelines
• Pictures
• Plans for online participation, and other ways to make the event more inclusive and reduce its environmental footprint
• Draft agreements with providers
• Alternatives you have considered

Have a look at our proposed DjangoCon Europe 2026 Licensing Agreement for the fine print on contractual requirements and involvement of the Django Software Foundation.

Submit your completed proposal by October 6th 2024 via our DjangoCon Europe 2026 expression of interest form, this time filling in as many fields as possible. We look forward to reviewing great proposals that continue the excellence the whole community associates with DjangoCon Europe.

Q&A Can I organize a conference alone?

We strongly recommend that a team of people submit an application.

I/we don’t have a legal entity yet, is that a problem?

Depending on your jurisdiction, this is usually not a problem. But please share your plans about the entity you will use or form in your application.

Do I/we need experience with organizing conferences?

The support group is here to help you succeed. From experience, we know that many core groups of 2-3 people have been able to run a DjangoCon with guidance from previous organizers and help from volunteers.

What is required in order to announce an event?

Ultimately, a contract with the venue confirming the dates is crucial, since announcing a conference makes people book calendars, holidays, buy transportation and accommodation etc. This, however, would only be relevant after the DSF board has concluded the application process. Naturally, the application itself cannot contain any guarantees, but it’s good to check concrete dates with your venues to ensure they are actually open and currently available, before suggesting these dates in the application.

Do we have to do everything ourselves?

No. You will definitely be offered lots of help by the community. Typically, conference organizers will divide responsibilities into different teams, making it possible for more volunteers to join. Local organizers are free to choose which areas they want to invite the community to help out with, and a call will go out through a blog post announcement on djangoproject.com and social media.

What kind of support can we expect from the Django Software Foundation?

The DSF regularly provides grant funding to DjangoCon organizers, to the extent of $6,000 in recent editions. We also offer support via specific working groups:

• The dedicated DjangoCon Europe support working group.
• The social media working group can help you promote the event.
• The Code of Conduct working group works with all event organizers.

In addition, a lot of Individual Members of the DSF regularly volunteer at community events. If your team aren’t Individual Members, we can reach out to them on your behalf to find volunteers.

What dates are possible in 2026?

For 2026, DjangoCon Europe should happen between January 5th and April 27th, or June 4th and June 28th. This is to avoid the following community events’ provisional dates:

• PyCon US 2026: May 2026
• EuroPython 2026: July 2026
• DjangoCon US 2026: September - October 2026
• DjangoCon Africa 2026: August - September 2026

We also want to avoid the following holidays:

• New Year's Day: Wednesday 1st January 2026
• Chinese New Year: Tuesday 17th February 2026
• Eid Al-Fitr: Friday 20th March 2026
• Passover: Wednesday 1st - Thursday 9th April 2026
• Easter: Sunday 5th April 2026
• Eid Al-Adha: Tuesday 26th - Friday 29th May 2026
• Rosh Hashanah: Friday 11th - Sunday 13th September 2026
• Yom Kippur: Sunday 20th - Monday 21st September 2026

What cities or countries are possible?

Any city in Europe. This can be a city or country where DjangoCon Europe has happened in the past (Vigo, Edinburgh, Porto, Copenhagen, Heidelberg, Florence, Budapest, Cardiff, Toulon, Warsaw, Zurich, Amsterdam, Berlin), or a new locale.

References Past calls

• Interested in organizing DjangoCon Europe 2016?
• Could you host DjangoCon Europe 2017?
• DjangoCon Europe 2019 - where will it be?
• Could you host DjangoCon Europe 2023?
• Last Chance for a DjangoCon Europe 2023

PyPy v7.3.17: release of python 2.7 and 3.10

The PyPy team is proud to release version 7.3.17 of PyPy.

This release includes a new RISC-V JIT backend, an improved REPL based on work by the CPython team, and better JIT optimizations of integer operations. Special shout-outs to Logan Chien for the RISC-V backend work, to Nico Rittinghaus for better integer optimization in the JIT, and the CPython team that has worked on the repl.

The release includes two different interpreters:

• PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)

• PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.14.

The interpreters are based on much the same codebase, thus the dual release. This is a micro release, all APIs are compatible with the other 7.3 releases. It follows after 7.3.16 release on April 23, 2024.

We recommend updating. You can find links to download the releases here:

https://pypy.org/download.html

We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our blog via a pull request to https://github.com/pypy/pypy.org

We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: bug fixes, PyPy and RPython documentation improvements, or general help with making RPython's JIT even better.

If you are a python library maintainer and use C-extensions, please consider making a HPy / CFFI / cppyy version of your library that would be performant on PyPy. In any case, both cibuildwheel and the multibuild system support building wheels for PyPy.

RISC-V backend for the JIT

PyPy's JIT has added support for generating 64-bit RISC-V machine code at runtime (RV64-IMAD, specifically). So far we are not releasing binaries for any RISC-V platforms, but there are instructions on how to cross-compile binaries.

REPL Improvements

The biggest user-visible change of the release is new features in the repl of PyPy3.10. CPython 3.13 has adopted and extended PyPy's pure-Python repl, adding a number of features and fixing a number or bugs in the process. We have backported and added the following features:

• Prompts and tracebacks use terminal colors, as well as terminal hyperlinks for file names.

• Bracketed paste enable pasting several lines of input into the terminal without auto-indentation getting in the way.

• A special interactive help browser (F1), history browser (F2), explicit paste mode (F3).

• Support for Ctrl-<left/right> to jump over whole words at a time.

See the CPython documentation for further details. Thanks to Łukasz Langa, Pablo Galindo Salgado and the other CPython devs involved in this work.

Better JIT optimizations of integer operations

The optimizers of PyPy's JIT have become much better at reasoning about and optimizing integer operations. This is done with a new "knownbits" abstract domain. In many programs that do bit-manipulation of integers, some of the bits of the integer variables of the program can be statically known. Here's a simple example:

x = a | 1 ... if x & 1: ... else: ...

With the new abstract domain, the JIT can optimize the if-condition to True, because it already knows that the lowest bit of x must be set. This optimization applies to all Python-integers that fit into a machine word (PyPy optimistically picks between two different representations for int, depending on the size of the value). Unfortunately there is very little impact of this change on almost all Python code, because intensive bit-manipulation is rare in Python. However, the change leads to significant performance improvements in Pydrofoil (the RPython-based RISC-V/ARM emulators that are automatically generated from high-level Sail specifications of the respective ISAs, and that use the RPython JIT to improve performance).

PyPy versions and speed.pypy.org

The keen-eyed will have noticed no mention of Python version 3.9 in the releases above. Typically we will maintain only one version of Python3, but due to PyPy3.9 support on conda-forge we maintained multiple versions from the first release of PyPy3.10 in PyPy v7.3.12 (Dec 2022). Conda-forge is sunsetting its PyPy support, which means we can drop PyPy3.9. Since that was the major driver of benchmarks at https://speed.pypy.org, we revamped the site to showcase PyPy3.9, PyPy3.10, and various versions of cpython on the home page. For historical reasons, the "baseline" for comparison is still cpython 3.7.19.

We will keep the buildbots building PyPY3.9 until the end of August, these builds will still be available on the nightly builds tab of the buildbot.

What is PyPy?

PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.

We also welcome developers of other dynamic languages to see what RPython can do for them.

We provide binary builds for:

• x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)

• 64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).

PyPy supports Windows 32-bit, Linux PPC64 big- and little-endian, Linux ARM 32 bit, RISC-V RV64IMAFD Linux, and s390x Linux but does not release binaries. Please reach out to us if you wish to sponsor binary releases for those platforms. Downstream packagers provide binary builds for debian, Fedora, conda, OpenBSD, FreeBSD, Gentoo, and more.

What else is new?

For more information about the 7.3.17 release, see the full changelog.

Please update, and continue to help us make pypy better.

Cheers, The PyPy Team

Qt for MCUs 2.5.4 LTS (Long-Term Support) has been released and is available for download. This patch release provides bug fixes and other improvements while maintaining source compatibility with Qt for MCUs 2.5. It does not add any new functionality.

por Paulo Henrique de Lima Santana (phls)

Listamos abaixo os links para os relatos e notícias do Debian Day 2024 realizado em Belém e Poços de Caldas:

• Belém - PA
• Poços de Caldas - MG

Setting up a multilingual website in Drupal opens the door to a global online marketplace, allowing businesses to reach different cultural audiences by presenting content in multiple languages. In this article, we will discuss the process of configuring a multilingual site on Drupal.

Greetings, Pythonistas- thank you so much for supporting the work of the Python Software Foundation (PSF) and the Python community! The current PSF Board has decided to invest more in connecting and serving the global Python community by establishing a forum to have regular conversations. The board members of the PSF with the support of PSF staff are excited to introduce monthly PSF Board Office Hours on the PSF Discord. The Office Hours will be sessions where you can share with us how we can help your community, express your perspectives, and provide feedback for the PSF.

What are the PSF Board Office Hours?

Similar to the PSF Grants Program Office Hours where PSF staff members help to answer questions regarding the PSF Grants Program, during the PSF Board Office Hours you can participate in a text-based live chat with PSF Board Directors. This is a chance to connect, share, and collaborate with the PSF Board and staff to improve our community together. Occasionally, we will have dedicated topics such as PyCon US and the PSF Board Elections for the office hour sessions.

Here is some of the work that we collaborate with staff and volunteers on:

• Promotion and outreach for the Python programming language
• Supporting local Python communities
• Organizing PyCon US
• Diversity and Inclusion in our community
• Support handling of Code of Conduct within our communities
• Support regional Python communities via the PSF Grants Program
• Furthering the mission of the PSF

Unless we have a dedicated topic for a session, you are not limited to talking with us about the above topics, although the discussions should be focused on Python, the PSF, and our community. If you think there’s something we can help with or we should know, we welcome you to come and talk to us!

Joining the office hours

The office hour sessions will take place on the PSF Discord server in the #psf-board channel. If you are new to Discord, make sure to check out a tutorial on how you can download the Discord app and sign up for free– then join us on the PSF Discord! To make the office hours more accessible, the office hours will be scheduled at alternating times so no matter where you are based, you can find a time that is most convenient for you! Here is a list of the dates and times:

• September 10th, 2024: 1pm UTC
• October 8th, 2024: 9pm UTC
• November 12th, 2024: 2pm UTC
• December 10th, 2024: 9pm UTC
• January 14th, 2025: 2pm UTC
• February 11th, 2025: 9pm UTC
• March 11th, 2025: 1pm UTC
• April 8th, 2025:  9pm UTC
• May 13th, 2025: 1pm UTC (Live from PyCon US!)
• June 10th, 2025: 9pm UTC
• July 9th, 2025:  1pm UTC
• August 12th, 2025: 9pm UTC

Each session lasts for an hour. Make sure to check what time these sessions are for you locally so you don't miss out! Sessions after August 13th, 2025, will be announced in the future.
 

Who will be at the office hours?

Some of the board members of the PSF will be attending each office hour, as well as members of the PSF Staff. The list of the PSF Board Directors can be found on our website. We are passionate Python community members who are happy to listen, help, and provide support to you. We are happy to follow up with you if there are any issues we cannot address immediately during the office hour sessions. As always, you can email us at psf-board@python.org with inquiries, feedback, or comments at any time.

A while ago a colleague of mine asked about our crash infrastructure in Plasma and whether I could give some overview on it. This seems very useful to others as well, I thought. Here I am, telling you all about it!

Our crash infrastructure is comprised of a number of different components.

• KCrash: a KDE Framework performing crash interception and prepartion for handover to…
• coredumpd: a systemd component performing process core collection and handover to…
• DrKonqi: a GUI for crashes sending data to…
• Sentry: a web service and UI for tracing and presenting crashes for developers

We’ve looked at KCrash previously. This time we look at coredumpd.

Coredumpd

coredumpd collects all crashes happening on the system, through the core_pattern system. It is shipped as part of systemd and as such mostly available out of the box.

It is fairly sophisticated and can manage the backlog of crashes, so old crashes get cleaned out from time to time. It also tightly integrates with journald giving us a well-defined interface to access crash metadata.

But before we dive into the inner workings of coredumpd, let’s talk about cores.

What are cores?

A core, or more precisely: a core dump file, is a copy of the memory image of a process and its process status (registers, mappings, etc.) in a file. Simply put, it’s like we took a copy of the running process from RAM and stored it in a file. The purpose of such a core is that it allows us to look at a snapshot of the process at that point in time without having the process still running. Using this data, we can perform analysis of the process to figure out what exactly went wrong and how we ended up in that situation.

The advantage is that since the process doesn’t need to be running anymore, we can investigate crashes even hours or days after they happened. That is of particular use when things crash while we are not able to deal with them immediately. For example if Plasma were to crash on logout there’d be no way to deal with it besides stopping the logout, which may not even be possible anymore. Instead we let the crash drop into coredumpd, let it collect a core file, and on next login we can tell the user about the crash.

With that out of the way, it’s time to dump a core!

Core Dumps

We already talked about KCrash and how it intercepts crashes to write some metadata to disk. Once it is done it calls raise() to generate one of those core dumps we just discussed. This actually very briefly turns over control to the kernel which will more or less simply invoke the defined core_pattern process. In our case, coredumpd.

coredumpd will immediately systemd-socket-activate itself and forward the data received from the kernel. In other words: it will start an instance of systemd-coredump@.service and the actual processing will happen in there. The advantage of this is that regular systemd security configuration can be applied as well as cgroup resource control and all that jazz — the core dumping happens in a regular systemd service.

The primary task here is to actually write the dump to a file. In addition, coredumpd will also collect lots of additional metadata besides what is in the core already. Most notably various bits and pieces of /proc information such as cgroup information, mount information, the auxillary vector (auxv), etc.

Once all the data is collected a journald entry is written and the systemd-coredump@.service instance quits again.

The journal entry will contain the metadata as entry fields as well as the path of the core dump on disk, so we can later access it. It essentially serves as a key-value store for the crash data. A severely shortened version looks like this:

Tue 2024-08-27 17:52:27.593233 CEST […] COREDUMP_UID=60106 COREDUMP_GID=60106 COREDUMP_SIGNAL_NAME=SIGSYS COREDUMP_SIGNAL=31 COREDUMP_TIMESTAMP=1724773947000000 COREDUMP_COMM=wine64 COREDUMP_FILENAME=/var/lib/systemd/coredump/core.wine64.….zst … Example

Since this is all rather abstract, we can look at a trivial example to illustrate things a bit better.

Let’s open two terminals. In the first we can watch the journal for the crash to appear.

journalctl -xef SYSLOG_IDENTIFIER=systemd-coredump

In the second terminal we run an instance of sleep in the background, and then trigger a segmentation fault crash.

sleep 99999999999& kill -SEGV $!

In the first terminal you’ll see the crash happening:

Aug 27 15:01:49 ajax systemd-coredump[35535]: Process 35533 (sleep) of user 60106 terminated abnormally with signal 11/SEGV, processing... Aug 27 15:01:49 ajax systemd-coredump[35549]: [🡕] Process 35533 (sleep) of user 60106 dumped core. Stack trace of thread 35533: #0 0x0000729f1b961dc0 n/a (/lib/ld-linux-x86-64.so.2 + 0x1cdc0) ELF object binary architecture: AMD x86-64

So far so interesting. “But where is the additional data from /proc hiding?” you might wonder. We need to look at the verbose entry to see all data.

journalctl -o verbose SYSLOG_IDENTIFIER=systemd-coredump

This actually already concludes coredumpd’s work. In the next post DrKonqi will step onto the stage.

#644 – AUGUST 27, 2024
View in Browser »

Exploring Astrophysics in Python With pandas and Matplotlib

This course uses three problems often covered in introductory astro-physics courses to play in Python. Along the way you’ll learn some astronomy and how to use a variety of datascience libraries like NumPy, Matplotlib, pandas, and pint.
REAL PYTHON course

What’s the Deal With setup.py, pyproject.toml and Wheels?

Packaging in Python has a bit of a history and if you’ve come across the variety of ways of specifying and building packages you might wonder “why?” This article gives you the history and current best practices.
BITECODE

Accelerate Your Data Science Workflows End-to-End Self-Paced Course

Discover how to create, accelerate, and deploy data pipelines with RAPIDS for GPU-accelerated data science workflows. Take this course for free when you join the NVIDIA Developer Program →
NVIDIA sponsor

How to Build a Query Language in Python

This guide walks you through how to build a custom query language in Python. The example given is a language to search through song lyrics.
JAMES G

Quiz: Python’s Built-in Exceptions

REAL PYTHON

Quiz: Getting Started With Testing in Python

REAL PYTHON

Articles & Tutorials Build a Blog Using Django, GraphQL, and Vue

In this step-by-step project, you’ll build a blog from the ground up. You’ll turn your Django blog data models into a GraphQL API and consume it in a Vue application for users to read. You’ll end up with an admin site and a user-facing site you can continue to refine for your own use.
REAL PYTHON

Exploring Robotics and Python Through Electronic Projects

Are you interested in learning robotics with Python? Can physical electronics-based projects grow a child’s interest in coding? This week on the show, we speak with author Marwan Alsabbagh about his book “Build Your Own Robot - Using Python, CRICKIT, and Raspberry Pi.”
REAL PYTHON podcast

How Pre-Commit Works

As a user of pre-commit hooks, do you know what happens when you run pre-commit install or why you have to run it in the first place? How does pre-commit actually work with Git? In this article, Stefanie takes you behind the scenes of how your pre-commit setup works.
STEFANIEMOLIN.COM • Shared by Stefanie Molin

Python range(): Represent Numerical Ranges

Master the Python range() function and learn how it works under the hood. You most commonly use ranges in loops. In this tutorial, you’ll learn how to iterate over ranges but also identify when there are better alternatives.
REAL PYTHON

Practices of Reliable Software Design

The author was asked what he would considered if he wrote a in-memory cache. This article talks about the eight principles he would use, many of which he wouldn’t have considered when a younger developer.
TWO WRONGS

Python’s Preprocessor

Every now and then you hear outrageous claims such as “Python has no preprocessor”, well it is there if you’re willing to dig deep enough. Learn how to hack Python’s compile step.
PYDONG

500 Python Interpreters

Work continues on removing and/or optimizing the GIL in Python. This article gives a little history so you can better understand why the GIL is there and what changes are coming.
IZZY MUERTE

Python Top Language of 2024

“Python continues to cement its overall dominance, buoyed by things like popular libraries for hot fields such as A.I.” Read the article to see where other languages have placed.
IEEE SPECTRUM

10 Python Programming Optimization Techniques

Optimization should be your last step, but once you’re there, just what can you do? This article covers ten different techniques that address memory size and code performance.
JAMES ONONIWU

uv: Unified Python Packaging

A new release of of uv is out and it has added a lot of features. This post talks about what is new and how it can simplify your packaging process.
SIMON WILLISON

PyPI Slashes Malware Response Time

“PyPI has drastically improved its malware response times, resolving 90% of issues in under 24 hours and removing 900 projects since March 2024.”
SARAH GOODING

Projects & Code Gists-Gone, a CLI for Bulk Deleting Your Github Gists

GITHUB.COM/BEN-N93 • Shared by Ben Nour

deltadb: A Lightweight Database Built on Polars and Deltalake

GITHUB.COM/UNAME-N

django-public-admin: A Public and Read-Only Django Admin

GITHUB.COM/CUDUCOS

sqlfluff: A Modular SQL Linter and Auto-Formatter

GITHUB.COM/SQLFLUFF

authentik: The Authentication Glue You Need

GITHUB.COM/GOAUTHENTIK

Events Weekly Real Python Office Hours Q&A (Virtual)

August 28, 2024
REALPYTHON.COM

PyCon PL 2024

August 29 to September 1, 2024
PYCON.ORG

SPb Python Drinkup

August 29, 2024
MEETUP.COM

PythOnRio Meetup

August 31, 2024
PYTHON.ORG.BR

Melbourne Python Users Group, Australia

September 2, 2024
J.MP

EARL 2024

September 4 to September 6, 2024
DATACOVE.CO.UK

PyCon Estonia 2024

September 5 to September 7, 2024
PYCON.EE

Happy Pythoning!
This was PyCoder’s Weekly Issue #644.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

KubeCon China 2024 was a whirlwind of innovation, community and technical deep dives. As it often happens at these community events, I was blown away by the energy, enthusiasm and sheer amount of knowledge being shared. Here are three key takeaways that stood out to me:

1. The focus on AI and machine learning

AI and machine learning are increasingly integrated into cloud-native applications. At KubeCon China, I saw numerous demonstrations of how these technologies are being used to automate tasks, optimize resource utilization and improve application performance. From AI-powered observability tools to machine learning-driven anomaly detection, the potential for AI and ML in the cloud-native space is astounding.

Mer Joyce and Anni Lai introduced the new draft of the Open Source AI Definition (v.0.0.9) and the Model Openness Framework. 

We also saw a robot on stage demonstrating that teaching a robotic arm to use a spoon to help disabled people is not a programming issue but a data issue. This was probably my biggest learning moment: A robot can be “taught” to execute tasks by imitating humans. Follow Xavier Tao and the dora-rs project.

2. The growing maturity of cloud-native technologies

It’s clear that cloud-native technologies have come of age. From Kubernetes adoption to the rise of serverless platforms and edge computing, the ecosystem is thriving. In his keynote, Chris Aniszczyk announced over 200 projects are hosted by the Cloud Native Computing Foundation and half of the contributors are not in the US. The conference showcased a wide range of tools, frameworks and use cases that demonstrate the versatility and scalability of cloud-native architectures.

The presentation by Kevin Wang (Huawei) and Saint Jiang (NIO) showed how Containerd, Kubernetes and KubeEdge power the transition to electric vehicles. Modern cars are computers… no, cars are full datacenters on wheels, a collection of sensors feeding distributed applications to optimize battery usage, feeding into centralized programs to constantly improve the whole mobility system.

3. AI technology is removing the language barrier

I was absolutely amazed by being able to follow the keynote sessions delivered in Chinese. I don’t speak Chinese but I could read the automatic translation in real time superimposed on the slides behind the speakers. This technology is absolutely jaw-droppingly amazing! Within a few years, there won’t be a career for simultaneous translators or for live transcribers.

Final thoughts

KubeCon + AI_Dev China was a testament to the power of Open Source collaboration hosted in one of the most amazing regions of the world. The conference brought together developers, operators and end-users from around the world to share their experiences, best practices and contributions to Open Source projects. This collaborative spirit is essential for driving innovation and ensuring the long-term success of cloud-native technologies.

BOSTON (August 27, 2024) -- Free Software Foundation (FSF) Board Member Odile Bénassy has stepped down from the Board after four years of service.

There’s a concept that I’ve heard called by a lot of different names, but my favorite name for it is “the Highlander problem”, which refers to the catchphrase of the campy-yet-still-quite-fun Highlander movie/TV franchise. In Highlander, immortal beings secretly live amongst us and sword-fight each other in hopes of being the last one standing, who will then get to rule the world forever. And when one of them is about to eliminate another, …

Read full entry

At Drupal GovCon 2024, an analysis of Federal government websites revealed Drupal as the leading CMS, powering 55% of identifiable sites. The presentation highlighted Drupal’s superiority in accessibility, with 9 of the top 10 most accessible government sites using the platform. Emphasizing the importance of Web Vitals, the discussion urged agencies to prioritize customer experience through performance improvements.

Here at KDE neon tower we have been busy rebasing our KDE software builds from Ubuntu 22.04 (jammy) to Ubuntu 24.04 (noble). This always takes longer than you’d think, mostly because it’s a moving target so we also have to keep updating the incoming releases from Plasma, Frameworks, Gear and even Calligra. We had a couple of delays when Jonathan caught Covid (4 years after it was worth sympathy to do so) and then the build server had issues and needed itself rebuilding. But the package archives are there and the Docker images are there and today the first ISO got built which boots successfully. Next steps are making sure all the software is up to date and getting the upgrade solid. Be with you soon!

First KDE neon ISO based on Noble

The Breeze icons used in message boxes always felt a little odd with a status icon placed inside some kind of speech bubble, effectively an icon within an icon. Three months ago they got replaced by more simplistic ones that I felt didn’t fit very well either. Therefore I put my Inkscape skills to the test and created a new set of Breeze-style dialog icons.

As you may know I have a secret passion for vector graphics. I am not very good at it but I love that you have proper shapes and objects to work with as opposed to a mush of pixels. Sure, more sophisticated graphics programs have layers and masks and what not but my proficiency there doesn’t go much beyond Kolourpaint. More importantly, though, I can easily copy paste together various bits and pieces of other icons to implement the icon I had in mind.

A few of the file icons I have worked on but didn’t finish (feel free to take as an inspiration)

Over the years I have created a couple of Breeze icons, mostly for (exotic) file types since I am on a quest to have a thumbnailer or at least a proper icon for every file type imaginable. Take for example the Apple Wallet icon we use for KDE Itinerary, our fantastic travel companion app: it’s just a ZIP file with a JSON description and graphical assets for a boarding pass or event ticket. I used a generic blue file icon combined with the wallet-open icon. My original idea was actually a plane alongside a QR code but I couldn’t help but view it as a plane that crashed into a building.

The other day I received concert tickets in the form of an Apple Wallet bundle which contains multiple passes, something I didn’t know existed and probably didn’t back when I created the original icon. For this purpose I used the generic package icon in the same light blue, zipper shifted to the right to match the Android APK icon, and wallet emblem added.

How the Apple Wallet passes icon came to be

Friday night I sat down to finally create some new dialog icons for error, warning, question, and information. Breeze come with has shield-shaped security icons similar to the message box icons we used to have. I just wanted a triangle with a Breeze-style gradient and angled drop shadow, how hard can it be? I ended up twisting the security-medium icon (an orange shield with exclamation mark), removing nodes, stretching objects, and straightening Bezier curves until I got what I wanted.

For the error icon I initially wanted a simple circle with an X. This time security-low (a red shield with a cross) served as a base. I drew a simple circle and used the shield’s gradient as fill. I couldn’t figure out how to bend the shield-like outline at the bottom to match the circle so I went looking for a similar circular icon I could steal it from. Luckily, there was exactly one circular 64 px icon I could copy. For question and information I also used purple and blue circles, respectively.

First iteration of the new dialog-error icon

Afterwards I figured I probably want a different shape for the error icon so it’s not all circles and to keep it from looking like a close button. That’s actually the main reason our message widget lost its icon, people confused the error symbol on the left with a close button to dismiss the message. I checked what other icon themes did and found squares were quite popular (also used by the KDE 4 Oxygen theme) as well as octagons. I tried a square by modifying the utilities-terminal (Konsole) icon but that just looked way too massive. After consulting the KDE Visual Design Group we settled on an octagon like a stop sign (reminds me of the glorious Windows 3.1 days which used an actual stop sign). Finally, Janet Blackquill and Andy Betts gave the icons some finishing touches and needed polish, taking into account our Colorful Icon HIG.

This post illustrates (in more than one sense of the word) nicely that contributing to KDE is more than writing actual source code. Creating artwork, writing translations, doing promo work, managing IT infrastructure, and of course hosting community events like Akademy is just as important for a community to thrive!

This course covers two problems from introductory astronomy to help you play with some Python libraries. You’ll use Astropy, NumPy, Matplotlib, and pandas to find planet conjunctions, and graph the best viewing times for a star.

In this course you’ll learn about:

• Astronomy concepts of conjunction and optimal viewing
• The Python package Astropy
• Using pandas to process data
• Building graphs with Matplotlib
• Python’s warning module

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]