Feeds

YAML is a portable and widely used data serialization format. Unlike the more compact JSON or verbose XML formats, YAML emphasizes human readability with block indentation, which should be familiar to most Python programmers. While Python comes with batteries included, it lacks built-in support for YAML. Still, you can read and write YAML documents in Python by installing a third-party library, such as PyYAML.

PyYAML allows you to serialize and deserialize Python data types, as well as custom objects, to and from YAML. It also provides ways to safely handle YAML data from untrusted sources. PyYAML is a popular choice for relatively simple use cases. However, you might want to explore alternatives like ruamel.yaml for YAML 1.2 compliance or StrictYAML for schema validation.

By the end of this tutorial, you’ll understand that:

• YAML stands for YAML Ain’t Markup Language, emphasizing its focus on data representation rather than document markup.
• YAML is often used for configuration and serialization due to its human-readable syntax.
• Python doesn’t support YAML natively, unlike the JSON and XML formats. You need to install a third-party library to work with YAML in Python programs.
• PyYAML is arguably the most popular YAML library for Python due to its simplicity.
• Alternatives to PyYAML include ruamel.yaml and StrictYAML, which offer more features.

To get the most out of this tutorial, you should be familiar with object-oriented programming in Python and know how to create a class. If you’re ready to jump in, then you can follow the link below to get the source code for the examples that you’ll code in this tutorial:

Get Your Code: Click here to download the free sample code you’ll use to work with YAML in Python.

Taking a Crash Course in YAML

In this section, you’re going to learn the basic facts about YAML, including its uses, syntax, and some of its unique and powerful features. If you’ve worked with YAML before, then you can skip ahead and continue reading from the next section, which covers using YAML in Python.

Historical Context

YAML, which rhymes with camel, is a recursive acronym that stands for YAML Ain’t Markup Language because it’s not a markup language! Interestingly enough, the original draft for the YAML specification defined the language as Yet Another Markup Language, but later the current backronym was adopted to more accurately describe the language’s purpose.

An actual markup language, such as Markdown or HTML, lets you annotate text with formatting or processing instructions intermixed with your content. Markup languages are, therefore, primarily concerned with text documents, whereas YAML is a data serialization format that integrates well with common data types native to many programming languages. There’s no inherent text in YAML, only data to represent.

YAML was originally meant to simplify Extensible Markup Language (XML), but in reality, it has a lot more in common with JavaScript Object Notation (JSON). In fact, it’s a superset of JSON.

Even though XML was initially designed to be a metalanguage for creating markup languages for documents, people quickly adopted it as the standard data serialization format. The HTML-like syntax of angle brackets made XML look familiar. Suddenly, everyone wanted to use XML as their configuration, persistence, or messaging format.

As the first kid on the block, XML dominated the scene for many years. It became a mature and trusted data interchange format and helped shape new concepts like building interactive web applications. After all, the letter X in AJAX, a technique for getting data from the server without reloading the page, stands for none other than XML.

Ironically, it was AJAX that ultimately led to XML’s decline in popularity. The verbose, complex, and redundant XML syntax wasted a lot of bandwidth when data was sent over the network. Parsing XML documents in JavaScript was slow and tedious because of XML’s fixed document object model (DOM), which wouldn’t match the application’s data model. The community finally acknowledged that they’d been using the wrong tool for the job.

That’s when JSON entered the picture. It was built from the ground up with data serialization in mind. Web browsers could parse it effortlessly because JSON is a subset of JavaScript, which they already supported. Not only was JSON’s minimalistic syntax appealing to developers, but it also made porting to other platforms easier than XML did. To this day, JSON remains the slimmest, fastest, and most versatile textual data interchange format on the Internet.

YAML came into existence the same year as JSON, and by pure coincidence, it was almost a complete superset of JSON on the syntactical and semantic levels. Starting from YAML 1.2, the format officially became a strict superset of JSON, meaning that every valid JSON document also happens to be a YAML document.

In practice, however, the two formats look different, as the YAML specification puts more emphasis on human readability by adding a lot more syntactic sugar and features on top of JSON. As a result, YAML is more applicable to configuration files edited by hand rather than as a transport layer.

Comparison With XML and JSON

If you’re familiar with XML or JSON, then you might be wondering what YAML brings to the table. All three are major data interchange formats, which share some overlapping features. For example, they’re all text based and more or less human readable. At the same time, they differ in many respects, which you’ll find out next.

Note: There are other, popular textual data formats like TOML, which the new build system in Python is based on. Currently, only external packaging and dependency management tools like Poetry can read TOML, but since Python 3.11 there’s a TOML parser in the standard library.

Common binary data serialization formats you’d find in the wild include Google’s Protocol Buffers and Apache’s Avro.

Now have a look at a sample document expressed in all three data formats but representing the same person. You can click to expand the collapsible sections and reveal data serialized in those formats:

XMLShow/Hide

XML <?xml version="1.0" encoding="UTF-8" ?> <person firstName="John" lastName="Doe"> <dateOfBirth>1969-12-31</dateOfBirth> <married>true</married> <spouse> <person firstName="Jane" lastName="Doe"> <dateOfBirth/> <!- This is a comment --> </person> </spouse> </person> Copied!

JSONShow/Hide

JSON { "person": { "dateOfBirth": "1969-12-31", "firstName": "John", "lastName": "Doe", "married": true, "spouse": { "dateOfBirth": null, "firstName": "Jane", "lastName": "Doe" } } } Copied!

YAMLShow/Hide

YAML %YAML 1.2 --- person: dateOfBirth: 1969-12-31 firstName: John lastName: Doe married: true spouse: dateOfBirth: null # This is a comment firstName: Jane lastName: Doe Copied! Read the full article at https://realpython.com/python-yaml/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python decorators allow you to modify or extend the behavior of functions and methods without changing their actual code. When you use a Python decorator, you wrap a function with another function, which takes the original function as an argument and returns its modified version. This technique provides a simple way to implement higher-order functions in Python, enhancing code reusability and readability.

You can use decorators in various practical scenarios, such as logging, enforcing access control, caching results, or measuring execution time. To create a custom decorator, define a function that takes another function as an argument, creates a nested wrapper function, and returns the wrapper. When applying a decorator, you place @decorator on the line before the function definition.

By the end of this tutorial, you’ll understand that:

• Python decorators allow you to wrap a function with another function to extend or modify its behavior without altering the original function’s code.
• Practical use cases for decorators include logging, enforcing access control, caching results, and measuring execution time.
• Custom decorators are written by defining a function that takes another function as an argument, defines a nested wrapper function, and returns the wrapper.
• Multiple decorators can be applied to a single function by stacking them, one per line, before the function definition.
• The order of decorators impacts the final output, as each decorator wraps the next, influencing the behavior of the decorated function.

You can find all the examples from this tutorial by downloading the accompanying materials below:

Get Your Code: Click here to download the free sample code that shows you how to create and use Python decorators.

Free Bonus: Click here to get access to a free "The Power of Python Decorators" guide that shows you three advanced decorator patterns and techniques you can use to write cleaner and more Pythonic programs.

Decorators Cheat Sheet: Click here to get access to a free three-page Python decorators cheat sheet that summarizes the techniques explained in this tutorial.

Decorators Q&A Transcript: Click here to get access to a 25-page chat log from our Python decorators Q&A session in the Real Python Community Slack where we discussed common decorator questions.

Take the Quiz: Test your knowledge with our interactive “Decorators” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Decorators

In this quiz, you'll revisit the foundational concepts of what Python decorators are and how to create and use them.

Python Functions

In order to understand decorators, you must first understand some finer points of how functions work. There are many aspects to functions, but in the context of decorators, a function returns a value based on the given arguments. Here’s a basic example:

Python >>> def add_one(number): ... return number + 1 ... >>> add_one(2) 3 Copied!

In general, functions in Python may also have side effects rather than just turning an input into an output. The print() function is an example of this: it returns None while having the side effect of outputting something to the console. However, to understand decorators, it’s enough to think about functions as tools that turn given arguments into values.

First-Class Objects

In functional programming, you work almost entirely with pure functions that don’t have side effects. While not a purely functional language, Python supports many functional programming concepts, including treating functions as first-class objects.

This means that functions can be passed around and used as arguments, just like any other object like str, int, float, list, and so on. Consider the following three functions:

Python greeters.py def say_hello(name): return f"Hello {name}" def be_awesome(name): return f"Yo {name}, together we're the awesomest!" def greet_bob(greeter_func): return greeter_func("Bob") Copied!

Here, say_hello() and be_awesome() are regular functions that expect a name given as a string. The greet_bob() function, however, expects a function as its argument. You can, for example, pass it the say_hello() or the be_awesome() function.

To test your functions, you can run your code in interactive mode. You do this with the -i flag. For example, if your code is in a file named greeters.py, then you run python -i greeters.py:

Python >>> greet_bob(say_hello) 'Hello Bob' >>> greet_bob(be_awesome) 'Yo Bob, together we're the awesomest!' Copied!

Note that greet_bob(say_hello) refers to two functions, greet_bob() and say_hello, but in different ways. The say_hello function is named without parentheses. This means that only a reference to the function is passed. The function isn’t executed. The greet_bob() function, on the other hand, is written with parentheses, so it will be called as usual.

This is an important distinction that’s crucial for how functions work as first-class objects. A function name without parentheses is a reference to a function, while a function name with trailing parentheses calls the function and refers to its return value.

Inner Functions

It’s possible to define functions inside other functions. Such functions are called inner functions. Here’s an example of a function with two inner functions:

Read the full article at https://realpython.com/primer-on-python-decorators/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Sorting a Python dictionary involves organizing its key-value pairs in a specific order. To sort a Python dictionary by its keys, use the sorted() function combined with .items(). This approach returns a list of tuples sorted by keys, which you can convert back to a dictionary using the dict() constructor. Sorting by values requires specifying a sort key using a lambda function or itemgetter().

You can sort a Python dictionary in descending order by setting the reverse argument of the sorted() function to True. Sorting dictionaries with non-comparable keys or values can be challenging, but you can manage it by providing default values or using a custom sort key.

By the end of this tutorial, you’ll understand that:

• You can sort a dictionary by its keys using sorted() with .items() and dict().
• To sort by values, you use sorted() with a key function like lambda or itemgetter().
• Sorting in descending order is possible by setting reverse=True in sorted().
• For non-comparable keys or values, you use default values or custom sort keys.
• Python dictionaries can’t be sorted in-place, so you need to create a new sorted dictionary.

Read on to learn how to effectively sort dictionaries using these techniques and the strategic implications of choosing the right data structure for your key-value data. But first, you’ll learn some foundational knowledge that will help you understand how to sort a dictionary in Python.

Free Download: Click here to download the code that you’ll use to sort key-value pairs in this tutorial.

Rediscovering Dictionary Order in Python

Before Python 3.6, dictionaries were inherently unordered. A Python dictionary is an implementation of the hash table, which is traditionally an unordered data structure.

As a side effect of the compact dictionary implementation in Python 3.6, dictionaries started to conserve insertion order. From 3.7, that insertion order has been guaranteed.

If you wanted to keep an ordered dictionary as a data structure before compact dictionaries, then you could use OrderedDict from the collections module. Similar to the modern compact dictionary, it also keeps insertion order, but neither type of dictionary sorts itself.

Another alternative for storing an ordered key-value pair data is to store the pairs as a list of tuples. As you’ll see later in the tutorial, using a list of tuples could be the best choice for your data.

An essential point to understand when sorting dictionaries is that even though they conserve insertion order, they’re not considered a sequence. A dictionary is like a set of key-value pairs, and sets are unordered.

Dictionaries also don’t have much reordering functionality. They’re not like lists, where you can insert elements at any position. In the next section, you’ll explore the consequences of this limitation further.

Understanding What Sorting a Dictionary Really Means

Because dictionaries don’t have much reordering functionality, when sorting a dictionary, it’s rarely done in-place. In fact, there are no methods for explicitly moving items in a dictionary.

If you wanted to sort a dictionary in-place, then you’d have to use the del keyword to delete an item from the dictionary and then add it again. Deleting and then adding again effectively moves the key-value pair to the end.

The OrderedDict class has a specific method to move an item to the end or the start, which may make OrderedDict preferable for keeping a sorted dictionary. However, it’s still not very common and isn’t very performant, to say the least.

The typical method for sorting dictionaries is to get a dictionary view, sort it, and then cast the resulting list back into a dictionary. So you effectively go from a dictionary to a list and back into a dictionary. Depending on your use case, you may not need to convert the list back into a dictionary.

Note: Sorted dictionaries aren’t a very common pattern. You’ll explore more about that topic later in the tutorial.

With those preliminaries out of the way, you’ll get to sorting dictionaries in the next section.

Sorting Dictionaries in Python

In this section, you’ll be putting together the components of sorting a dictionary so that, in the end, you can master the most common way of sorting a dictionary:

Python >>> people = {3: "Jim", 2: "Jack", 4: "Jane", 1: "Jill"} >>> # Sort by key >>> dict(sorted(people.items())) {1: 'Jill', 2: 'Jack', 3: 'Jim', 4: 'Jane'} >>> # Sort by value >>> dict(sorted(people.items(), key=lambda item: item[1])) {2: 'Jack', 4: 'Jane', 1: 'Jill', 3: 'Jim'} Copied!

Don’t worry if you don’t understand the snippets above—you’ll review it all step-by-step in the following sections. Along the way, you’ll learn how to use the sorted() function with sort keys, lambda functions, and dictionary constructors.

Using the sorted() Function Read the full article at https://realpython.com/sort-python-dictionary/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Python’s walrus operator (:=) allows you to assign values to variables as part of an expression. It can simplify your code by combining assignment and evaluation in a single statement. You use it to streamline constructs like list comprehensions, loops, and conditionals, making your code more concise and readable. The walrus operator is particularly useful when you want to avoid repetitive function calls or calculations.

The walrus operator, also known as an assignment expression, differs from regular assignments by returning the assigned value, enabling inline assignments within expressions. Regular assignment (=) doesn’t return a value, which helps prevent common programming errors.

By the end of this tutorial, you’ll understand that:

• The walrus operator (:=) is a syntax for assigning variables within expressions in Python.
• Assignment expressions use the walrus operator in Python.
• You can use the walrus operator to streamline code, such as in loops or conditionals.
• Assignment expressions return the assigned value, unlike regular assignments.
• Regular assignments don’t return values to prevent unintended behavior and bugs.
• Best practices for the walrus operator include using it for clarity and efficiency without overcomplicating code.

Understanding the walrus operator will enhance your ability to write efficient and concise Python code, especially when dealing with complex conditions and loops.

Get Your Code: Click here to download the free sample code that shows you how to use Python’s walrus operator.

Take the Quiz: Test your knowledge with our interactive “The Walrus Operator: Python's Assignment Expressions” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

The Walrus Operator: Python's Assignment Expressions

In this quiz, you'll test your understanding of Python's walrus operator. This operator was introduced in Python 3.8, and understanding it can help you write more concise and efficient code.

Walrus Operator Fundamentals

First, look at some different terms that programmers use to refer to this new syntax. You’ve already seen a few in this tutorial.

The := operator is officially known as the assignment expression operator. During early discussions, it was dubbed the walrus operator because the := syntax resembles the eyes and tusks of a walrus lying on its side. You may also see the := operator referred to as the colon equals operator. Yet another term used for assignment expressions is named expressions.

Hello, Walrus!

To get a first impression of what assignment expressions are all about, start your REPL and play around with the following code:

Python 1>>> walrus = False 2>>> walrus 3False 4 5>>> (walrus := True) 6True 7>>> walrus 8True Copied!

Line 1 shows a traditional assignment statement where the value False is assigned to walrus. Next, on line 5, you use an assignment expression to assign the value True to walrus. After both lines 1 and 5, you can refer to the assigned values by using the variable name walrus.

You might be wondering why you’re using parentheses on line 5, and you’ll learn why the parentheses are needed later on in this tutorial.

Note: A statement in Python is a unit of code. An expression is a special statement that can be evaluated to some value.

For example, 1 + 2 is an expression that evaluates to the value 3, while number = 1 + 2 is an assignment statement that doesn’t evaluate to a value. Although running the statement number = 1 + 2 doesn’t evaluate to 3, it does assign the value 3 to number.

In Python, you often see simple statements like return statements and import statements, as well as compound statements like if statements and function definitions. These are all statements, not expressions.

There’s a subtle—but important—difference between the two types of assignments with the walrus variable. An assignment expression returns the value, while a traditional assignment doesn’t. You can see this in action when the REPL doesn’t print any value after walrus = False on line 1 but prints out True after the assignment expression on line 5.

You can see another important aspect about walrus operators in this example. Though it might look new, the := operator does not do anything that isn’t possible without it. It only makes certain constructs more convenient and can sometimes communicate the intent of your code more clearly.

Now you have a basic idea of what the := operator is and what it can do. It’s an operator used in assignment expressions, which can return the value being assigned, unlike traditional assignment statements. To get deeper and really learn about the walrus operator, continue reading to see where you should and shouldn’t use it.

Implementation

Like most new features in Python, assignment expressions were introduced through a Python Enhancement Proposal (PEP). PEP 572 describes the motivation for introducing the walrus operator, the details of the syntax, and examples where the := operator can be used to improve your code.

This PEP was originally written by Chris Angelico in February 2018. Following some heated discussion, PEP 572 was accepted by Guido van Rossum in July 2018.

Since then, Guido announced that he was stepping down from his role as benevolent dictator for life (BDFL). Since early 2019, the Python language has been governed by an elected steering council instead.

The walrus operator was implemented by Emily Morehouse, and made available in the first alpha release of Python 3.8.

Motivation Read the full article at https://realpython.com/python-walrus-operator/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Last Tuesday, during lunch hours I had a talk at KTH computer science students' organization. The topic was Open Source and career. My main goal was tell the attendees that contribution size does not matter, but continuing contributing to various projects can change someone's life and career in a positive way. I talked about the history of the Free Software movement and Open Source. I also talked a bit about Aaron Swartz and asked the participants to watch the documentary The Internet's Own Boy. Some were surprised to hear about Sunet's Open Source work.

There were around 70 people and few people later message how they think about contribution after my talk. The best part was one student who messaged next day and said that he contributed one small patch to a project.

I also told them about PyLadies Stockholm and other local efforts from various communities. There was also a surprising visit of the #curl channel on IRC, thanks to bagder and icing :)

Drupal Advent Calendar day 14 - Multilingual and Concurrent Editing lostcarpark_admin Sat, 12/14/2024 - 09:00

Welcome to another door of the Starshot focused Drupal Advent Calendar. This might be that day when you open the door and there’s an odd-shaped piece of chocolate, and you’re not quite sure what it’s supposed to represent. Starshot, like any large project, has some tracks that have advanced more than others, and it’s only natural that some tracks are still at an early stage. Today we are taking a brief look at a couple of those tracks, along with some tasty bonus information.

Multilingual Track

Drupal has excellent support for translation and multilingual websites in core. However, it can be…

Tags

• multilingual
• translation
• concurrent editing
• collaboration

This week's headliner change is something that I think will make a lot of people happy: better fractional scaling! Vlad and Xaver have been hard at work to snap everything to the screen's pixel grid, with the effect that using a fractional scale factor now results in a lot less blurriness as well as no more gaps between windows and their shadows. You'll see it in the screenshot below (which was taken at 175% scale) but the effects are subtly better everywhere. Really great stuff!

And lots more too, of course:

Notable New Features

At very high zoom levels, KWin's Zoom effect switches to a sharp pixel-perfect representation and overlays a grid on top of the screen. This makes it easy to see how individual pixels look relative to other ones, which can be useful for artists and designers. (Vlad Zahorodnii, 6.3.0. Link)

KWin now offers you the option to prefer screen color accuracy at the expense of some system performance, should that be your preference (e.g. if you're a digital artist and not a gamer). (Xaver Hugl, 6.3.0. Link)

If the feature to be able to maximize a window horizontally or vertically by double-clicking on one of its edges doesn't agree with you, you can now disable it. (Vlad Zahorodnii, 6.3.0. Link)

Notable UI Improvements

Landed a huge overhaul of how fractional scale factors are handled in KWin. Now it makes an effort to always snap things to the screen's pixel grid, greatly reducing blurriness and visual gaps everywhere. I've been using these patches with a 175% scale factor for a week, and everything looks just fantastic! (Vlad Zahorodnii and Xaver Hugl, 6.3.0. Link)

On login, Plasma panels now appear on screen only after their contents have been fully loaded. (Niccolò Venerandi, 6.3.0. Link)

Notable Bug Fixes

Fixed a nasty bug affecting people using the X11 session that could sometimes cause the lock screen to be all black. (Philip Müller, 6.2.5. Link)

Fixed a specific instance where you could end up with a black screen when wiggling the pointer while the screen is about to lock. (David Redondo, 6.2.5 Link)

Fixed a visual bug in Discover that caused UI elements to overlap on expanded list items on the Updates page. (Aleix Pol Gonzalez, 6.2.5. Link)

Fixed the application menu appearing in a wrong position when opened via the window titlebar with Qt 6.8. (David Redondo, 6.2.5. Link)

Fixed a bug that could cause windows on a screen that gets disconnected to become lost and stuck in an off-screen position in the new screen arrangement. (Vlad Zahorodnii and Xaver Hugl, 6.3.0. Link)

You can no longer slightly break the Overview effect's Desktop Grid view by dragging windows outside of the screen area. (Niccolò Venerandi, 6.3.0. Link)

Dragging an image from the clipboard to the desktop now shows the normal drop menu, rather than creating an empty Media Frame widget. (Fushan Wen, 6.3.0. Link)

Non-rectangular-region screenshots taken in Spectacle and copied to the clipboard can now be pasted into Dolphin as expected. (Fushan Wen, 6.3.0. Link)

Standalone (not in System Tray) "Power and Battery" and "Brightness and Color" widgets once again work properly, as expected. (Jakob Petsovits, 6.3.0. Link)

Fixed a bug in the Breeze Dark icon theme that caused places/folder icons to remain colorful at small sizes where symbolic icons are normally expected. (David Redondo, Frameworks 6.9. Link)

Plasma and lots of apps no longer crash when your /etc/fstab file contains any loop mounts in it. (Nicolas Fella, Frameworks 6.10. Link)

Other bug information of note:

• 2 Very high priority Plasma bugs (down from 3 last week). Current list of bugs
• 32 15-minute Plasma bugs (down from 35 last week). Current list of bugs
• 104 KDE bugs of all kinds fixed over the last week. Full list of bugs

Notable in Performance & Technical

Ported the clipboard to use a standard SQLite database, rather than its own internal custom format. This improves reliability, support for saving many data types, and memory efficiency especially with images. (Fushan Wen, 6.3.0. Link)

How You Can Help

KDE has become important in the world, and your time and contributions have helped us get there. As we grow, we need your support to keep KDE sustainable.

Thankfully, thousands of you have stepped up in the past week to do just that financially, donating a record-breaking amount of money to KDE e.V., which is just incredible, awe-inspiring even.

So that's a great way to help out. But if you've got more time than money or want to make a difference more directly, then you can help KDE by becoming an active community member and getting involved somehow. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine!

You don’t have to be a programmer, either. Many other opportunities exist:

• Triage and confirm bug reports, maybe even identify their root cause
• Contribute designs for wallpapers, icons, and app interfaces
• Design and maintain websites
• Translate user interface text items into your own language
• Promote KDE in your local community
• …And a ton more things!

To get a new Plasma feature or a bugfix mentioned here, feel free to push a commit to the relevant merge request on invent.kde.org.

This release fixes some bugs. Have a look at the changelog for more details.

Changelog

Bugfixes:

• Fix displaying files of each message in appropriate message bubble (melvo)
• Fix sending fallback messages for clients not supporting XEP-0447: Stateless file sharing (melvo)
• Fix margins within message bubbles (melvo)
• Fix hiding hidden message part (melvo)
• Fix displaying marker for new messages (melvo)

Download

• Source code (.tar.xz) (sig signed with 04EFAD0F7A4D9724)
• Linux (Flatpak on Flathub)

Or install Kaidan for your distribution:

Check out the important work our volunteers accomplished at today's Free Software Directory (FSD) IRC meeting.

Check out the important work our volunteers accomplished at today's Free Software Directory (FSD) IRC meeting.

Join the FSF and friends on Friday, December 13 from 12:00 to 15:00 EST (17:00 to 20:00 UTC) to help improve the Free Software Directory.

This is the story of an investigation conducted by Jochen Sprickerhof, Helmut Grohne, and myself. It was true teamwork, and we would have not reached the bottom of the issue working individually. We think you will find it as interesting and fun as we did, so here is a brief writeup. A few of the steps mentioned here took several days, others just a few minutes. What is described as a natural progression of events did not always look very obvious at the moment at all.

Let us go through the Six Stages of Debugging together.

Stage 1: That cannot happen

Official Debian GCC builds start failing on multiple architectures in late November.

The build error happens on the build servers when running the testuite, but we know this cannot happen. GCC builds are not meant to fail in case of testsuite failures! Return codes are not making the build fail, make is begin called with -k, it just cannot happen.

A lot of the GCC tests are always failing in fact, and an extensive log of the results is posted to the debian-gcc mailing list, but the packages always build fine regardless.

On the build daemons, build failures take several hours.

Stage 2: That does not happen on my machine

Building on my machine running Bookworm is just fine. The Build Daemons run Bookworm and use a Sid chroot for the build environment, just like I am. Same kernel.

Debian packages are built by a network of autobuilding machines using a program called sbuild. In my last blog post I mentioned the transition from the schroot backend to a new one based on unshare.

The only obvious difference between my setup and the Debian buildds is that I am using sbuild 0.85.0 from bookworm, and the buildds have 0.86.3~bpo12+1 from bookworm-backports. Trying again with 0.86.3~bpo12+1, the build fails on my system too. The build daemons were updated to the bookworm-backports version of sbuild at some point in late November. Ha.

Stage 3: That should not happen

There are quite a few sbuild versions in between 0.85.0 and 0.86.3~bpo12+1, but looking at recent sbuild bugs shows that sbuild 0.86.0 was breaking "quite a number of packages". Indeed, with 0.86.0 the build still fails. Trying the version immediately before, 0.85.11, the build finishes correctly. This took more time than it sounds, one run including the tests takes several hours. We need a way to shorten this somehow.

The Debian packaging of GCC allows to specify which languages you may want to skip, and by default it builds Ada, Go, C, C++, D, Fortran, Objective C, Objective C++, M2, and Rust. When running the tests sequentially, the build logs stop roughly around the tests of a runtime library for D, libphobos. So can we still reproduce the failure by skipping everything except for D? With DEB_BUILD_OPTIONS=nolang=ada,go,c,c++,fortran,objc,obj-c++,m2,rust the build still fails, and it fails faster than before. Several minutes, not hours. This is progress, and time to file a bug. The report contains massive spoilers, so no link. :-)

Stage 4: Why does that happen?

Something is causing the build to end prematurely. It’s not the OOM killer, and the kernel does not have anything useful to say in the logs. Can it be that the D language tests are sending signals to some process, and that is what’s killing make ? We start tracing signals sent with bpftrace by writing the following script, signals.bt:

tracepoint:signal:signal_generate { printf("%s PID %d (%s) sent signal %d to PID %d\n", comm, pid, args->sig, args->pid); }

And executing it with sudo bpftrace signals.bt.

The build takes its sweet time, and it fails. Looking at the trace output there’s a suspicious process.exe terminating stuff.

process.exe (PID: 2868133) sent signal 15 to PID 711826

That looks interesting, but we have no clue what PID 711826 may be. Let’s change the script a bit, and trace signals received as well.

tracepoint:signal:signal_generate { printf("PID %d (%s) sent signal %d to %d\n", pid, comm, args->sig, args->pid); } tracepoint:signal:signal_deliver { printf("PID %d (%s) received signal %d\n", pid, comm, args->sig); }

The working version of sbuild was using dumb-init, whereas the new one features a little init in perl. We patch the current version of sbuild by making it use dumb-init instead, and trace two builds: one with the perl init, one with dumb-init.

Here are the signals observed when building with dumb-init.

PID 3590011 (process.exe) sent signal 2 to 3590014 PID 3590014 (sleep) received signal 9 PID 3590011 (process.exe) sent signal 15 to 3590063 PID 3590063 (std.process tem) received signal 9 PID 3590011 (process.exe) sent signal 9 to 3590065 PID 3590065 (std.process tem) received signal 9

And this is what happens with the new init in perl:

PID 3589274 (process.exe) sent signal 2 to 3589291 PID 3589291 (sleep) received signal 9 PID 3589274 (process.exe) sent signal 15 to 3589338 PID 3589338 (std.process tem) received signal 9 PID 3589274 (process.exe) sent signal 9 to 3589340 PID 3589340 (std.process tem) received signal 9 PID 3589274 (process.exe) sent signal 15 to 3589341 PID 3589274 (process.exe) sent signal 15 to 3589323 PID 3589274 (process.exe) sent signal 15 to 3589320 PID 3589274 (process.exe) sent signal 15 to 3589274 PID 3589274 (process.exe) received signal 9 PID 3589341 (sleep) received signal 9 PID 3589273 (sbuild-usernsex) sent signal 9 to 3589320 PID 3589273 (sbuild-usernsex) sent signal 9 to 3589323

There are a few additional SIGTERM being sent when using the perl init, that’s helpful. At this point we are fairly convinced that process.exe is worth additional inspection. The source code of process.d shows something interesting:

1221 @system unittest 1222 { [...] 1247 auto pid = spawnProcess(["sleep", "10000"], [...] 1260 // kill the spawned process with SIGINT 1261 // and send its return code 1262 spawn((shared Pid pid) { 1263 auto p = cast() pid; 1264 kill(p, SIGINT);

So yes, there’s our sleep and the SIGINT (signal 2) right in the unit tests of process.d, just like we have observed in the bpftrace output.

Can we study the behavior of process.exe in isolation, separatedly from the build? Indeed we can. Let’s take the executable from a failed build, and try running it under /usr/libexec/sbuild-usernsexec.

First, we prepare a chroot inside a suitable user namespace:

unshare --map-auto --setuid 0 --setgid 0 mkdir /tmp/rootfs cd /tmp/rootfs cat /home/ema/.cache/sbuild/unstable-arm64.tar | unshare --map-auto --setuid 0 --setgid 0 tar xf - unshare --map-auto --setuid 0 --setgid 0 mkdir /tmp/rootfs/whatever unshare --map-auto --setuid 0 --setgid 0 cp process.exe /tmp/rootfs/

Now we can run process.exe on its own using the perl init, and trace signals at will:

/usr/libexec/sbuild-usernsexec --pivotroot --nonet u:0:100000:65536 g:0:100000:65536 /tmp/rootfs ema /whatever -- /process.exe

We can compare the behavior of the perl init vis-a-vis the one using dumb-init in milliseconds instead of minutes.

Stage 5: Oh, I see.

Why does process.exe send more SIGTERMs when using the perl init is now the big question. We have a simple reproducer, so this is where using strace becomes possible.

sudo strace --user ema --follow-forks -o sbuild-dumb-init.strace ./sbuild-usernsexec-dumb-init --pivotroot --nonet u:0:100000:65536 g:0:100000:65536 /tmp/dumbroot ema /whatever -- /process.exe

We start comparing the strace output of dumb-init with that of perl-init, looking in particular for different calls to kill.

Here is what process.exe does under dumb-init:

3593883 kill(-2, SIGTERM) = -1 ESRCH (No such process)

No such process. Under perl-init instead:

3593777 kill(-2, SIGTERM <unfinished ...>

The process is there under perl-init!

That is a kill with negative pid. From the kill(2) man page:

If pid is less than -1, then sig is sent to every process in the process group whose ID is -pid.

It would have been very useful to see this kill with negative pid in the output of bpftrace, why didn’t we? The tracepoint used, tracepoint:signal:signal_generate, shows when signals are actually being sent, and not the syscall being called. To confirm, one can trace tracepoint:syscalls:sys_enter_kill and see the negative PIDs, for example:

PID 312719 (bash) sent signal 2 to -312728

The obvious question at this point is: why is there no process group 2 when using dumb-init?

Stage 6: How did that ever work?

We know that process.exe sends a SIGTERM to every process in the process group with ID 2. To find out what this process group may be, we spawn a shell with dumb-init and observe under /proc PIDs 1, 16, and 17. With perl-init we have 1, 2, and 17. When running dumb-init, there are a few forks before launching the program, explaining the difference. Looking at /proc/2/cmdline we see that it’s bash, ie. the program we are running under perl-init. When building a package, that is dpkg-buildpackage itself.

The test is accidentally killing its own process group.

Now where does this -2 come from in the test?

2363 // Special values for _processID. 2364 enum invalid = -1, terminated = -2;

Oh. -2 is used as a special value for PID, meaning "terminated". And there’s a call to kill() later on:

2694 do { s = tryWait(pid); } while (!s.terminated); [...] 2697 assertThrown!ProcessException(kill(pid));

What sets pid to terminated you ask?

Here is tryWait:

2568 auto tryWait(Pid pid) @safe 2569 { 2570 import std.typecons : Tuple; 2571 assert(pid !is null, "Called tryWait on a null Pid."); 2572 auto code = pid.performWait(false);

And performWait:

2306 _processID = terminated;

The solution, dear reader, is not to kill.

Change the display of an event after it happens Anonymous (not verified) Fri, 12/13/2024 - 07:00 Tags Drupal Drupal Planet ECA Engagement Event Management

• Read more about Change the display of an event after it happens
• Add new comment

Event Calendars seem to be very common on the Drupal sites we build. One of the best ways of improving engagement on a site is to add content about the event after it happens. People who attended an event might come back for a recap, or to see pictures or notes from other participants, while people who did not attend can get a sense of what a future event might be like based on your past events.

Migrating from Drupal 7 to Drupal 10 or 11 can be quite challenging. Common issues, such as neglecting a detailed website analysis or failing to prioritize user training, frequently result in delays, increased costs, and frustration. In this blog post, we’ll explore the top pitfalls in Drupal migration and provide tips on how to avoid them, helping you make the transition smoother and more predictable.

Let’s go for my web review for the week 2024-50.

Census III of Free and Open Source Software

Tags: tech, foss, supply-chain

Interesting report, some findings are kind of unexpected. It’s interesting to see how much npm and maven dominate the supply chain. Clearly there’s a need for a global scheme to identify dependencies, hopefully we’ll get there.

https://www.linuxfoundation.org/research/census-iii

Open Source Archetypes: A Framework For Purposeful Open Source

Tags: tech, foss, business, strategy

An important white paper which probably went unnoticed. It gives a nice overview of the strategies one can build around Open Source components.

https://blog.mozilla.org/wp-content/uploads/2018/05/MZOTS_OS_Archetypes_report_ext_scr.pdf

Fool Me Twice We Don’t Get Fooled Again

Tags: tech, social-media, fediverse

Excellent post from Cory Doctorow about why he is only on Mastodon. Not being federated should indeed just be a deal breaker by now. Empty promises should be avoided.

https://pluralistic.net/2023/08/06/fool-me-twice-we-dont-get-fooled-again/

Firefox is the superior browser

Tags: tech, web, browser, firefox

Obviously I agree with this. It’s time people stop jumping on chromium based browsers.

https://asindu.xyz/posts/switching-to-firefox/

TRELLIS: Structured 3D Latents for Scalable and Versatile 3D Generation

Tags: tech, 3d, ai, machine-learning, generator

Looks like a nice model to produce 3D assets. Should speed up a bit the work of artists for producing background elements, I guess there will be manual adjustments needed in the end still.

https://trellis3d.github.io/

Who and What comprise AI Skepticism? - by Benjamin Riley

Tags: tech, ai, machine-learning, gpt, criticism

Excellent post showing all the nuances of AI skepticism. Can you find in which category you are? I definitely match several of them.

https://buildcognitiveresonance.substack.com/p/who-and-what-comprises-ai-skepticism

Reverse engineering of the Pentium FDIV bug

Tags: tech, cpu, hardware

It’s interesting to see such a reverse engineering of this infamous bug straight from the gates layout.

https://oldbytes.space/@kenshirriff/113606898880486330

How to Think About Time

Tags: tech, time

A good summary on the various concepts needed to reason about time.

https://errorprone.info/docs/time

Galloping Search - blag

Tags: tech, algorithm

Nice principle for a search in a sorted list when you don’t know the upper bound.

https://avi.im/blag/2024/galloping-search/

I’m daily driving Jujutsu, and maybe you should too

Tags: tech, version-control, git

Jujutsu is indeed alluring… but its long term support is questionable, that’s what keeps me away from it for now.

https://drewdevault.com/2024/12/10/2024-12-10-Daily-driving-jujutsu.html

mise-en-place

Tags: tech, tools, developer-experience

A single tool to manage your environment and dev tools across projects? Seems a bit young and needs a proper community still. I’m surely tempted to give it a spin though.

https://mise.jdx.dev/

Raw loops vs. STL algorithms

Tags: tech, c++, algorithm

An old one now, but since I keep giving this advice it seems relevant still. If you’re using raw loops at least that no again, there is likely a good alternative in the STL.

https://www.meetingcpp.com/blog/items/raw-loops-vs-stl-algorithms.html

Generic programming to fight the rigidity in the C++ projects

Tags: tech, architecture, type-systems, generics, c++

A good reminder that genericity can help fight against the rigidity one can accumulate using purely object oriented couplings… but it comes at a price in terms of complexity.

https://codergears.com/Blog/?p=945

Nobody Gets Fired for Picking JSON, but Maybe They Should? · mcyoung

Tags: tech, json, safety, type-systems

JSON is full of pitfalls. Here is a good summary. Still it is very widespread.

https://mcyoung.xyz/2024/12/10/json-sucks/

JSON5 – JSON for Humans

Tags: tech, json

Interesting JSON superset which makes it more usable for humans. I wonder if it’ll see more parsers appear.

https://json5.org/

Improving my desktop’s responsiveness with the cgroup V2 ‘cpu.idle’ setting

Tags: tech, systemd, cgroups

Nice little systemd trick, definitely an alias to add to your setup.

https://utcc.utoronto.ca/~cks/space/blog/linux/CgroupV2CpuIdleForResponsiveness

“Rules” that terminal programs follow

Tags: tech, shell, tools, unix

Good list of the undocumented rules terminal programs tend to follow. It’s nice to have this kind of consistency even though a bit by accident.

https://jvns.ca/blog/2024/11/26/terminal-rules/

htmy

Tags: tech, web, backend, frontend, python, htmx

The idea is interesting even though it probably needs to mature. It’s interesting to see this kind of libraries popup though, there’s clearly some kind of “backend - frontend split” fatigue going on.

https://volfpeter.github.io/htmy/

The errors of TeX (1989)

Tags: tech, latex, history, estimates, craftsmanship

A very precious document. Shows great organization in the work of Knuth of course but the self-reflection has profound lessons pertaining to estimates, type of errors we make, etc.

https://yurichev.com/mirrors/knuth1989.pdf

An Undefeated Pull Request Template

Tags: tech, codereview

This is indeed a nice template for submitting changes for review. It’s very thorough and helps reviewers.

https://ashleemboyer.com/blog/pull-request-template/

On the criteria to be used in decomposing systems into modules

Tags: tech, design, architecture, research

We’re still struggling about how to modularize our code. Sometimes we should go back to the basics, this paper by Parnas from 1972 basically gave us the code insights needs to modularize programs properly.

https://dl.acm.org/doi/pdf/10.1145⁄361598.361623

TDD as the crack cocaine of software

Tags: tech, tdd, flow

Indeed, it is often overlooked that TDD can really help finding a state of flow. Unlike other addictive activities presented in this article it requires a non negligible initial effort though, that’s why I wouldn’t describe it as an addiction though.

https://jefclaes.be/2014/12/tdd-as-crack-cocaine-of-software.html

Demo Driven Development

Tags: tech, agile, product-management

A good reminder of what agile is about from the product management perspective. If you can regularly demo your work you ensure a feeling of progress.

https://oanasagile.blogspot.com/2013/12/demo-driven-development.html

The 6 Mistakes You’re Going to Make as a New Manager

Tags: tech, leadership, management

Good points, this is indeed often where we are struggling when we move to a leadership role. This changes the nature of the work at least in part and we need to adjust to it.

https://terriblesoftware.org/2024/12/04/the-6-mistakes-youre-going-to-make-as-a-new-manager/

Bye for now!

Drupal Advent Calendar day 13 - Accessibility Tools track james Fri, 12/13/2024 - 09:00

Welcome back to the Drupal Advent Calendar. For our thirteenth door we are joined by Gareth Alexander, who is leading the Drupal CMS Accessibility Tools track.

When creating content there are so many things to consider: Target Audience, SEO issues like keyword relevance, making content that is actually engaging and relevant, and then there is the accessibility of your content as well.

With the Drupal CMS accessibility tools track we hope to provide a way to help with one part of that. These tools will help guide a content author to make and keep their content as accessible as possible with…

Tags

• accessibility
• Drupal Starshot

Beyond the headline features, there's a lot more new and improved under the hood in Spyder 6. Daniel Althviz, Spyder's release manager and co-maintainer, was at the forefront of much of it, and we're here to share the highlights with all of you and what he plans to work on next!

In this episode, I continued a migration of my JourneyInbox app from Heroku to DigitalOcean. We configured the secrets using 1Password and created the droplet that will host the app.

Like each month, have a look at the work funded by Freexian’s Debian LTS offering.

Debian LTS contributors

In November, 20 contributors have been paid to work on Debian LTS, their reports are available:

• Abhijith PA did 14.0h (out of 6.0h assigned and 8.0h from previous period).
• Adrian Bunk did 53.0h (out of 15.0h assigned and 85.0h from previous period), thus carrying over 47.0h to the next month.
• Andrej Shadura did 7.0h (out of 7.0h assigned).
• Arturo Borrero Gonzalez did 1.0h (out of 10.0h assigned), thus carrying over 9.0h to the next month.
• Bastien Roucariès did 20.0h (out of 20.0h assigned).
• Ben Hutchings did 0.0h (out of 24.0h assigned), thus carrying over 24.0h to the next month.
• Chris Lamb did 18.0h (out of 18.0h assigned).
• Daniel Leidert did 17.0h (out of 26.0h assigned), thus carrying over 9.0h to the next month.
• Emilio Pozuelo Monfort did 40.5h (out of 60.0h assigned), thus carrying over 19.5h to the next month.
• Guilhem Moulin did 7.25h (out of 7.5h assigned and 12.5h from previous period), thus carrying over 12.75h to the next month.
• Jochen Sprickerhof did 3.5h (out of 10.0h assigned), thus carrying over 6.5h to the next month.
• Lee Garrett did 14.75h (out of 15.25h assigned and 44.75h from previous period), thus carrying over 45.25h to the next month.
• Lucas Kanashiro did 10.0h (out of 54.0h assigned and 10.0h from previous period), thus carrying over 54.0h to the next month.
• Markus Koschany did 20.0h (out of 40.0h assigned), thus carrying over 20.0h to the next month.
• Roberto C. Sánchez did 6.75h (out of 9.75h assigned and 14.25h from previous period), thus carrying over 17.25h to the next month.
• Santiago Ruano Rincón did 24.75h (out of 23.5h assigned and 1.5h from previous period), thus carrying over 0.25h to the next month.
• Sean Whitton did 2.0h (out of 6.0h assigned), thus carrying over 4.0h to the next month.
• Sylvain Beucler did 21.5h (out of 9.5h assigned and 50.5h from previous period), thus carrying over 38.5h to the next month.
• Thorsten Alteholz did 11.0h (out of 11.0h assigned).
• Tobias Frost did 12.0h (out of 10.5h assigned and 1.5h from previous period).

Evolution of the situation

In November, we have released 38 DLAs.

The LTS coordinators, Roberto and Santiago, delivered a talk at the Mini-DebConf event in Toulouse, France. The title of the talk was “How LTS goes beyond LTS”. The talk covered work done by the LTS Team during the past year. This included contributions related to individual packages in Debian (such as tomcat, jetty, radius, samba, apache2, ruby, and many others); improvements to tooling and documentation useful to the Debian project as a whole; and contributions to upstream work (apache2, freeimage, node-dompurify, samba, and more). Additionally, several contributors external to the LTS Team were highlighted for their contributions to LTS. Readers are encouraged to watch the video of the presentation for a more detailed review of various ways in which the LTS team has contributed more broadly to the Debian project and to the free software community during the past year.

We wish to specifically thank Salvatore (of the Debian Security Team) for swiftly handling during November the updates of needrestart and libmodule-scandeps-perl, both of which involved arbitrary code execution vulnerabilities. We are happy to see increased involvement in LTS work by contributors from outside the formal LTS Team.

The work of the LTS Team in November was otherwise unremarkable, encompassing the customary triage, development, testing, and release of numerous DLAs, along with some associated contributions to related packages in stable and unstable.

Thanks to our sponsors

Sponsors that joined recently are in bold.

• Platinum sponsors:
  • Toshiba Corporation (for 110 months)
  • Civil Infrastructure Platform (CIP) (for 78 months)
  • VyOS Inc (for 42 months)
• Gold sponsors:
  • Roche Diagnostics International AG (for 120 months)
  • Akamai - Linode (for 114 months)
  • Babiel GmbH (for 104 months)
  • Plat’Home (for 103 months)
  • CINECA (for 78 months)
  • University of Oxford (for 60 months)
  • Deveryware (for 47 months)
  • EDF SA (for 32 months)
  • Dataport AöR (for 7 months)
  • CERN (for 5 months)
• Silver sponsors:
  • Domeneshop AS (for 125 months)
  • Nantes Métropole (for 119 months)
  • Univention GmbH (for 111 months)
  • Université Jean Monnet de St Etienne (for 111 months)
  • Ribbon Communications, Inc. (for 105 months)
  • Exonet B.V. (for 95 months)
  • Leibniz Rechenzentrum (for 89 months)
  • Ministère de l’Europe et des Affaires Étrangères (for 73 months)
  • Cloudways by DigitalOcean (for 62 months)
  • Dinahosting SL (for 60 months)
  • Bauer Xcel Media Deutschland KG (for 54 months)
  • Platform.sh SAS (for 54 months)
  • Moxa Inc. (for 48 months)
  • sipgate GmbH (for 46 months)
  • OVH US LLC (for 44 months)
  • Tilburg University (for 44 months)
  • GSI Helmholtzzentrum für Schwerionenforschung GmbH (for 35 months)
  • Soliton Systems K.K. (for 32 months)
  • THINline s.r.o. (for 8 months)
  • Copenhagen Airports A/S
• Bronze sponsors:
  • Evolix (for 125 months)
  • Seznam.cz, a.s. (for 125 months)
  • Intevation GmbH (for 122 months)
  • Linuxhotel GmbH (for 122 months)
  • Daevel SARL (for 121 months)
  • Bitfolk LTD (for 120 months)
  • Megaspace Internet Services GmbH (for 120 months)
  • Greenbone AG (for 119 months)
  • NUMLOG (for 119 months)
  • WinGo AG (for 118 months)
  • Entr’ouvert (for 110 months)
  • Adfinis AG (for 107 months)
  • Laboratoire LEGI - UMR 5519 / CNRS (for 102 months)
  • Tesorion (for 102 months)
  • GNI MEDIA (for 101 months)
  • Bearstech (for 93 months)
  • LiHAS (for 93 months)
  • Catalyst IT Ltd (for 88 months)
  • Supagro (for 83 months)
  • Demarcq SAS (for 82 months)
  • Université Grenoble Alpes (for 68 months)
  • TouchWeb SAS (for 60 months)
  • SPiN AG (for 57 months)
  • CoreFiling (for 53 months)
  • Institut des sciences cognitives Marc Jeannerod (for 48 months)
  • Observatoire des Sciences de l’Univers de Grenoble (for 44 months)
  • Tem Innovations GmbH (for 39 months)
  • WordFinder.pro (for 38 months)
  • CNRS DT INSU Résif (for 37 months)
  • Alter Way (for 30 months)
  • Institut Camille Jordan (for 20 months)
  • SOBIS Software GmbH (for 5 months)