Feeds

The majority of machine learning projects that you read about or work on are built around batch processes. The model is trained, and then validated, and then deployed, with each step being a discrete and isolated task. Unfortunately, the real world is rarely static, leading to concept drift and model failures. River is a framework for building streaming machine learning projects that can constantly adapt to new information. In this episode Max Halford explains how the project works, why you might (or might not) want to consider streaming ML, and how to get started building with River.Preamble

This is a cross-over episode from our new show The Machine Learning Podcast, the show about going from idea to production with machine learning.

Summary

The majority of machine learning projects that you read about or work on are built around batch processes. The model is trained, and then validated, and then deployed, with each step being a discrete and isolated task. Unfortunately, the real world is rarely static, leading to concept drift and model failures. River is a framework for building streaming machine learning projects that can constantly adapt to new information. In this episode Max Halford explains how the project works, why you might (or might not) want to consider streaming ML, and how to get started building with River.

Announcements

• Hello and welcome to the Machine Learning Podcast, the podcast about machine learning and how to bring it from idea to delivery.
• Building good ML models is hard, but testing them properly is even harder. At Deepchecks, they built an open-source testing framework that follows best practices, ensuring that your models behave as expected. Get started quickly using their built-in library of checks for testing and validating your model’s behavior and performance, and extend it to meet your specific needs as your model evolves. Accelerate your machine learning projects by building trust in your models and automating the testing that you used to do manually. Go to themachinelearningpodcast.com/deepchecks today to get started!
• Your host is Tobias Macey and today I’m interviewing Max Halford about River, a Python toolkit for streaming and online machine learning

Interview

• Introduction
• How did you get involved in machine learning?
• Can you describe what River is and the story behind it?
• What is "online" machine learning?
  • What are the practical differences with batch ML?
  • Why is batch learning so predominant?
  • What are the cases where someone would want/need to use online or streaming ML?
• The prevailing pattern for batch ML model lifecycles is to train, deploy, monitor, repeat. What does the ongoing maintenance for a streaming ML model look like?
  • Concept drift is typically due to a discrepancy between the data used to train a model and the actual data being observed. How does the use of online learning affect the incidence of drift?
• Can you describe how the River framework is implemented?
  • How have the design and goals of the project changed since you started working on it?
• How do the internal representations of the model differ from batch learning to allow for incremental updates to the model state?
• In the documentation you note the use of Python dictionaries for state management and the flexibility offered by that choice. What are the benefits and potential pitfalls of that decision?
• Can you describe the process of using River to design, implement, and validate a streaming ML model?
  • What are the operational requirements for deploying and serving the model once it has been developed?
• What are some of the challenges that users of River might run into if they are coming from a batch learning background?
• What are the most interesting, innovative, or unexpected ways that you have seen River used?
• What are the most interesting, unexpected, or challenging lessons that you have learned while working on River?
• When is River the wrong choice?
• What do you have planned for the future of River?

Contact Info

• Email
• @halford_max on Twitter
• MaxHalford on GitHub

Parting Question

• From your perspective, what is the biggest barrier to adoption of machine learning today?

Closing Announcements

• Thank you for listening! Don’t forget to check out our other shows. The Data Engineering Podcast covers the latest on modern data management. Podcast.__init__ covers the Python language, its community, and the innovative ways it is being used.
• Visit the site to subscribe to the show, sign up for the mailing list, and read the show notes.
• If you’ve learned something or tried out a project from the show then tell us about it! Email hosts@themachinelearningpodcast.com) with your story.
• To help other people find the show please leave a review on iTunes and tell your friends and co-workers

Links

• River
• scikit-multiflow
• Federated Machine Learning
• Hogwild! Google Paper
• Chip Huyen concept drift blog post
• Dan Crenshaw Berkeley Clipper MLOps
• Robustness Principle
• NY Taxi Dataset
• RiverTorch
• River Public Roadmap
• Beaver tool for deploying online models
• Prodigy ML human in the loop labeling

The intro and outro music is from Hitman’s Lovesong feat. Paola Graziano by The Freak Fandango Orchestra/CC BY-SA 3.0

Following last week animated PDF adventure, I have been reading a series of one page dungeons... And yesterday I had the opportunity to play the best one in my opinion: The Sky-Blind Spire by Michael Prescott.

It has everythng I love on one page: a maze to explore, mysteries to …

— Permalink

Sometimes when you see an elegant algorithm, you think "looks great, I just need it to also do X". Perhaps you are able to build X directly out of what the algorithm gives you; fantastic. Or, perhaps you can alter the algorithm a bit, and it works just as well while also doing X. Sometimes, though, you alter the algorithm and things go pear-shaped.

Tonight's little note builds on yesterday's semi-space collector article and discusses an worse alternative to the Cheney scanning algorithm.

To recall, we had this visit_field function that takes a edge in the object graph, as the address of a field in memory containing a struct gc_obj*. If the edge points to an object that was already copied, visit_field updates it to the forwarded address. Otherwise it copies the object, thus computing the new address, and then updates the field.

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) { size_t size = heap_object_size(obj); struct gc_obj *new_obj = (struct gc_obj*)heap->hp; memcpy(new_obj, obj, size); forward(obj, new_obj); heap->hp += align_size(size); return new_obj; } void visit_field(struct gc_obj **field, struct gc_heap *heap) { struct gc_obj *from = *field; struct gc_obj *to = is_forwarded(from) ? forwarded(from) : copy(heap, from); *field = to; }

Although a newly copied object is in tospace, all of its fields still point to fromspace. The Cheney scan algorithm later visits the fields in the newly copied object with visit_field, which both discovers new objects and updates the fields to point to tospace.

One disadvantage of this approach is that the order in which the objects are copied is a bit random. Given a hierarchical memory system, it's better if objects that are accessed together in time are close together in space. This is an impossible task without instrumenting the actual data access in a program and then assuming future accesses will be like the past. Instead, the generally-accepted solution is to ensure that objects that are allocated close together in time be adjacent in space. The bump-pointer allocator in a semi-space collector provides this property, but the evacuation algorithm above does not: it would need to preserve allocation order, but instead its order is driven by graph connectivity.

I say that the copying algorithm above is random but really it favors a breadth-first traversal; if you have a binary tree, first you will copy the left and the right nodes of the root, then the left and right children of the left, then the left and right children of the right, then grandchildren, and so on. Maybe it would be better to keep parent and child nodes together? After all they are probably allocated that way.

So, what if we change the algorithm:

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) { size_t size = heap_object_size(obj); struct gc_obj *new_obj = (struct gc_obj*)heap->hp; memcpy(new_obj, obj, size); forward(obj, new_obj); heap->hp += align_size(size); trace_heap_object(new_obj, heap, visit_field); // * return new_obj; } void visit_field(struct gc_obj **field, struct gc_heap *heap) { struct gc_obj *from = *field; struct gc_obj *to = is_forwarded(from) ? forwarded(from) : copy(heap, from); *field = to; }

Here we favor a depth-first traversal: we eagerly call trace_heap_object within copy. No need for the Cheney scan algorithm; tracing does it all.

void collect(struct gc_heap *heap) { flip(heap); uintptr_t scan = heap->hp; trace_roots(heap, visit_field); }

The thing is, this works! It might even have better performance for some workloads, depending on access patterns. And yet, nobody does this. Why?

Well, consider a linked list with a million nodes; you'll end up with a million recursive calls to copy, as visiting each link eagerly traverses the next. While I am all about unbounded recursion, an infinitely extensible stack is something that a language runtime has to provide to a user, and here we're deep into implementing-the-language-runtime territory. At some point a user's deep heap graph is going to cause a gnarly system failure via stack overflow.

Ultimately stack space needed by a GC algorithm counts towards collector memory overhead. In the case of a semi-space collector you already need twice the amount memory as your live object graph, and if you recursed instead of iterated this might balloon to 3x or more, depending on the heap graph shape.

Hey that's my note! All this has been context for some future article, so this will be on the final exam. Until then!

Everyone’s talking about ChatGPT. If you haven’t used it yourself, then you’ve probably seen lots of screenshots on social media. Or maybe you’ve had a friend or family member (like me) talk about it nonstop.

I’ve indeed spent a lot of time playing with ChatGPT. I’ve gotten it to do some wild and fun things, some of which have made me laugh very hard. I thought that it would be fun to share my experiences, including the many laughs, and point to what I see as some real strengths, some serious weaknesses, and also some questions about where to go from here.

I should add that I’m not an expert in machine learning or artificial intelligence. My comments come as an experienced software developer, not as someone who is close to this technology’s present or future.

What’s so special about ChatGPT?

When I was in college, natural-language recognition was bad and slow. If you were fortunate, you could connect to a research lab’s computer and it questions, using a limited vocabulary on a specific topic. Any answer was like magic.

Since then, chatbots — software to which you can write commands, questions, and requests — are everywhere. Whether it’s my bank, supermarket, or hotel-booking site, the first line of support is a chatbot. These chatbots work in a specific domain, have limited information at hand, and are primarily meant to ensure that the simplest, most common questions don’t have to take up a human’s valuable time.

ChatGPT, from a company called OpenAI, is also a chatbot. But it’s far more sophisticated than these customer-service bots we’ve gotten used to:

1. It handles a very wide variety of inputs. You can ask many types of questions, and it’ll largely understand what you’re saying.
2. It has access to a large corpus of knowledge. I’ve heard that it was trained on hundreds of millions of documents.
3. It can produce output in a lot of different formats, including text, dialogue, screenplays, and poetry.

The combination of these three things, freely available from your browser, makes it easy to get answers to lots of questions. The results are quite different from Google: Rather than evaluating which of the links are most likely to answer your question, you just get a response.

For example:

Here, we see that you can ask a question of ChatGPT in free-form English. It gives you a fairly concise, fairly reasonable answer.

Unlike Google, it isn’t finding this answer in its memory, or from having scanned many Web pages. Rather, it’s making an association between the words in my question and the words it has in its training documents. It’s then creating a new document, on the fly, trying to answer my question.

Another nice thing about ChatGPT is that it has “state,” meaning that it remembers what you’ve asked and told it from one question to another. So you don’t need to restate an entire question; you can just tweak your request. Admittedly, the results can be a bit … weird:

Let’s have some fun

The fun begins when you ask it to start to explain things that don’t make any sense. For example:

I don’t know if I’d really call this the style of Thomas Jefferson. And it’s a ridiculous argument. And yet, it’s thoroughly entertaining. Moreover, if we ignore the content, it’s still pretty amazing that ChatGPT was able to understand my question and compose a coherent three-paragraph answer on the subject.

I decided to see how far I could push this:

I mean, it’s not wrong. But any human asked this question would think that you were being absurd for making such a request. ChatGPT doesn’t (often) have such a filter, and is willing to entertain almost anything.

I decided to see if we could push this any further:

Again, I’m super impressed on a technical level: It created a song, in seconds. It included lots of context from Hamlet (which I didn’t provide) as part of the song. And it was willing to go along with my totally ridiculous request.

I then added a new request:

Yes, the audience of a musical-comedy version of Hamlet would likely be “shocked and confused” if Darth Vader were to kill everyone during the finale. That’s a very astute comment.

I decided to try something else:

Again, the fact that it’s able to produce such text at all is a remarkable achievement. It knows that the villain should be dressed in black, it uses appropriate language to describe it, and it talks about the doctors curing the virus.

But even I recognize that this isn’t a very sophisticated song.

That didn’t stop me from continuing, though:

It can write other songs, as well:

If you’re starting to think that all of these songs look and feel very similar… I’d agree with you. It’s amazing to see such text produced so quickly, but that doesn’t mean it can compete with skilled humans.

That said, this one was pretty good:

ChatGPT can create limericks, too:

It’s not exactly the right rhythm you would expect from a limerick, but.. it’s not terrible.

To its credit, ChatGPT has been programmed to avoid violence and discrimination. There are ways around it, but it does try:

TV episodes and crossovers

It can even describe TV episodes based on existing series. For example:

Of course, I asked it to write a song:

But why stop with one series, when we can have a special crossover episode of multiple, completely different, TV series?

Notice how everything needs to end well? They keep pushing for peace and harmony, especially when it comes to characters in children’s series. For example:

It’s not wrong, you know. Color me impressed.

Sometimes, it just flat-out refuses:

I even asked it to create a screenplay for when a volcano destroys Mr. Hooper’s store. The result? Everyone is safe, rebuilds the store together, and learns the value of teamwork.

Here, I asked it to write a screenplay, I thought that we could get some violence by including a horror film in the crossover:

This other one worked well, too:

By the way, it’s amazing to me that ChatGPT talks about Tim Russert as the host of Meet the Press.

I asked it to create a screenplay in which a candidate takes a programming exam. That was a bit dry, so I asked again, this time saying that it was a “screwball comedy” with a “young, zany” developer candidate. Here’s what I got:

Um, what the heck are “inflatable clown shoes”?

Anyway: Many developers have pointed to ChatGPT’s ability to write code, as well as to debug it. I haven’t done much with the debugging, but I did ask it to write some code. Actually, I asked it to write a final exam for a Python course, along with solution code and test functions:

Amazing, right? And yet, the questions it came up with were simplistic. The solutions it provided weren’t the best. If someone were to give me these sorts of solutions on a final exam, they would pass, but they wouldn’t get a fantastic grade. (More on that to come later.)

Remember that ChatGPT can express the same information in multiple formats. So, I asked it to describe the Point class that it created for this exam in a few ways:

As you can see from this song, ChatGPT can produce a limited number of output styles. As long as you fit into one of those known styles, you’ll be fine. For example:

You would think that changing the debate style would change .. something other than the names. But no:

The names have changed, but the content hasn’t. (I should note that I was previously able to just say “In the style of the Talmud,” but now I had to name some scholars. I’m guessing that some tweaking is going on behind the scenes.) It’s using the context to make some very small changes to the output, but the core of its “debate” output is pretty much the same.

For example:

And then:

These debates always ended up pretty much the same. They would sprinkle in some words to give it a different context, but that’s about it.

Trickery and mistakes

For now, at least, ChatGPT will insist that it’s not connected to the Internet, that its knowledge stops at 2021, and that you can’t use it for general queries. But if you ask it to pretend to be someone with knowledge of a subject, it’s more than happy to play along:

It’s also willing to be quite wrong:

Just for the record, I didn’t invent Python. Python doesn’t have a CEO. Moreover, I wasn’t born and raised in Israel, I attended MIT (but did spend a semester at the Technion), never did a project for Google, didn’t co-found the Python Software Foundation, didn’t write either of the books that they ascribe to me, wasn’t awarded the PSF’s Community Service Award, and didn’t get the TIOBE Prize.

But sure, other than that, it’s accurate.

It also gets things just plain ol’ wrong. And then, it can sometimes get defensive about its answers:

And many people have discovered that while it can write haiku, the number of syllables is often wrong:

As part of my creation of TV episodes, I asked it to write one with Donald Trump as the contestant on “Let’s Make a Deal.” Notice that ChatGPT seems unaware of the fact that a game-show contestant typically doesn’t get prizes that he himself created. However — spoiler alert! — I couldn’t have chosen a better final prize:

Trusting ChatGPT (or not)

Someone on Twitter suggested that you could ask ChatGPT to tell Jewish morality tales typical among Hassidic Jews. I asked ChatGPT to tell me such a story based on last Saturday’s weekly portion in Genesis (known in Hebrew as “Vayishlach”), and rye bread. (Why rye bread? Because I wanted something totally weird and random.) The result, while superficial, was better than I expected:

But then I said, “What if I were to add constraints?” I asked ChatGPT not just to connect Vayishlach and rye bread, but also the Muppets and smartphone addiction. It came up with the following:

Very impressive, right? Yes, it is — except that religiously observant Jews don’t use the phone on Shabbat, aka the Sabbath. The notion that a Hasidic story would have someone using his phone at the Shabbat table is totally absurd. The fact that ChatGPT would include this fact demonstrates how powerful the technology is, and how little common sense it has, too.

I decided to see how far I could push this, and came up with the following:

In case it’s not obvious, bacon is completely prohibited by Jewish law, and the notion that a Hasidic story would describe someone gaining insights about the Torah while eating bacon and eggs is quite funny to anyone in the know.

The problems with ChatGPT

And here, one of the real problems with ChatGPT: If it’s meant to be used for entertainment, then it’s a huge success. But OpenAI is clearly aiming to answer lots of questions on a wide variety of subjects. The style of answer that it provides, and the breadth of topics it aims to deal with invites people to get help. Already, we’re hearing about students using ChatGPT to write college admission essays and answer questions for take-home tests.

If you’re not an expert, you might not realize how incredibly, obviously wrong ChatGPT is in its answers. The fact that it doesn’t provide any sources, or tell you how it knows what it knows, is a problem. ChatGPT is always quite confident that it has a great answer, described on Twitter as “mansplaining as a service.” It’s often right, but not always — and distinguishing between the two requires real knowledge.

Moreover, it comes up with very bland prose. It might be grammatically correct, but it’s quite a monotonous snooze to read paragraphs of ChatGPT’s text. Especially if an instructor has seen a student’s previous written assignments, I have to assume that they would be suspicious to get such things handed to them by a students.

On Slate’s “Political Gabfest” this past week, the bonus segment talked about ChatGPT, and brought up the problem of cheating in school. John Dickerson mentioned that he has heard of AI tools which analyze the text that a student has turned in, and then produces questions for the student to answer about their own work. If the student can answer those questions, then it’s clearly their own work. If not, then they likely got help.

I love the idea of giving students oral exams after they’ve written papers. That strikes me as a great educational model. But will teachers agree? Are they prepared for it? Are students willing to do that? It sounds like a huge overhaul in education, merely to ensure that AI isn’t being used to produce mediocre papers. And yet, it might be needed, because we know that many students will gladly copy and cheat.

I do think that ChatGPT is an amazing technology. But, as with all technologies, it has to be used in the right ways. I can imagine using it for rough drafts and outlines, or for brainstorming. If and when it provides sources, it could indeed be easier and more effective to find useful info than Google. And, as was mentioned on this week’s “Hard Fork” podcast, the fact that Microsoft has invested heavily in OpenAI might breathe some life and competition into the Google vs. Bing fight.

Also: This is likely not the last such super-chatbot that we’ll see. OpenAI is undoubtedly working on new ones. And other companies are investing heavily in AI, too.

This also raises all sorts of questions about copyright infringement: Whose documents were used to seed ChatGPT? If a phrase from my article is quoted by ChatGPT, and someone else uses that phrase in a book that earns them millions, am I due something for it? Heck, if OpenAI makes lots of money via ChatGPT’s commercial successors, will they pay me for having used my data?

As I wrote above, I’m sure that others will be creating chatbots like ChatGPT in the near future. This might be because I asked ChatGPT to propose a budget, and it doesn’t seem like creating a competitor would be all that expensive:

Then again, this might be another case of it being confidently wrong.

Anyway, what do you think? I’ve had great fun playing with ChatGPT over the last week, and I’m fascinated by what it can do. But I do worry that people will believe it in its current form. And if it gets much better in the next few years, we might indeed have some new issues to deal with.

By the way, I recommend that you listen to the latest episode of “Hard Fork” (https://www.nytimes.com/2022/12/09/podcasts/can-chatgpt-make-this-podcast.html), which talks about ChatGPT and some of its implications.

The post My Week with ChatGPT appeared first on Reuven Lerner.

Whilst working on a module I found that I needed to do something that was difficult to find information for. I think part of that is because the solution is simpler than I was thinking, but it did cause me to get lost in source code and internet issues for a while.

What I needed to do was to load in a list of configuration entities that matched certain parameters. The configuration entity I was dealing with was used to manipulate the output of a form, but only if certain conditions were met first. I thought that this information might be useful to others who are looking to load configuration entities.

In this article I will show how to search for configuration entities and how to load those entities once found. I'll be using blocks for the examples here, but the same rules will apply to any configuration entities you want to load.

Loading The Entity Query Service

What we need to do first is get the entity query service. In case it wasn't obvious (and it wasn't to me) you can search for configuration entities in the same way as you would any other sort of entity. If you have used entity query to find users or pages of content then this is the same mechanism. There are, however, a couple of ways to go about doing this.

The first (and simplest) way of loading entity query is to just grab the object for the configuration entity we want to load.

$entityQuery = \Drupal::entityQuery('block');

Whilst using this method does allow us to search for configuration entities, it doesn't allow us to load those entities. To do that we need to use entity_type.manager service to find the storage for the configuration entity we want to load.

Read more.

Fund xjm as a Drupal core release manager Hi, I'm xjm. I'm a Drupal core committer, one of eighteen people who are trusted to accept (merge) code changes to the Drupal project for the hundreds of thousands of sites that rely on it.

My role on the committer team is that of a release manager. We're the folks who actually create the Drupal core releases that you can install on your site.

xjm Sun, 12/11/2022 - 12:44

In this blog post I want to describe a recent bug finding technique that I've added to the PyPy JIT testing infrastructure. This technique uses the Z3 theorem prover to find bugs in the optimizer of PyPy's JIT, in particular its integer operation optimizations. The approach is based on things I have learned from John Regehr's blog (this post is a good first one to read), Twitter, and on his (et al) paper Alive2: Bounded Translation Validation for LLVM. The work was triggered by a recent miscompilation bug my current bachelor student Nico Rittinghaus found.

Background: Python Integers in the PyPy JIT

The optimizer of PyPy's JITs operates on traces, which are linear sequences of instructions with guards. The instructions in the traces operate on different machine-level data types, machine integers, doubles, pointers, bools, etc. In this post we'll be mostly concerned with machine integers.

To given some wider context I'll explain a bit how Python ints in the user code relate to the types that are used in traces when the PyPy Python implementation is used. When PyPy turns a regular Python 3 function into a trace, there is a lot of work happening in the JIT frontend to try to observe and infer the types that the Python function concretely uses at runtime. The traces are generated under these typing assumptions. Therefore, code that uses ints in the Python code can typically be translated into traces that operate on machine integers. In order to make sure that the Python integer semantics are upheld, many of the operations in the traces need to check that the integer results of some operations still fit into a machine integer. If that is not the case (a rare situation for most programs), the trace is left via a guard, execution falls back to the interpreter, and there a big integer representation is chosen for the too big value (the big integer representation is done via a pointer and some storage on the heap).

All of this machinery is not going to be too relevant for the rest of the post. For the post it's important to know that trace instructions operate on machine integers and other low-level types, and some of the operations can optionally check whether the results still fit into a machine integer. These trace operations are improved by the optimizer, which tries to transform the trace into one that behaves the same, but is less costly to execute.

Background: Bounds Analysis in PyPy's JIT

The optimizer of PyPy's JIT has an analysis based on abstract interpretation that tries to find out whether the integer values stored in a variable are actually not using the full 64 bit (or 32 bit) range, but instead fit into some smaller range. This means that for every integer variable x in a trace, the JIT compiler tracks upper and lower bounds of the runtime value of that variable: a range [a, b] such that for every concrete runtime value v that gets stored in variable x, a <= v <= b must be true. a and b start out as the most general MININT and MAXINT, but sometimes there is extra information that makes it possible to improve these known bounds, and that is often useful to optimize the code.

A typical example is that the JIT knows that the length of a string is non-negative, so for this kind of code: x = len(s) where s is a string, x gets a range [0, MAXINT] assigned. With this information we could for example remove a check x + 10 < 0 completely, because it can never be true.

The bounds information is useful for optimization, but the analysis of the bounds is also a source of bugs in the JIT, because the reasoning is often subtle and easy to get wrong in corner cases. We already use a number of testing techniques to try to make sure that it is correct. A simple one is property-based testing using Hypothesis on the operations on bounds. Even though Hypothesis is fantastic, it unfortunately does not catch absolutely all the bugs even if we'd like it too, as we'll see in the next section.

Motivation: A JIT Miscompilation

I am currently supervising a Bachelor thesis by Nico Rittinghaus, who is extending the integer analysis in the JIT. He'll probably write a separate blog post about that soon. In the process of his work, the current bounds analysis code got a lot of scrutiny, and we found out that one of the unit tests of the bounds analysis was actually incorrect, and the example code in that unit test was optimized incorrectly. This case of incorrect optimization is not a big deal for regular Python code, because it involved a "wrapping integer addition operation", i.e. one where overflowing results just wrap around to negative values. All the additions and other arithmetic operations that the PyPy Python frontend generates actually have overflow checks (to be able to switch to a big integer representation if needed). However, it's still possible to trigger the problem with the __pypy__.intop.int_add API which is a function that exposes wraparound arithmetic on Python ints.

Here's the miscompilation. The JIT optimizes the following function:

import __pypy__ def wrong(x): a = __pypy__.intop.int_add(x, 10) if a < 15: if x < 6: return 0 return 1 return 2

Into the following code:

import __pypy__ def wrong(x): a = __pypy__.intop.int_add(x, 10) if a < 15: return 0 return 2

Basically the faulty reasoning of the JIT looks like this: if int_add(x, 10) < 15 then it must follow that x < 5, which is stronger than x < 6, so the second if is always true. This sounds good, but is actually wrong if the addition + 10 wrapped around. So if x == MAXINT, then int_add(x, 10) == MININT + 9 < 15. But MAXINT < 5 is not correct.

Note how the same reasoning with overflow-checking addition is correct! If x + 10 < 15 and the + didn't overflow, then indeed x < 6. And if your mind bends starting to think about all this, you understand some of the difficulty of getting the JIT correct in this area.

How could we have avoided this bug?

One exercise I try to do after finding bugs is to reflect on ways that the bug could have been avoided. I think this is particularly important in the JIT, where bugs are potentially really annoying to find and can cause very strange behaviour in basically arbitrary Python code.

It's easy to always answer this question with "try to think more carefully when working", but that approach cannot be relied on in complicated situations, because humans don't concentrate perfectly for long stretches of time.

A situation-specific problem I identified was the bad design of the range analysis API. A range is not just represented by two numbers, instead it's two numbers and two bools that are supposed to represent that some operation did or did not underflow/overflow. The meaning of these bools was quite hard to grasp and easy to get wrong, so probably they should never have been introduced in the first place (and my bugfix indeed removed them).

But in the rest of this blog post I want to talk about another, systematic approach that can be applied to the problem of mis-optimizations of integer operations, and that is done by applying an SMT solver to the problem.

An SMT solver (Satisfyability Modulo Theories) is a tool that can be used to find out whether mathematical formulas are "satisfiable", i.e. whether some chosen set of inputs exists that will make the formulas evaluate to true. SMT solvers are commonly used in a wide range of CS applications including program correctness proofs, program synthesis, etc. The most widely known one is probably Z3 by Microsoft Research which has the nice advantage of coming with an easy-to-use Python binding.

Going into this I basically knew next to nothing about SMT solvers (despite having been embedded in a formal methods research group for years!) so it was an interesting new world to learn about.

As briefly mentioned in the introduction, the approach I took followed a similar (but much more properly executed) one applied to LLVM operations, called Alive2. Krister Waldfridsson has done similar work for GCC recently, described on his blog.

Z3 Proof of Concept

The first thing I did was to try to get Z3 find the above bug, by encoding the input program into an SMT formula by hand and trying to get Z3 to prove the condition that the JIT thinks is always true. The Z3 code for this looks as follows:

from z3 import BitVec, Implies, prove x = BitVec('x', 64) a = x + 10 cond1 = a < 15 cond2 = x < 6 prove(Implies(cond1, cond2))

Here, x is defined to be a bit vector variable of width 64, which is a datatype that can be used to represent bounded machine integers. Addition on bit vectors performs wraparound arithmetic, like the __pypy__.intop.int_add call in the original code. The JIT optimized the second condition away, so essentially it was convinced that the first condition implies the second one. The above snippet tries to get Z3 to confirm this.

When run, the above program prints:

counterexample [x = 9223372036854775803]

Which shows the bug. As a small side-note, I thought it was cool that the process of "proving" something in Z3 basically means trying to find an example for the negation of the formula. If no counterexample can be found for the negation, the original formula is true. If the original formula turns out to be false (like here) we get a nice example that shows the problem to go with it.

It's not realistic to hand-translate all the hundreds of unit-tests into Z3 formulas and then ask Z3 to prove the optimizations. Instead, we want to have a program that does this for us.

SMT Checking of the JIT Optimizer

What we want from this program is the following: given an unoptimized trace and its optimized version, we want to use Z3 to check whether the optimized trace behaves identically to the unoptimized one. One question is what "behaves identically" means. What we care about is the outputs of the trace being the same values, no matter how they are computed. Also, for every guard we want to make sure that it fails in identical ways in the optimized and unoptimized versions. A guard is only allowed to be optimized away if it can never fail. The code that comes after a guard can assume that the guard has not failed, because otherwise execution would have left the trace. All of this should be true regardless for the values of the input variables of the trace.

So in order to check that the two traces are behaving identically, we do the following:

• We create Z3 variables for every input variable. We use the same input variables both for the unoptimized as well as the optimized trace.

• We align the two traces at the corresponding guards. Thankfully the optimizer keeps track of which optimized guard corresponds to which unoptimized input guard.

• All the operations before a guard are translated into Z3 formulas, for both versions of the trace.

• For two corresponding guards, we ask Z3 to prove that the guard conditions are identical.

• For a guard that was optimized away we ask Z3 to prove that the condition is always true.

• After a guard, we tell Z3 that from now on it can assume that the guard condition is true.

• We repeat this, guard for guard, until we reach the end of the trace. There, we ask Z3 to prove that the output variables in the unoptimized trace and the optimized trace are identical (every trace can return one or many values).

I implemented this, it's not a lot of code, basically a couple of hundred lines of (somewhat hacky) Python code. So far I only support integer operations. Here are some parts of the code to give you a flavor of what this looks like.

This is the code that translates operations into Z3 formulas:

def add_to_solver(self, ops, state): for op in ops: if op.type != 'v': # is it an operation with a result res = self.newvar(op) else: # or does it return void res = None # ... # convert arguments if op.numargs() == 1: arg0 = self.convertarg(op, 0) elif op.numargs() == 2: arg0 = self.convertarg(op, 0) arg1 = self.convertarg(op, 1) # compute results if opname == "int_add": expr = arg0 + arg1 elif opname == "int_sub": expr = arg0 - arg1 elif opname == "int_mul": expr = arg0 * arg1 elif opname == "int_and": expr = arg0 & arg1 elif opname == "int_or": expr = arg0 | arg1 elif opname == "int_xor": expr = arg0 ^ arg1 # ... more operations, some shown below self.solver.add(res == expr)

New Z3 variables are defined by the helper function newvar, which adds the operation to a dictionary box_to_z3 mapping boxes (=variables) to Z3 variables. Due to the SSA property that traces have, a variable must be defined before its first use.

Here's what newvar looks like (LONG_BIT is a constant that is either 64 or 32, depending on the target architecture):

def newvar(self, box, repr=None): # ... some logic around making the string representation # somewhat nicer omitted result = z3.BitVec(repr, LONG_BIT) self.box_to_z3[box] = result return result

The convert method turns an operation argument (either a constant or a variable) into a Z3 formula (either a constant bit vector or an already defined Z3 variable). convertarg is a helper function that takes an operation, reads its nth argument and converts it.

def convert(self, box): if isinstance(box, ConstInt): return z3.BitVecVal(box.getint(), LONG_BIT) return self.box_to_z3[box] def convertarg(self, box, arg): return self.convert(box.getarg(arg))

The lookup of variables in box_to_z3 that convert does cannot fail, because the variable must have been defined before use.

Comparisons return the bit vector 0 or bit vector 1, we use a helper function cond to turn the Z3 truth value of the comparison into a bit vector:

def cond(self, z3expr): return z3.If(z3expr, TRUEBV, FALSEBV) def add_to_solver(self, ops, state): # ... start as above # more cases elif opname == "int_eq": expr = self.cond(arg0 == arg1) elif opname == "int_ne": expr = self.cond(arg0 != arg1) elif opname == "int_lt": expr = self.cond(arg0 < arg1) elif opname == "int_le": expr = self.cond(arg0 <= arg1) elif opname == "int_gt": expr = self.cond(arg0 > arg1) elif opname == "int_ge": expr = self.cond(arg0 >= arg1) elif opname == "int_is_true": expr = self.cond(arg0 != FALSEBV) elif opname == "uint_lt": expr = self.cond(z3.ULT(arg0, arg1)) elif opname == "uint_le": expr = self.cond(z3.ULE(arg0, arg1)) elif opname == "uint_gt": expr = self.cond(z3.UGT(arg0, arg1)) elif opname == "uint_ge": expr = self.cond(z3.UGE(arg0, arg1)) elif opname == "int_is_zero": expr = self.cond(arg0 == FALSEBV) # ... rest as above

So basically for every trace operation that operates on integers I had to give a translation into Z3 formulas, which is mostly straightforward.

Guard operations get converted into a Z3 boolean by their own helper function, which looks like this:

def guard_to_condition(self, guard, state): opname = guard.getopname() if opname == "guard_true": return self.convertarg(guard, 0) == TRUEBV elif opname == "guard_false": return self.convertarg(guard, 0) == FALSEBV elif opname == "guard_value": return self.convertarg(guard, 0) == self.convertarg(guard, 1) # ... some more exist, shown below

Some operations are a bit trickier. An important example in the context of this blog post are integer operations that check for overflow. The overflow operations return a result, but also a boolean whether the operation overflowed or not.

def add_to_solver(self, ops, state): # ... more cases elif opname == "int_add_ovf": expr = arg0 + arg1 m = z3.SignExt(LONG_BIT, arg0) + z3.SignExt(LONG_BIT, arg1) state.no_ovf = m == z3.SignExt(LONG_BIT, expr) elif opname == "int_sub_ovf": expr = arg0 - arg1 m = z3.SignExt(LONG_BIT, arg0) - z3.SignExt(LONG_BIT, arg1) state.no_ovf = m == z3.SignExt(LONG_BIT, expr) elif opname == "int_mul_ovf": expr = arg0 * arg1 m = z3.SignExt(LONG_BIT, arg0) * z3.SignExt(LONG_BIT, arg1) state.no_ovf = m == z3.SignExt(LONG_BIT, expr) # ...

The boolean is computed by comparing the result of the bit vector operation with the result of converting the input bit vectors into an abstract (arbitrary precision) integer and the result back to bit vectors. Let's go through the addition case step by step, the other cases work analogously.

The addition in the first elif that computes expr is an addition on bit vectors, therefore it is performing wraparound arithmetic. z3.SignExt(LONG_BIT, arg0) sign-extends arg0 from a bit vector of LONG_BIT bits to an abstract, arbitrary precision integer. The addition in the second line is therefore an addition between abstract integers, so it will never overflow and just compute the correct result as an integer.

The condition to check for overflow is now: if the results of the two different ways to do the addition are the same, then overflow did not occur. So in order to compute state.no_ovf in the addition case the code converts the result of the bit vector wraparound addition to an abstract integer (using SignExt again), and then compares that to the integer result.

This boolean can then be checked by the guard operations guard_no_overflow and guard_overflow.

def guard_to_condition(self, guard, state): # ... more cases elif opname == "guard_no_overflow": assert state.no_ovf is not None return state.no_ovf elif opname == "guard_overflow": assert state.no_ovf is not None return z3.Not(state.no_ovf) # ... more cases Finding the Bug, Again

Let's actually make all of this more concrete by applying it to the trace of our original bug. The input trace and the incorrectly optimized trace for that look like this (differences highlighted):

# input # optimized [i0] [i0] i1 = int_add(i0, 10) i1 = int_add(i0, 10) i2 = int_lt(i1, 15) i2 = int_lt(i1, 15) guard_true(i2) guard_true(i2) i3 = int_lt(i0, 6) jump(0) guard_true(i3) jump(0)

Note that the trace represents just one of the paths through the control flow graph of the original function, which is typical for tracing JITs (the other paths could incrementally get added later).

The first guards in both these traces correspond to each other, so the first chunks to check are the first three operations (lines 1-4). Those operations don't get changed by the optimizer at all.

These two identical traces get translated to the following Z3 formulas:

i1unoptimized == input_i0 + 10 i2unoptimized == If(i1unoptimized < 15, 1, 0) i1optimized == input_i0 + 10 i2optimized == If(i1optimized < 15, 1, 0)

To check that the two corresponding guards are the same, the solver is asked to prove that (i2unoptimized == 1) == (i2optimized == 1). This is correct, because the formulas for i2unoptimized and i2optimized are completely identical.

After checking that the guards behave the same, we add the knowledge to the solver that the guards passed. So the Z3 formulas become:

i1unoptimized == input_i0 + 10 i2unoptimized == If(i1unoptimized < 15, 1, 0) i1optimized == input_i0 + 10 i2optimized == If(i1optimized < 15, 1, 0) i1optimized == 1 i2optimized == 1

Now we continue with the remaining operations of the two traces (lines 6-8).

We start by adding the int_lt operation in the unoptimized trace to the Z3 formulas:

... i3unoptimized == If(input_i0 < 6, 1, 0)

Because the second guard was optimized away, we need to ask Z3 to prove that i3unoptimized == 1 is always true, which fails and gives the following counterexample:

input_i0 = 9223372036854775800 i1unoptimized = 9223372036854775810 i2unoptimized = 0 i1optimized = 9223372036854775810 i2optimized = 1 i3unoptimized = 0

Thus demonstrating the bug. The fact that the Z3-based equivalence check also managed to find the original motivating bug without manually translating it to a formula is a good confirmation that the approach works.

Second bug

So with this code I applied the Z3-based equivalence check to all our optimizer unit tests. In addition to the bug we've been discussing the whole post, it also found another buggy test! I had found it too by hand by staring at all the tests in the process of writing all the Z3 infrastructure, but it was still a good confirmation that the process worked. This bug was in the range analysis for int_neg, integer negation. It failed to account that -MININT == MININT and therefore did a mis-optimization along the following lines:

import __pypy__ def wrong(x): a = __pypy__.intop.int_sub(0, x) if a < 0: if x > 0: return 0 return 1 return 2

Which was wrongly optimized into:

import __pypy__ def wrong(x): a = __pypy__.intop.int_sub(0, x) if a < 0: return 0 return 2

This is wrong precisely for x == MININT.

Generating Random Traces

These two bugs were the only two that the Z3 checker found for existing unit tests. To try to find some more bugs I combined PyPy's existing random trace generator with the Z3 optimization checker. The random trace generator has so far been mostly used to find bugs in the machine code backends, particularly also in the register allocator. So far we haven't used it with our optimizer, but my experiments show that we should have!

I'm going to describe a little bit how the random trace generator works. It's actually not that complicated, but there's one neat trick to it.

The basic idea is straightforward, it starts out with an empty trace with a random number of input variables. Then it adds some number of operations to the trace, either regular operations or guards. Every operation takes already existing variables as input.

The neat trick is that our random trace generator keeps a concrete random example value for every one of the input variables, and an example result for every operation. In this way, it is possible to generate guards that are consistent with the example values to ensure that running the trace to its end is possible with at least one set of values.

Here's an example random trace that is generated, together with the random example inputs and the results of every operation at the end of every line:

[i0, i1, i2, i3, i4, i5] # example values: 9, 11, -8, -95, 46, 57 i6 = int_add_ovf(i3, i0) # -86 guard_no_overflow() i7 = int_sub(i2, -35/ci) # 27 i8 = uint_ge(i3, i5) # 1 guard_true(i8) i9 = int_lt(i7, i8) # 0 i10 = int_mul_ovf(34/ci, i7) # 918 guard_no_overflow() i11 = int_and(i10, 63/ci) # 22 i12 = int_rshift(i3, i11) # -1 i13 = int_is_zero(i7) # 0 i14 = int_is_true(i13) # 0 guard_false(i13) i15 = int_lt(i8, i4) # 1 i16 = int_and(i6, i0) # 8 i17 = uint_ge(i6, -6/ci) # 0 finish()

Note how every guard generated is true for the example values.

I have been running this combination of random trace generation and Z3 checking for many nights and it has found some bugs, which I'll describe in the next section. It should probably be run for a lot longer, but still a useful exercise already.

In this mode, I'm giving every Z3 call a time limit to make sure that the random tests don't just take arbitrarily long. This means that asking Z3 to prove something can have three outcomes, either it's proved, or Z3 finds a counterexample, or Z3 times out.

Bugs Found

In addition to the two bugs I've already described, I'll briefly list the additional bugs that were found by optimizing random traces and then trying to prove the equivalence with Z3.

Most of the bugs were actually identified by optimizing random traces alone, not by the Z3 component. They manifested as assert failures in the JIT compiler.

• The JIT concluded after 12 == int_mul(x, 12) that x == 1, which is incorrect if overflow occurred (a counterexample is 0x8000000000000001).

• An amusing bug, where from 0 == int_lshift(0x1000000000000000, x) with x <= 0 <= 15, the JIT concluded that 0x1000000000000000 == 0, triggering an assert. This wrong conclusion was again caused by not taking the possibility of overflow into account.

• A corner case in an optimization for chained integer additions with a constant, where in complex enough expressions, the wrong IR API was used (which works correctly in simple cases). Again, this triggered an assert.

This shows that we should have been fuzzing our JIT optimizer already (not a surprising observation in hindsight, fuzz all the things!).

Thankfully, there was also one further bug that really failed in the Z3 verifier. It's a bug in common subexpression elimination / arithmetic simplification, which again does not take overflow correctly into account.

The buggy trace looks like this (unfortunately it's not easily possible to show this bug in Python code).

[a, b] c = int_add(a, b) r = int_sub_ovf(c, b) guard_no_ovf() finish(r)

This was optimized to:

[a, b] finish(a)

Which is incorrect, because the guard can fail given the right inputs. But the optimizer concluded that the subtraction is safe, because its the inverse of an earlier addition, not taking into account that this earlier addition can have overflowed.

Note that a related optimization is actually correct. Given this code:

[a, b] c = int_add_ovf(a, b) guard_no_ovf() r = int_sub(c, b) finish(r)

It can be optimized to:

[a, b] c = int_add_ovf(a, b) guard_no_ovf() finish(a) Future Work and Conclusion

In the current form the Z3 checker is only a start, even though it has already been concretely useful. There are various directions into which we could extend it. In addition to generate random tests completely from scratch, we could also start from the existing manually written unit-tests and randomly mutate those.

I also want to extend the Z3 checker with support more operations, heap operations in particular (but it's not quite clear to me how to model garbage collection).

I also want to try to switch the code away from the Z3 API and use the more general smtlib interface directly, in order to be able to use other SMT checkers than Z3, eg CVC4.

But all in all this was a fun and not too hard way to find a bunch of bugs in our optimizer! And the infrastructure is now in place, which means that we run some random test cases every time we execute our tests. This is going to be particularly useful when we do further work on the integer reasoning of the JIT (like Nico is doing, for example). As of time of writing of this post, all the bugs mentioned have been fixed and the Z3 code has landed on the default branch and runs as part of PyPy's CI infrastructure.

Acknowledgements

Thanks to Saam Barati, Max Bernstein, Joshua Schmidt and Martin Berger, for great feedback on drafts of this post!

Hang out with Python devs long enough, and you&aposll hear all about Tim Peter&aposs Zen Of Python.

The Zen, which you can conveniently read by executing import this in a Python REPL, presents 19 of the 20 guiding principles behind Python&aposs design. Recently I&aposve come to appreciate one aphorism more than the others: "Explicit is better than implicit."

The most common interpretation I&aposve seen — so common that it currently inhabits Google&aposs featured snippet when searching for the phrase — is that verbose code is better than terse code because verbosity is, apparently, the key to readability... or something.

Sure, using better variable names and replacing magic numbers with named constants (or, in Python&aposs case, "constants") are all great things. But when was the last time you hunted down implicit inputs in your code and made them explicit?

✉️This article was originally published in my Curious About Code newsletter. Never miss an issue. Subscribe here →How To Recognize Implicit Inputs and Outputs

How many inputs and outputs does the following function have?

def find_big_numbers(): with open("numbers.txt", "r") as f: for line in f: if line.strip().isnumeric(): number = float(line) if number > 100: print(number)

find_big_numbers() has no parameters and always returns None. If you couldn&apost see the function body and couldn&apost access the standard output stream, would you even believe that this function does anything?

And yet, find_big_numbers() has two inputs and another output besides None:

• numbers.txt is an implicit input. The function won&apost work without it, but it is impossible to know that the file is required without reading the function body.
• The magic number 100 on line 6 is an implicit input. You can&apost define a "big number" without it, but there is no way to know that threshold without reading the function body.
• Values may or may not print to stdout, depending on the contents of numbers.txt. This is an implicit output because the function does not return those values.

Implicit outputs are often called side effects.

Try It Yourself

Identify all of the inputs and outputs of the is_adult function in this code snippet:

from datetime import date birthdays = { "miles": date(2000, 1, 14), "antoine": date(1987, 3, 25), "julia": date(2009, 11, 2), } children = set() adults = set() def is_adult(name): birthdate = birthdays.get(name) if birthdate: today = date.today() days_old = (today - birthdate).days years_old = days_old // 365 if years_old >= 18: print(f"{name} is an adult") adults.add(name) return True else: print(f"{name} is not an adult") children.add(name) return False&#x1F914;There is more wrong with this code than just implicit inputs and outputs. What else would you do to clean this up?Why You Should Avoid Implicit Input And Output

One good reason is their penchant for violating the principle of least surprise.

Of course, not all implicit inputs and outputs are bad. A method like .write(), which Python file objects use to write data to a file, has an implicit output: the file. There&aposs no way to eliminate it. But it isn&apost surprising. Writing to a file is the whole point.

On the other hand, a function like is_adult() from the previous code snippet does lots of surprising things. Less extreme examples abound.

&#x1F4A1;It&aposs a good exercise to read through some of your favorite library&aposs code on GitHub and see if you can spot the implicit and explicit outputs. Ask yourself: do any of them surprise you?

Avoiding implicit input and output also improves your code&aposs testability and re-usability. To see how, let&aposs refactor the find_big_numbers() function from earlier.

How To Remove Implicit Input And Output

Here&aposs find_big_numbers() again so you don&apost have to scroll up:

def find_big_numbers(): with open("numbers.txt", "r") as f: for line in f: if line.strip().isnumeric(): number = float(line) if number > 100: print(number)

Earlier, we identified two implicit inputs, the numbers.txt file and the number 100, and one implicit output, the values printed to stdout. Let&aposs work on the inputs first.

You can move the file name and the threshold value to parameters of the function:

def find_big_numbers(path, threshold=100): with open(path, "r") as f: for line in f: if line.strip().isnumeric(): number = float(line) if number > threshold: print(number)

This has already dramatically improved the testability and re-usability. If you want to try it on a different file, pass the path as an argument. (As a bonus, the file can now be anywhere on your computer.) You can also change the threshold for "big numbers," if needed.

But the output is hard to test.

If you want to know that the function produced the correct values, you need to intercept stdout. It&aposs possible. But why not just return a list of all of the values:

def find_big_numbers(path, threshold=100): big_numbers = [] with open(path, "r") as f: for line in f: if line.strip().isnumeric(): number = float(line) if number > threshold: big_numbers.append(number) return big_numbers

Now find_big_numbers() has an explicit return statement that returns a list of big numbers found in the file.

&#x1F914;Aside from removing implicit input and output, there is still a lot that could be done to improve find_big_numbers(). How would you go about cleaning it up?

You can test find_big_numbers() by calling it with the path of a file whose contents are known and comparing the list returned to the list of correct values:

# test_nums.txt looks like: # 29 # 375 # 84 >>> expected_nums = [375.0] >>> actual_nums = find_big_numbers("test_nums.txt") >>> assert(actual_nums == expected_nums)

find_big_numbers() is more reusable now, too. You aren&apost limited to printing the numbers to stdout. You can send those big numbers wherever you want.

&#x1F9E0;Let&aposs recap:

Implicit inputs are data used by a function or program that aren&apost explicitly passed as arguments. You can eliminate implicit inputs by refactoring them into parameters.

Implicit outputs are data sent somewhere external to the function or program that aren&apost explicitly returned. You can remove explicit outputs by replacing them with suitable return values.

Not all implicit input and output can be avoided, such as functions whose purpose is to read or write data from files and databases or to send an email. Still, eliminating as many implicit inputs and outputs as possible improves the testability and re-usability of your code.&#x1F914;So here&aposs the question: Did we remove all of the implicit inputs and outputs from find_big_numbers()?

Curious about what happened to the 20th line in the Zen of Python? There are all sorts of theories floating around the internet. This one strikes me as reasonably likely.

Read more about implicit inputs and outputs in Eric Normand&aposs excellent book Grokking Simplicity. Get instant access from Manning* or order it on Amazon*.

*Affiliate link. See my affiliate disclosure for more information.

Want more like this?

One email, every Saturday, with one actionable tip.
Always less than 5 minutes of your time.

Subscribe now Processing your application Great! Check your inbox and confirm your subscription There was an error sending the email

TL;DR; Still no sound, but I learned a couple of new things…

So, I finally got around to upgrading my laptop. I decided to go for my fourth XPS13, and this time I opted for a maxed out XPS13 Plus. A really nice machine. However, the driver stack isn’t quite there yet. Yes, I should have read up more before buying, but I didn’t and I know it will be sorted out over time.

As a vim user, the touch Esc key will be a challenge. Perhaps this is where I learn to bind to capslock, but I’ve not come to that point yet.

So, after installing using the netinst image with non-free drivers (and my phone over USB tether for networking since the wifi still didn’t work), I had to move to testing for anything to work. Then I installed firmware-iwlwifi, iwlwifi and firmware-sof-signed from non-free. This got me into a graphical desktop and most things work (I could configure the touch pad for tap-to-click, and so on). I run a KDE desktop, so I installed some Plymouth stuff, breeze for SDDM and such, but that shouldn’t affect the issues described here.

My current issues are the sound, the webcam and hibernation. The two latter items aren’t huge problems, but I do need sound. Both the webcam and sound issues are known. Hibernation is mostly about getting around to configuring it with the encrypted disk setup.

So, let’s start by diving into the audio issue. The early boot process looks like this:

[ 20.595666] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if info 0x040100 [ 20.595750] sof-audio-pci-intel-tgl 0000:00:1f.3: SoundWire enabled on CannonLake+ platform, using SOF driver [ 20.595778] sof-audio-pci-intel-tgl 0000:00:1f.3: enabling device (0000 -> 0002) [ 20.596001] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100 [ 20.596073] sof-audio-pci-intel-tgl 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915]) [ 20.602984] sof-audio-pci-intel-tgl 0000:00:1f.3: use msi interrupt mode [ 20.651283] sof-audio-pci-intel-tgl 0000:00:1f.3: hda codecs found, mask 4 [ 20.652476] sof-audio-pci-intel-tgl 0000:00:1f.3: firmware: direct-loading firmware intel/sof/sof-adl.ri [ 20.652482] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864 [ 20.652483] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:0 [ 20.652488] sof-audio-pci-intel-tgl 0000:00:1f.3: unknown sof_ext_man header type 3 size 0x30 [ 20.747032] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864 [ 20.747036] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:0 [ 25.758149] sof_sdw sof_sdw: snd_soc_register_card failed -517 [ 25.768995] sof_sdw sof_sdw: snd_soc_register_card failed -517 [ 25.799027] sof_sdw sof_sdw: snd_soc_register_card failed -517

If I later force a reload of the module, it all works:

sudo modprobe -r snd-sof-pci-intel-tgl; sudo modprobe snd-sof-pci-intel-tgl

Gives:

[ 169.407671] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if info 0x040100 [ 169.407784] sof-audio-pci-intel-tgl 0000:00:1f.3: SoundWire enabled on CannonLake+ platform, using SOF driver [ 169.408027] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100 [ 169.408133] sof-audio-pci-intel-tgl 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915]) [ 169.414240] sof-audio-pci-intel-tgl 0000:00:1f.3: use msi interrupt mode [ 169.428614] sof-audio-pci-intel-tgl 0000:00:1f.3: hda codecs found, mask 4 [ 169.428802] sof-audio-pci-intel-tgl 0000:00:1f.3: firmware: direct-loading firmware intel/sof/sof-adl.ri [ 169.428809] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864 [ 169.428810] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:0 [ 169.428814] sof-audio-pci-intel-tgl 0000:00:1f.3: unknown sof_ext_man header type 3 size 0x30 [ 169.547087] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864 [ 169.547115] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:0 [ 169.563645] sof-audio-pci-intel-tgl 0000:00:1f.3: firmware: direct-loading firmware intel/sof-tplg/sof-adl-rt1316-l12-rt714-l0.tplg [ 169.563665] sof-audio-pci-intel-tgl 0000:00:1f.3: Topology: ABI 3:22:1 Kernel ABI 3:23:0 [ 169.564019] sof_sdw sof_sdw: ASoC: Parent card not yet available, widget card binding deferred [ 169.605390] sof_sdw sof_sdw: hda_dsp_hdmi_build_controls: no PCM in topology for HDMI converter 3 [ 169.627693] input: sof-soundwire HDMI/DP,pcm=5 as /devices/pci0000:00/0000:00:1f.3/sof_sdw/sound/card0/input26 [ 169.627738] input: sof-soundwire HDMI/DP,pcm=6 as /devices/pci0000:00/0000:00:1f.3/sof_sdw/sound/card0/input27 [ 169.631375] input: sof-soundwire HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:1f.3/sof_sdw/sound/card0/input28

So, something happens after sof-adl.ri is loaded. This causes the driver to just stop before loading the sof-adl-rtl316-l12-rt714-l0.tplg firmware, causing the snd_soc_register_card to fail. Let’t have a look at the reasons.

First up, the excellent Arch Linux Wiki says to include the two firmware files and a bunch of modules in the initramfs. However, Arch and Debian uses different tools to build the initramfs, so let’s confirm the issue first:

lsinitramfs -l /boot/initrd.img-6.0.0-5-amd64 | grep 'intel/sof'

This call results in a list of nothing. Just to confirm, grepping for firmware or intel returns long lists of files. So, the firmware is not in the early initramfs image. How do I get the firmware files into the initramfs? Apparently I have to write what is known as an initramfs-tools hook script.

There lives a bunch of them over at /usr/share/initramfs-tools/hooks, so I started a very brute force one called intel-sof-firmware and tried to learn from the surrounding scripts and the manpage linked above. The result can be found in a gist here.

Notice that this is happy code. There is way too few sanity checks in there to make this useful to the general public. Your milage may vary.

So, I added the modules to /etc/initramfs/modules, and then updated the early initramfs images with this command:

sudo update-initramfs -k all -u

And then verified that the files made it to the image (I also had a look at the file listing in general to ensure that the image is ok):

lsinitramfs -l /boot/initrd.img-6.0.0-5-amd64 | grep 'intel/sof' drwxr-xr-x 2 root root 0 Dec 11 15:51 usr/lib/firmware/intel/sof drwxr-xr-x 2 root root 0 Dec 11 15:51 usr/lib/firmware/intel/sof-tplg -rw-r--r-- 1 root root 28907 Dec 11 15:51 usr/lib/firmware/intel/sof-tplg/sof-adl-rt1316-l12-rt714-l0.tplg -rw-r--r-- 1 root root 525056 Dec 11 15:51 usr/lib/firmware/intel/sof/sof-adl.ri

It all seems to work, so let’s reboot and see what comes out the other side. Notice – this can brick your computer if you make a mistake. And if something to do with a computer can brick your computer, you know that it will brick your computer. Don’t say I did not tell you.

Side note: I run an encrypted lvm setup, and if you try to fix this problem by removing all modules (go from many to netbook in the initramfs.conf) you will enjoy re-installing your machine. I’m sure you can unbrick it by booting from a USB stick and fixing stuff, but since I’ve not really installed anything, I don’t really care.

Guess what – after a couple of hours digging at this – I still have no sound…

sudo dmesg | grep sof

Results in this (yay, a new error code – that must mean that I’m doing something):

[ 1.760559] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if info 0x040100 [ 1.760629] sof-audio-pci-intel-tgl 0000:00:1f.3: SoundWire enabled on CannonLake+ platform, using SOF driver [ 1.760643] sof-audio-pci-intel-tgl 0000:00:1f.3: enabling device (0000 -> 0002) [ 1.760785] sof-audio-pci-intel-tgl 0000:00:1f.3: DSP detected with PCI class/subclass/prog-if 0x040100 [ 3.491343] sof-audio-pci-intel-tgl 0000:00:1f.3: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915]) [ 3.569000] sof-audio-pci-intel-tgl 0000:00:1f.3: use msi interrupt mode [ 3.585475] sof-audio-pci-intel-tgl 0000:00:1f.3: codec #2 probe error, ret: -2 [ 3.585859] sof-audio-pci-intel-tgl 0000:00:1f.3: no hda codecs found! [ 3.585990] sof-audio-pci-intel-tgl 0000:00:1f.3: firmware: direct-loading firmware intel/sof/sof-adl.ri [ 3.585996] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864 [ 3.585998] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:0 [ 3.586007] sof-audio-pci-intel-tgl 0000:00:1f.3: unknown sof_ext_man header type 3 size 0x30 [ 3.702740] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware info: version 2:2:0-57864 [ 3.702756] sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware: ABI 3:22:1 Kernel ABI 3:23:0 [ 3.709727] sof-audio-pci-intel-tgl 0000:00:1f.3: firmware: direct-loading firmware intel/sof-tplg/sof-adl-rt1316-l12-rt714-l0.tplg [ 3.709742] sof-audio-pci-intel-tgl 0000:00:1f.3: Topology: ABI 3:22:1 Kernel ABI 3:23:0 [ 3.709825] sof-audio-pci-intel-tgl 0000:00:1f.3: error: can't connect DAI HDA0.OUT stream iDisp1 [ 3.709920] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to add widget id 0 type 27 name : HDA0.OUT stream iDisp1 [ 3.710026] sof_sdw sof_sdw: ASoC: failed to load widget HDA0.OUT [ 3.710086] sof_sdw sof_sdw: ASoC: topology: could not load header: -22 [ 3.710162] sof-audio-pci-intel-tgl 0000:00:1f.3: error: tplg component load failed -22 [ 3.710247] sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to load DSP topology -22 [ 3.710326] sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_component_probe on 0000:00:1f.3: -22 [ 3.710447] sof_sdw sof_sdw: ASoC: failed to instantiate card -22 [ 3.710692] sof_sdw sof_sdw: snd_soc_register_card failed -22 [ 3.710758] sof_sdw: probe of sof_sdw failed with error -22

I suspect I’m missing another driver in the initramfs. Still, the modprobe trick shown above still fixes sound, so I guess I’ll leave it for today…

Earlier this year, we released Akvorado, a flow collector, enricher, and visualizer. It receives network flows from your routers using either NetFlow v9, IPFIX, or sFlow. Several pieces of information are added, like GeoIP and interface names. The flows are exported to Apache Kafka, a distributed queue, then stored inside ClickHouse, a column-oriented database. A web frontend is provided to run queries. A live version is available for you to play.

Akvorado’s web frontend

Several alternatives exist:

• Kentik, a popular cloud-based solution,
• ElastiFlow, a self-hosted proprietary solution, or
• your own assembly of open source solutions by picking a flow collector (pmacct, GoFlow2, or vFlow), a distributed queue1 (Apache Kafka or RabbitMQ), a database (ClickHouse, Elasticsearch, or Apache Pinot), and a web frontend (Grafana, Kibana, or Apache Superset).

Akvorado differentiates itself from these solutions because:

• it is open source (licensed under the AGPLv3 license), and
• it bundles flow collection, storage, and a web interface into a single “product.”

The proposed deployment solution relies on Docker Compose to set up Akvorado, Zookeeper, Kafka, and ClickHouse. I hope it should be enough for anyone to get started quickly. Akvorado is performant enough to handle 100 000 flows per second with 64 GB of RAM and 24 vCPU. With 2 TB of disk, you should expect to keep data for a few years.

I spent some time writing a fairly complete documentation. It seems redundant to repeat its content in this blog post. There is also a section about its internal design if you are interested in how it is built. I also did a FRnOG presentation earlier this year, and a ClickHouse meetup presentation, which focuses more on how ClickHouse is used. I plan to write more detailed articles on specific aspects of Akvorado. Stay tuned! 📻

1. While the collector could write directly to the database, the queue buffers flows if the database is unavailable. It also enables you to process flows with another piece of software (like an anti-DDoS system). ↩︎

In previous articles in this series, you read about positional and keyword arguments, making arguments optional by adding a default value, and including any number of positional and keyword arguments using *args and **kwargs. In this article, it’s the turn of another flavour of argument. You’ll look at parameters that can only accept positional arguments and those that can only be keyword arguments. Let’s see how to create positional-only and keyword-only arguments in Python.

Overview Of The Intermediate Python Functions Series

Here’s an overview of the seven articles in this series:

1. Introduction to the series: Do you know all your functions terminology well?
2. Choosing whether to use positional or keyword arguments when calling a function
3. Using optional arguments by including default values when defining a function
4. Using any number of optional positional and keyword arguments: *args and **kwargs
5. [This article] Using positional-only arguments and keyword-only arguments: the “rogue” forward slash / or asterisk * in function signatures
6. Type hinting in functions
7. Best practices when defining and using functions

Positional-Only Arguments in Python

When looking at documentation, you may have seen a “rogue” forward slash / in a function signature such as this:

def greet_person(person, /, repeat):

Let’s explore this by starting with the following function, which is one you’ve already seen in earlier articles in this series. Note how the string is being multiplied by the integer repeat:

def greet_person(person, repeat): print(f"Hello {person}, how are you doing today?\n" * repeat) # 1. greet_person("Zahra", 2) # 2. greet_person("Zahra", repeat=2) # 3. greet_person(person="Zahra", repeat=2)

The function has two parameters:

• person
• repeat

All three function calls work:

Hello Zahra, how are you doing today? Hello Zahra, how are you doing today? Hello Zahra, how are you doing today? Hello Zahra, how are you doing today? Hello Zahra, how are you doing today? Hello Zahra, how are you doing today?

You can choose to use positional arguments as in the first function call in the code above. The arguments are matched to parameters based on their position in the function call.

You can choose to use named arguments (also called keyword arguments) as in the third function call above. Both arguments are named using the parameter name as a keyword when calling the function.

You can even use a mixture of positional and keyword arguments as long as the positional arguments come first. This is the case in the second function call above.

You can refresh your memory about positional and keyword arguments in the first article in this series.

Adding the “rogue” forward slash to the function signature

Let’s make one small change to the function definition by adding a lone forward slash among the parameters:

def greet_person(person, /, repeat): print(f"Hello {person}, how are you doing today?\n" * repeat)

You’ve added the forward slash between the two parameters. Now, you can try calling the function in the same three ways as in the previous section.

#1.

This is the scenario in which both arguments are positional:

def greet_person(person, /, repeat): print(f"Hello {person}, how are you doing today?\n" * repeat) # 1. greet_person("Zahra", 2)

This works fine. Here’s the output:

Hello Zahra, how are you doing today? Hello Zahra, how are you doing today?

In this scenario, the forward slash didn’t affect the output. We’ll get back to this first example shortly.

#2.

In this example, you call the function with one positional argument and one keyword argument. The first argument is positional, and the second one is named (keyword):

def greet_person(person, /, repeat): print(f"Hello {person}, how are you doing today?\n" * repeat) # 2. greet_person("Zahra", repeat=2)

Again, there are no issues when we run this code:

Hello Zahra, how are you doing today? Hello Zahra, how are you doing today?

So far, so good. But let’s look at the third example now.

#3.

In this call, both arguments are named (keyword) arguments:

def greet_person(person, /, repeat): print(f"Hello {person}, how are you doing today?\n" * repeat) # 3. greet_person(person="Zahra", repeat=2)

When you run this code, it raises an error:

Traceback (most recent call last): File "...", line 5, in <module> greet_person(person="Zahra", repeat=2) TypeError: greet_person() got some positional-only arguments passed as keyword arguments: 'person'

Let’s read the error message in the last line:

greet_person() got some positional-only arguments passed as keyword arguments: 'person'

The error message mentions a positional-only argument. This is because the parameter person can only accept positional arguments, not keyword arguments.

The forward slash / in the function signature is the point where the change occurs. Any argument assigned to parameters before the forward slash / can only be passed as positional arguments.

The parameters after the forward slash can accept both positional and keyword arguments. This is why the first and second function calls worked fine.

Moving the forward slash

Let’s move the forward slash / to the end of the parameters in the function call:

def greet_person(person, repeat, /): print(f"Hello {person}, how are you doing today?\n" * repeat)

And let’s look at the same function calls you used earlier.

#1.

Both arguments are positional arguments in this example:

def greet_person(person, repeat, /): print(f"Hello {person}, how are you doing today?\n" * repeat) # 1. greet_person("Zahra", 2)

This code works fine:

Hello Zahra, how are you doing today? Hello Zahra, how are you doing today?

Let’s recall the rule about the forward slash / in the function signature. All parameters before the forward slash / must be assigned to positional arguments and cannot be keyword arguments. These are positional-only arguments. In this example, both arguments are positional. Therefore, this works.

#2.

In the second example, the second argument is a keyword argument:

def greet_person(person, repeat, /): print(f"Hello {person}, how are you doing today?\n" * repeat) # 2. greet_person("Zahra", repeat=2)

This raises an error since repeat cannot be assigned a keyword argument:

Traceback (most recent call last): File "...", line 12, in <module> greet_person("Zahra", repeat=2) TypeError: greet_person() got some positional-only arguments passed as keyword arguments: 'repeat' #3.

In the final example, both arguments are keyword arguments:

def greet_person(person, repeat, /): print(f"Hello {person}, how are you doing today?\n" * repeat) # 3. greet_person(person="Zahra", repeat=2)

This also raises an error:

Traceback (most recent call last): File "...", line 21, in <module> greet_person(person="Zahra", repeat=2) TypeError: greet_person() got some positional-only arguments passed as keyword arguments: 'person, repeat'

Note that the error message now lists both person and repeat as parameters that have been assigned keyword arguments. In the second example, only repeat was listed in the error message.

Summary for positional-only arguments

When you define a function, you can force the user to use positional-only arguments for some of the arguments:

• You can add a forward slash / as one of the parameters in the function definition
• All arguments assigned to parameters before the / must be positional arguments

It’s up to you as a programmer to decide when to use positional-only arguments when defining functions. You may feel that restricting the user to positional-only arguments makes the function call more readable, neater, or less likely to lead to bugs. You’ll look at another example later in this article which illustrates such a case.

Keyword-Only Arguments in Python

Another “rogue” symbol you may see when reading documentation or code is the asterisk * as in this example:

def greet(host, *, guest):

Let’s work our way to this version of the function by starting with a simpler one which doesn’t have the asterisk:

def greet(host, guest): print(f"{host} says hello to {guest}") # 1. greet("James", "Claire") # 2. greet(host="James", guest="Claire") # 3. greet(guest="Claire", host="James")

The function has two parameters, and you can use either positional arguments or keyword arguments when calling this function. When you run this code, you’ll see that all three function calls work:

James says hello to Claire James says hello to Claire James says hello to Claire

You can add an asterisk to the function definition. You can start by adding it at the beginning:

def greet(*, host, guest): print(f"{host} says hello to {guest}")

Let’s look at the three function calls in the example above. You’ll explore them in reverse order starting from the third one.

#3.

In this example, both arguments are keyword arguments:

def greet(*, host, guest): print(f"{host} says hello to {guest}") # 3. greet(guest="Claire", host="James")

Note that the arguments in the function call are not in the same order as the parameters in the function definition. You’ve seen earlier in this series that you can do this when using keyword arguments. The order of the arguments is no longer needed to assign arguments to parameters. The output from this code is:

James says hello to Claire #2.

The second function call is very similar to the third. Both arguments are keyword (named) arguments, but the order is different:

def greet(*, host, guest): print(f"{host} says hello to {guest}") # 2. greet(host="James", guest="Claire")

The output is the same as in #3:

James says hello to Claire #1.

In the first function call, both arguments are now positional arguments:

def greet(*, host, guest): print(f"{host} says hello to {guest}") # 1. greet("James", "Claire")

In this case, we encounter a problem:

Traceback (most recent call last): File "...", line 5, in <module> greet("James", "Claire") TypeError: greet() takes 0 positional arguments but 2 were given

The error message says:

greet() takes 0 positional arguments but 2 were given

Let’s dive a bit deeper into what this error message is saying. This function cannot take any positional arguments. Both arguments must be keyword (or named) arguments. The asterisk * in the function definition forces this behaviour. Any argument assigned to parameters after the asterisk * can only be passed as keyword (named) arguments.

You can see why forcing keyword-only arguments can be beneficial in this case. The function takes two people’s names as arguments, but it’s important to distinguish between the host and the guest. Forcing keyword-only arguments can minimise bugs as the user doesn’t need to remember or check every time who comes first in the function call, the host or the guest.

Moving the asterisk to a different position

Let’s reinforce what’s happening by making one more change and placing the asterisk * in between the two parameters:

def greet(host, *, guest): print(f"{host} says hello to {guest}")

You can try the three function calls again. Let’s lump #2. and #3. together.

#2. and #3.

Both arguments are keyword arguments in these two calls:

def greet(host, *, guest): print(f"{host} says hello to {guest}") # 2. greet(host="James", guest="Claire") # 3. greet(guest="Claire", host="James")

The output is the same from both calls:

James says hello to Claire James says hello to Claire #1.

In the first function call, both arguments are positional:

def greet(host, *, guest): print(f"{host} says hello to {guest}") # 1. greet("James", "Claire")

This raises an error:

Traceback (most recent call last): File "...", line 5, in <module> greet("James", "Claire") TypeError: greet() takes 1 positional argument but 2 were given

The error message states:

greet() takes 1 positional argument but 2 were given

The asterisk * forces the parameters which come after it to be keyword-only arguments. Therefore, guest must be assigned a keyword-only argument. The parameter host can be a positional argument.

The problem in this call is with the second argument "Claire", and not with the first one "James". You can confirm this with a fourth example.

#4.

In this call, the first argument is positional and the second is a keyword argument:

def greet(host, *, guest): print(f"{host} says hello to {guest}") # 4. greet("James", guest="Claire")

The output is:

James says hello to Claire

You can pass either a positional or a keyword argument to host, which comes before the asterisk *. However, you can only pass a keyword (named) argument to guest, which comes after the asterisk *.

Summary for keyword-only arguments

When you define a function, you can force the user to use keyword-only arguments for some of the arguments:

• You can add an asterisk * as one of the parameters in the function definition
• All arguments assigned to parameters after the * must be keyword (named) arguments

Positional-Only And Keyword-Only Arguments in Python

Let’s finish this article with another example which combines both positional-only and keyword-only arguments in the same function.

Look at the function definition and the four function calls below. Do you think any of them will raise an error?

def greet(greeting, /, repeat, *, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 1. greet("Hello!", 3, "James", "Claire") # 2. greet("Hello!", 3, host="James", guest="Claire") # 3. greet("Hello!", repeat=3, host="James", guest="Claire") # 4. greet(greeting="Hello", repeat=3, host="James", guest="Claire")

Note that the function definition has both a / and an *. Let’s explore all four function calls.

#1.

All four arguments are positional arguments in this function call:

def greet(greeting, /, repeat, *, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 1. greet("Hello!", 3, "James", "Claire")

This raises an error:

Traceback (most recent call last): File "...", line 6, in <module> greet("Hello!", 3, "James", "Claire") TypeError: greet() takes 2 positional arguments but 4 were given

The part of the function signature which leads to this error is the asterisk *. Any parameter after the asterisk * must be matched to a keyword-only argument. Therefore the arguments "James" and "Claire" lead to this error. This function can take at most two positional arguments, as mentioned in the error message:

greet() takes 2 positional arguments but 4 were given

You can confirm that it’s the asterisk * which causes the problem by removing it and calling the same function:

# Version without an * def greet(greeting, /, repeat, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 1. (no asterisk in signature) greet("Hello!", 3, "James", "Claire")

This version which doesn’t have the asterisk * works:

James says 'Hello!' to Claire James says 'Hello!' to Claire James says 'Hello!' to Claire #2.

You’ll go back to the function definition with both a / and an * and look at the second call:

def greet(greeting, /, repeat, *, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 2. greet("Hello!", 3, host="James", guest="Claire")

The first two arguments are positional:

• "Hello" is assigned to greeting
• 3 is assigned to repeat

The third and fourth arguments are keyword (named) arguments:

• host="James"
• guest="Claire"

The output from this code is:

James says 'Hello!' to Claire James says 'Hello!' to Claire James says 'Hello!' to Claire

You’ve seen earlier that only the last two arguments are forced to be keyword-only arguments by the asterisk * which precedes them. Therefore, this function call works perfectly fine.

#3.

Let’s look at the third call:

def greet(greeting, /, repeat, *, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 3. greet("Hello!", repeat=3, host="James", guest="Claire")

Only the first argument is positional:

• "Hello" is assigned to greeting

The rest are keyword (named) arguments:

• repeat=3
• host="James"
• guest="Claire"

This works. Here’s the output:

James says 'Hello!' to Claire James says 'Hello!' to Claire James says 'Hello!' to Claire

The last two arguments must be keyword arguments since the parameters host and guest come after the asterisk * in the function signature.

The parameter repeat is “sandwiched” between the forward slash / and the asterisk * in the function signature. This means the argument is neither keyword-only nor positional-only. It’s not keyword-only because it comes before the asterisk *. It’s not positional-only because it comes after the forward slash /. This is a “normal” parameter and the argument passed to it can be either positional or keyword.

#4.

In this final example, all arguments are keyword arguments:

def greet(greeting, /, repeat, *, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 4. greet(greeting="Hello", repeat=3, host="James", guest="Claire")

This raises an error:

Traceback (most recent call last): File "...", line 6, in <module> greet(greeting="Hello", repeat=3, host="James", guest="Claire") TypeError: greet() got some positional-only arguments passed as keyword arguments: 'greeting'

The argument must be a positional one since greeting is before the forward slash / in the function definition.

Final Words

In summary, you can define a function to have some positional-only arguments, some keyword-only arguments, and some arguments that could be either positional or keyword:

• Parameters before the forward slash /: You must use positional-only arguments
• Parameters after the asterisk * : You must use keyword-only arguments
• Parameters in between / and * : You can use either positional arguments or keyword arguments

The following pseudo-definition summarises these points:

def template(positional_only, /, positional_or_keyword, *, keyword_only):

I’ll add a bit more in an appendix to help you remember which of the / or * symbols does what.

This concludes our discussion about the various types of arguments you can have in a Python function. In the remaining articles in this series, you’ll read about type hinting and best practices when defining and using functions.

Next Article: <Link will be posted here when the next article in the series is posted>

Further Reading

• Chapter 3: Power-up Your Coding: Create Your Own Functions for an in-depth introduction to Python functions
• Chapter 6: Functions Revisited. This chapter covers topics that will be dealt with later on in the series
• The White Room: Understanding Programming. In this article, I briefly referred to parameters as boxes which store data. This is part of a broader analogy I like to use. You can read more in this chapter
• Using Python Optional Arguments When Defining Functions is an article I wrote for Real Python if you want to read ahead.

Appendix: How to Remember Which Symbol Does What

I sometimes struggle to recall which of the / or * symbols does what. To remember which is which, you can note that the asterisk * is the same symbol you use to create *args. In fact, you could replace the * with *args and the function will work in a similar manner:

def greet(greeting, /, repeat, *args, host, guest): for _ in range(repeat): print(f"{host} says '{greeting}' to {guest}") # 2. greet("Hello!", 3, host="James", guest="Claire")

The output is:

James says 'Hello!' to Claire James says 'Hello!' to Claire James says 'Hello!' to Claire

The *args parameter is “mopping up” all remaining positional arguments before the keyword arguments. Therefore, all parameters after the *args must be used with keyword arguments. Recall that keyword arguments always come after positional ones in a function call.

In this example, there are no additional positional arguments to “mop up”, so args is an empty tuple. Note that when using *args as in this example, all arguments before the *args need to be positional so the / is no longer needed in this case. Using * instead of *args gives you more flexibility as you can have arguments which are either positional or keyword preceding it.

Get the latest blog updates

No spam promise. You’ll get an email when a new blog post is published

• Twitter
• LinkedIn

The post Using positional-only and keyword-only arguments in Python [Intermediate Python Functions Series #5] appeared first on The Python Coding Book.

Review: The Fifth Elephant, by Terry Pratchett

Series: Discworld #24 Publisher: Harper Copyright: 2000 Printing: May 2014 ISBN: 0-06-228013-9 Format: Mass market Pages: 455

The Fifth Elephant is the 24th Discworld and fifth Watch novel, and largely assumes you know who the main characters are. This is not a good place to start.

The dwarves are electing a new king. The resulting political conflict is spilling over into the streets of Ankh-Morpork, but that's not the primary problem. First, the replica Scone of Stone, a dwarven artifact used to crown the Low King of the Dwarves, is stolen from the Dwarf Bread Museum. Then, Vimes is dispatched to Überwald, ostensibly to negotiate increased fat exports with the new dwarven king. And then Angua disappears, apparently headed towards her childhood home in Überwald, which immediately prompts Carrot to resign and head after her. The City Watch is left in the hands of now-promoted Captain Colon.

We see lots of Lady Sybil for the first time since Guards! Guards!, and there's a substantial secondary plot with Angua and Carrot and a tertiary plot with Colon making a complete mess of things back home, but this is mostly a Vimes novel. As usual, Vetinari is pushing him outside of his comfort zone, but he's not seriously expecting Vimes to act like an ambassador. He's expecting Vimes to act like a policeman, even though he's way outside his jurisdiction. This time, that means untangling a messy three-sided political situation involving the dwarves, the werewolves, and the vampires.

There is some Igor dialogue in this book, but thankfully Pratchett toned it down a lot and it never started to bother me.

I do enjoy Pratchett throwing Vimes and his suspicious morality at political problems and watching him go at them sideways. Vimes's definition of crimes is just broad enough to get him fully invested in a problem, but too narrow to give him much patience with the diplomatic maneuvering. It makes him an unpredictable diplomat in a clash of cultures way that's fun to read about. Cheery and Detritus are great traveling companions for this, since both of them also unsettle the dwarves in wildly different ways.

I also have to admit that Pratchett is doing more interesting things with the Angua and Carrot relationship than I had feared. In previous books, I was getting tired of their lack of communication and wasn't buying the justifications for it, but I think I finally understand why the communication barriers are there. It's not that Angua refuses to talk to Carrot (although there's still a bit of that going on). It's that Carrot's attitude towards the world is very strange, and gets stranger the closer you are to him.

Carrot has always been the character who is too earnest and straightforward and good for Ankh-Morpork and yet somehow makes it work, but Pratchett is doing something even more interesting with the concept of nobility. A sufficiently overwhelming level of heroic ethics becomes almost alien, so contrary to how people normally think that it can make conversations baffling. It's not that Carrot is perfect (sometimes he does very dumb things), it's that his natural behavior follows a set of ethics that humans like to pretend they follow but actually don't and never would entirely. His character should be a boring cliche or an over-the-top parody, and yet he isn't at all.

But Carrot's part is mostly a side plot. Even more than Jingo, The Fifth Elephant is establishing Vimes as a force to be reckoned with, even if you take him outside his familiar city. He is in so many ways the opposite of Vetinari, and yet he's a tool that Vetinari is extremely good at using. Colon of course is a total disaster as the head of the Watch, and that's mostly because Colon should never be more than a sergeant, but it's also because even when he's taking the same action as Vimes, he's not doing it for the same reasons or with the same stubborn core of basic morality and loyalty that's under Vimes's suspicious conservatism.

The characterization in the Watch novels doesn't seem that subtle or deep at first, but it accumulates over the course of the series in a way that I think is more effective than any of the other story strands. Vetinari, Vimes, and Carrot all represent "right," or at least order, in overlapping stories of right versus wrong, but they do so in radically different ways and with radically different goals. Each time one of them seems ascendant, each time one of their approaches seems more clearly correct, Pratchett throws them at a problem where a different approach is required. It's a great reading experience.

This was one of the better Discworld novels even though I found the villains to be a bit tedious and stupid. Recommended.

Followed by The Truth in publication order. The next Watch novel is Night Watch.

Rating: 8 out of 10

New article by Richard Stallman, On Privacy at School.

Good day, hackfolk. Today's article is about semi-space collectors. Many of you know what these are, but perhaps not so many have seen an annotated implementation, so let's do that.

Just to recap, the big picture here is that a semi-space collector divides a chunk of memory into two equal halves or spaces, called the fromspace and the tospace. Allocation proceeds linearly across tospace, from one end to the other. When the tospace is full, we flip the spaces: the tospace becomes the fromspace, and the fromspace becomes the tospace. The collector copies out all live data from the fromspace to the tospace (hence the names), starting from some set of root objects. Once the copy is done, allocation then proceeds in the new tospace.

In practice when you build a GC, it's parameterized in a few ways, one of them being how the user of the GC will represent objects. Let's take as an example a simple tag-in-the-first-word scheme:

struct gc_obj { union { uintptr_t tag; struct gc_obj *forwarded; // for GC }; uintptr_t payload[0]; };

We'll divide all the code in the system into GC code and user code. Users of the GC define how objects are represented. When user code wants to know what the type of an object is, it looks at the first word to check the tag. But, you see that GC has a say in what the representation of user objects needs to be: there's a forwarded member too.

static const uintptr_t NOT_FORWARDED_BIT = 1; int is_forwarded(struct gc_obj *obj) { return (obj->tag & NOT_FORWARDED_BIT) == 1; } void* forwarded_addr(struct gc_obj *obj) { return obj->forwarded; } void forward(struct gc_obj *from, struct gc_obj *to) { from->forwarded = to; }

forwarded is a forwarding pointer. When GC copies an object from fromspace to tospace, it clobbers the first word of the old copy in fromspace, writing the new address there. It's like when you move to a new flat and have your mail forwarded from your old to your new address.

There is a contract between the GC and the user in which the user agrees to always set the NOT_FORWARDED_BIT in the first word of its objects. That bit is a way for the GC to check if an object is forwarded or not: a forwarded pointer will never have its low bit set, because allocations are aligned on some power-of-two boundary, for example 8 bytes.

struct gc_heap; // To implement by the user: size_t heap_object_size(struct gc_obj *obj); size_t trace_heap_object(struct gc_obj *obj, struct gc_heap *heap, void (*visit)(struct gc_obj **field, struct gc_heap *heap)); size_t trace_roots(struct gc_heap *heap, void (*visit)(struct gc_obj **field, struct gc_heap *heap));

The contract between GC and user is in practice one of the most important details of a memory management system. As a GC author, you want to expose the absolute minimum interface, to preserve your freedom to change implementations. The GC-user interface does need to have some minimum surface area, though, for example to enable inlining of the hot path for object allocation. Also, as we see here, there are some operations needed by the GC which are usually implemented by the user: computing the size of an object, tracing its references, and tracing the root references. If this aspect of GC design interests you, I would strongly recommend having a look at MMTk, which has been fruitfully exploring this space over the last two decades.

struct gc_heap { uintptr_t hp; uintptr_t limit; uintptr_t from_space; uintptr_t to_space; size_t size; };

Now we get to the implementation of the GC. With the exception of how to inline the allocation hot-path, none of this needs to be exposed to the user. We start with a basic definition of what a semi-space heap is, above, and below we will implement collection and allocation.

static uintptr_t align(uintptr_t val, uintptr_t alignment) { return (val + alignment - 1) & ~(alignment - 1); } static uintptr_t align_size(uintptr_t size) { return align(size, sizeof(uintptr_t)); }

All allocators have some minimum alignment, which is usually a power of two at least as large as the target language's ABI alignment. Usually it's a word or two; here we just use one word (4 or 8 bytes).

struct gc_heap* make_heap(size_t size) { size = align(size, getpagesize()); struct gc_heap *heap = malloc(sizeof(struct gc_heap)); void *mem = mmap(NULL, size, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0); heap->to_space = heap->hp = (uintptr_t) mem; heap->from_space = heap->limit = space->hp + size / 2; heap->size = size; return heap; }

Making a heap is just requesting a bunch of memory and dividing it in two. How you get that space differs depending on your platform; here we use mmap and also the platform malloc for the struct gc_heap metadata. Of course you will want to check that both the mmap and the malloc succeed :)

struct gc_obj* copy(struct gc_heap *heap, struct gc_obj *obj) { size_t size = heap_object_size(obj); struct gc_obj *new_obj = (struct gc_obj*)heap->hp; memcpy(new_obj, obj, size); forward(obj, new_obj); heap->hp += align_size(size); return new_obj; } void flip(struct gc_heap *heap) { heap->hp = heap->from_space; heap->from_space = heap->to_space; heap->to_space = heap->hp; heap->limit = heap->hp + heap->size / 2; } void visit_field(struct gc_obj **field, struct gc_heap *heap) { struct gc_obj *from = *field; struct gc_obj *to = is_forwarded(from) ? forwarded(from) : copy(heap, from); *field = to; } void collect(struct gc_heap *heap) { flip(heap); uintptr_t scan = heap->hp; trace_roots(heap, visit_field); while(scan < heap->hp) { struct gc_obj *obj = scan; scan += align_size(trace_heap_object(obj, heap, visit_field)); } }

Here we have the actual semi-space collection algorithm! It's a tiny bit of code about which people have written reams of prose, and to be fair there are many things to say—too many for here.

Personally I think the most interesting aspect of a semi-space collector is the so-called "Cheney scanning algorithm": when we see an object that's not yet traced, in visit_field, we copy() it to tospace, but don't actually look at its fields. Instead collect keeps track of the partition of tospace that contains copied objects which have not yet been traced, which are those in [scan, heap->hp). The Cheney scan sweeps through this space, advancing scan, possibly copying more objects and extending heap->hp, until such a time as the needs-tracing partition is empty. It's quite a neat solution that requires no additional memory.

inline struct gc_obj* allocate(struct gc_heap *heap, size_t size) { retry: uintptr_t addr = heap->hp; uintptr_t new_hp = align_size(addr + size); if (heap->limit < new_hp) { collect(heap); if (heap->limit - heap->hp < size) { fprintf(stderr, "out of memory\n"); abort(); } goto retry; } heap->hp = new_hp; return (struct gc_obj*)addr; }

Finally, we have the allocator: the reason we have the GC in the first place. The fast path just returns heap->hp, and arranges for the next allocation to return heap->hp + size. The slow path calls collect() and then retries.

Welp, that's a semi-space collector. Until next time for some notes on ephemerons again. Until then, have a garbage holiday season!

My NovaCustom NV41PZ laptop arrived a couple of days ago, and today I had some time to install it. You may want to read about my purchasing decision process first. I expected a rough ride to get it to work, given the number of people claiming that modern laptops can’t run fully free operating systems. I first tried the Trisquel 10 live DVD and it booted fine including network, but the mouse trackpad did not work. Before investigating it, I noticed a forum thread about Trisquel 11 beta3 images, and being based on Ubuntu 22.04 LTS and has Linux-libre 5.15 it seemed better to start with more modern software. Everything installed fine, and I didn’t need to adjust anything except to manually install GNOME since I prefer that over MATE. I’ve been running it for a couple of hours now, and here is a brief summary of the hardware components that works.

CPUAlder Lake Intel i7-1260PMemory2x32GB Kingston DDR4 SODIMM 3200MHzStorageSamsung 980 Pro 2TB NVMEBIOSDasharo CorebootGraphicsIntel XeScreen (internal)14″ 1920×1080Screen (HDMI)Connected to Dell 27″ 2560×1440Screen (USB-C)Connected to Dell 27″ 2560×1440 via Wavlink port extenderWebcamBuiltin 1MP CameraMicrophoneIntel Alder LakeKeyboardISO layout, all function keys workingMouseTrackpad, tap clicking and gesturesEthernet RJ45Realtek RTL8111/8168/8411 with r8169 driverMemory cardO2 Micro comes up as /dev/mmcblk0Docking stationWavlink 4xUSB, 2xHDMI, DP, RJ45, …ConnectivityUSB-A, USB-CAudioIntel Alder LakeHardware components and status

So what’s not working? Unfortunately, NovaCustom does not offer any WiFi or Bluetooth module that is compatible with Trisquel, so the AX211 (1675x) Wifi/Bluetooth card in it is just dead weight. I imagine it would be possible to get the card to work if non-free firmware is loaded. I don’t need Bluetooth right now, and use the Technoetic N-150 USB WiFi dongle when I’m not connected to wired network.

Compared against my X201, the following factors have improved.

• Faster – CPU benchmark suggests it is 8 times faster than my old i7-620M. While it feels snappier it is not a huge difference. While NVMe should improve SSD performance, benchmark wise the NVMe 980Pro only seems around 2-3 faster than the SATA-based 860 Evo. Going from 6GB to 64GB is 10 times more memory, which is useful for disk caching.
• BIOS is free software.
• EC firmware is free.
• Operating system follows the FSDG.

I’m still unhappy about the following properties with both the NV41PZ and the X201.

• CPU microcode is not available under free license.
• Intel Mangement Engine is still present in the CPU.
• No builtin WiFi/Bluetooth that works with free software.
• Some other secondary processors (e.g., disk or screen) may be running non-free software but at least none requires non-free firmware.

Hopefully my next laptop will have improved on this further. I hope to be able to resolve the WiFi part by replacing the WiFi module, there appears to be options available but I have not tested them on this laptop yet. Does anyone know of a combined WiFi and Bluetooth M.2 module that would work on Trisquel?

While I haven’t put the laptop to heavy testing yet, everything that I would expect a laptop to be able to do seems to work fine. Including writing this blog post!

(quoted from my other blog at since a new OS might be interesting for many and this is published in separate planets)

ALP - The Adaptable Linux Platform – is a new operating system from SUSE to run containerized and virtualized workloads. It is in early prototype phase, but the development is done completely openly so it’s easy to jump in to try it.

For this trying out, I used the latest encrypted build – as of the writing, 22.1 – from ALP images. I imported it in virt-manager as a Generic Linux 2022 image, using UEFI instead of BIOS, added a TPM device (which I’m interested in otherwise) and referring to an Ignition JSON file in the XML config in virt-manager.

The Ignition part is pretty much fully thanks to Paolo Stivanin who studied the secrets of it before me. But here it goes - and this is required for password login in Cockpit to work in addition to SSH key based login to the VM from host - first, create config.ign file:

{
"ignition": { "version": "3.3.0" },
"passwd": {
"users": [
{
"name": "root",
"passwordHash": "YOURHASH",
"sshAuthorizedKeys": [
"ssh-... YOURKEY"
]
}
]
},
"systemd": {
"units": [{
"name": "sshd.service",
"enabled": true
}]
},
"storage": {
"files": [
{
"overwrite": true,
"path": "/etc/ssh/sshd_config.d/20-enable-passwords.conf",
"contents": {
"source": "data:,PasswordAuthentication%20yes%0APermitRootLogin%20yes%0A"
},
"mode": 420
}
]
}
}

…where password SHA512 hash can be obtained using openssl passwd -6 and the ssh key is your public ssh key.

That file is put to eg /tmp and referred in the virt-manager’s XML like follows:

<sysinfo type="fwcfg">
<entry name="opt/com.coreos/config" file="/tmp/config.ign"/>
</sysinfo>

Now we can boot up the VM and ssh in - or you could log in directly too but it’s easier to copy-paste commands when using ssh.

Inside the VM, we can follow the ALP documentation to install and start Cockpit:

podman container runlabel install registry.opensuse.org/suse/alp/workloads/tumbleweed_containerfiles/suse/alp/workloads/cockpit-ws:latest
podman container runlabel --name cockpit-ws run registry.opensuse.org/suse/alp/workloads/tumbleweed_containerfiles/suse/alp/workloads/cockpit-ws:latest
systemctl enable --now cockpit.service

Check your host’s IP address with ip -a, and open IP:9090 in your host’s browser:

Login with root / your password and you shall get the front page:

…and many other pages where you can manage your ALP deployment via browser:

All in all, ALP is in early phases but I’m really happy there’s up-to-date documentation provided and people can start experimenting it whenever they want. The images from the linked directory should be fairly good, and test automation with openQA has been started upon as well.

You can try out the other example workloads that are available just as well.

I’m about to migrate to a new laptop, having done a brief pre-purchase review of options on Fosstodon and reaching a decision to buy the NovaCustom NV41. Given the rapid launch and decline of Mastodon instances, I thought I’d better summarize my process and conclusion on my self-hosted blog until the fediverse self-hosting situation improves.

Since 2010 my main portable computing device has been the Lenovo X201 that replace the Dell Precision M65 that I bought in 2006. I have been incredibly happy with the X201, even to the point that in 2015 when I wanted to find a replacement, I couldn’t settle on a decision and eventually realized I couldn’t articulate what was wrong with the X201 and decided to just buy another X201 second-hand for my second office. There is still no deal-breaker with the X201, and I’m doing most of my computing on it including writing this post. However, today I can better articulate what is lacking with the X201 that I desire, and the state of the available options on the market has improved since my last attempt in 2015.

Briefly, my desired properties are:

• Portable – weight under 1.5kg
• Screen size 9-14″
• ISO keyboard layout, preferably Swedish layout
• Mouse trackpad, WiFi, USB and external screen connector
• Decent market availability: I should be able to purchase it from Sweden and have consumer protection, warranty, and some hope of getting service parts for the device
• Manufactured and sold by a vendor that is supportive of free software
• Preferably RJ45 connector (for data center visits)
• As little proprietary software as possible, inspired by FSF’s Respect Your Freedom
• Able to run a free operating system

My workload for the machine is Emacs, Firefox, Nextcloud client, GNOME, Evolution (mail & calendar), LibreOffice Calc/Writer, compiling software and some podman/qemu for testing. I have used Debian as the main operating system for the entire life of this laptop, but have experimented with PureOS recently. My current X201 is useful enough for this, although a faster machine wouldn’t hurt.

Based on my experience in 2015 that led me to make no decision, I changed perspective. This is a judgement call and I will not be able to fulfil all criteria. I will have to decide on a balance and the final choice will include elements that I really dislike, but still it will hopefully be better than nothing. The conflict for me mainly center around these parts:

• Non-free BIOS. This is software that runs on the main CPU and has full control of everything. I want this to run free software as much as possible. Coreboot is the main project in this area, although I prefer the more freedom-oriented Libreboot.
• Proprietary and software-upgradeable parts of the main CPU. This includes CPU microcode that is not distributed as free software. The Intel Management Engine (AMD and other CPU vendors has similar technology) falls into this category as well, and is problematic because it is an entire non-free operating system running within the CPU, with many security and freedom problems. This aspect is explored in the Libreboot FAQ further. Even if these parts can be disabled (Intel ME) or not utilized (CPU microcode), I believe the mere presence of these components in the design of the CPU is a problem, and I would prefer a CPU without these properties.
• Non-free software in other microprocessors in the laptop. Ultimately, I tend agree with the FSF’s “secondary processor” argument but when it is possible to chose between a secondary processor that runs free software and one that runs proprietary software, I would prefer as many secondary processors as possible to run free software. The libreboot binary blob reduction policy describes a move towards stronger requirements.
• Non-free firmware that has to be loaded during runtime into CPU or secondary processors. Using Linux-libre solves this but can cause some hardware to be unusable.
• WiFi, BlueTooth and physical network interface (NIC/RJ45). This is the most notable example of secondary processor problem with running non-free software and requiring non-free firmware. Sometimes these may even require non-free drivers, although in recent years this has usually been reduced into requiring non-free firmware.

The simplest choice for me would be to buy one of the FSF RYF certified laptops, and I actually already have a X200 with libreboot that I bought earlier for comparison. The reason the X200 didn’t work out as a replacement for me was the lack of a mouse trackpad, concerns about non-free EC firmware, Intel ME uncertainty and the non-free CPU microcode that I already have with my X201, but primarily that for some reason that I can’t fully articulate it feels weird to use a laptop manufactured by Lenovo but modified by third parties to be useful. I believe in market forces to pressure manufacturers into Doing The Right Thing, and feel that there is no incentive for Lenovo to use libreboot in the future when this market niche is already fulfilled by re-sellers modifying Lenovo laptops. So I’d be happier buying a laptop from someone who is natively supportive of they way I’m computing. I’m sure this aspect could be discussed a lot more, and maybe I’ll come back to do that, and could even reconsider my thinking (the right-to-repair argument may be compelling). I will definitely continue to monitor the list of RYF-certified laptops to see if future entries are more suitable options for me.

Eventually I decided to buy the NovaComputing NV41 laptop, and it arrived quickly and I’m in the process of setting it up. I hope to write a separate blog about it next.

I’m about to migrate to a new laptop, having done a brief pre-purchase review of options on Fosstodon and reaching a decision to buy the NovaCustom NV41. Given the rapid launch and decline of Mastodon instances, I thought I’d better summarize my process and conclusion on my self-hosted blog until the fediverse self-hosting situation improves.

Since 2010 my main portable computing device has been the Lenovo X201 that replace the Dell Precision M65 that I bought in 2006. I have been incredibly happy with the X201, even to the point that in 2015 when I wanted to find a replacement, I couldn’t settle on a decision and eventually realized I couldn’t articulate what was wrong with the X201 and decided to just buy another X201 second-hand for my second office. There is still no deal-breaker with the X201, and I’m doing most of my computing on it including writing this post. However, today I can better articulate what is lacking with the X201 that I desire, and the state of the available options on the market has improved since my last attempt in 2015.

Briefly, my desired properties are:

• Portable – weight under 1.5kg
• Screen size 9-14″
• ISO keyboard layout, preferably Swedish layout
• Mouse trackpad, WiFi, USB and external screen connector
• Decent market availability: I should be able to purchase it from Sweden and have consumer protection, warranty, and some hope of getting service parts for the device
• Manufactured and sold by a vendor that is supportive of free software
• Preferably RJ45 connector (for data center visits)
• As little proprietary software as possible, inspired by FSF’s Respect Your Freedom
• Able to run a free operating system

My workload for the machine is Emacs, Firefox, Nextcloud client, GNOME, Evolution (mail & calendar), LibreOffice Calc/Writer, compiling software and some podman/qemu for testing. I have used Debian as the main operating system for the entire life of this laptop, but have experimented with PureOS recently. My current X201 is useful enough for this, although a faster machine wouldn’t hurt.

Based on my experience in 2015 that led me to make no decision, I changed perspective. This is a judgement call and I will not be able to fulfil all criteria. I will have to decide on a balance and the final choice will include elements that I really dislike, but still it will hopefully be better than nothing. The conflict for me mainly center around these parts:

• Non-free BIOS. This is software that runs on the main CPU and has full control of everything. I want this to run free software as much as possible. Coreboot is the main project in this area, although I prefer the more freedom-oriented Libreboot.
• Proprietary and software-upgradeable parts of the main CPU. This includes CPU microcode that is not distributed as free software. The Intel Management Engine (AMD and other CPU vendors has similar technology) falls into this category as well, and is problematic because it is an entire non-free operating system running within the CPU, with many security and freedom problems. This aspect is explored in the Libreboot FAQ further. Even if these parts can be disabled (Intel ME) or not utilized (CPU microcode), I believe the mere presence of these components in the design of the CPU is a problem, and I would prefer a CPU without these properties.
• Non-free software in other microprocessors in the laptop. Ultimately, I tend agree with the FSF’s “secondary processor” argument but when it is possible to chose between a secondary processor that runs free software and one that runs proprietary software, I would prefer as many secondary processors as possible to run free software. The libreboot binary blob reduction policy describes a move towards stronger requirements.
• Non-free firmware that has to be loaded during runtime into CPU or secondary processors. Using Linux-libre solves this but can cause some hardware to be unusable.
• WiFi, BlueTooth and physical network interface (NIC/RJ45). This is the most notable example of secondary processor problem with running non-free software and requiring non-free firmware. Sometimes these may even require non-free drivers, although in recent years this has usually been reduced into requiring non-free firmware.

The simplest choice for me would be to buy one of the FSF RYF certified laptops, and I actually already have a X200 with libreboot that I bought earlier for comparison. The reason the X200 didn’t work out as a replacement for me was the lack of a mouse trackpad, concerns about non-free EC firmware, Intel ME uncertainty and the non-free CPU microcode that I already have with my X201, but primarily that for some reason that I can’t fully articulate it feels weird to use a laptop manufactured by Lenovo but modified by third parties to be useful. I believe in market forces to pressure manufacturers into Doing The Right Thing, and feel that there is no incentive for Lenovo to use libreboot in the future when this market niche is already fulfilled by re-sellers modifying Lenovo laptops. So I’d be happier buying a laptop from someone who is natively supportive of they way I’m computing. I’m sure this aspect could be discussed a lot more, and maybe I’ll come back to do that, and could even reconsider my thinking (the right-to-repair argument may be compelling). I will definitely continue to monitor the list of RYF-certified laptops to see if future entries are more suitable options for me.

Eventually I decided to buy the NovaComputing NV41 laptop, and it arrived quickly and I’m in the process of setting it up. I hope to write a separate blog about it next.

WebAuthn improves login security a lot by making it significantly harder for a user's credentials to be misused - a WebAuthn token will only respond to a challenge if it's issued by the site a secret was issued to, and in general will only do so if the user provides proof of physical presence[1]. But giving people tokens is tedious and also I have a new laptop which only has USB-C but does have a working fingerprint reader and I hate the aesthetics of the Yubikey 5C Nano, so I've been thinking about what WebAuthn looks like done without extra hardware.

Let's talk about the broad set of problems first. For this to work you want to be able to generate a key in hardware (so it can't just be copied elsewhere if the machine is compromised), prove to a remote site that it's generated in hardware (so the remote site isn't confused about what security assertions you're making), and tie use of that key to the user being physically present (which may range from "I touched this object" to "I presented biometric evidence of identity"). What's important here is that a compromised OS shouldn't be able to just fake a response. For that to be possible, the chain between proof of physical presence to the secret needs to be outside the control of the OS.

For a physical security token like a Yubikey, this is pretty easy. The communication protocol involves the OS passing a challenge and the source of the challenge to the token. The token then waits for a physical touch, verifies that the source of the challenge corresponds to the secret it's being asked to respond to the challenge with, and provides a response. At the point where keys are being enrolled, the token can generate a signed attestation that it generated the key, and a remote site can then conclude that this key is legitimately sequestered away from the OS. This all takes place outside the control of the OS, meeting all the goals described above.

How about Macs? The easiest approach here is to make use of the secure enclave and TouchID. The secure enclave is a separate piece of hardware built into either a support chip (for x86-based Macs) or directly on the SoC (for ARM-based Macs). It's capable of generating keys and also capable of producing attestations that said key was generated on an Apple secure enclave ("Apple Anonymous Attestation", which has the interesting property of attesting that it was generated on Apple hardware, but not which Apple hardware, avoiding a lot of privacy concerns). These keys can have an associated policy that says they're only usable if the user provides a legitimate touch on the fingerprint sensor, which means it can not only assert physical presence of a user, it can assert physical presence of an authorised user. Communication between the fingerprint sensor and the secure enclave is a private channel that the OS can't meaningfully interfere with, which means even a compromised OS can't fake physical presence responses (eg, the OS can't record a legitimate fingerprint press and then send that to the secure enclave again in order to mimic the user being present - the secure enclave requires that each response from the fingerprint sensor be unique). This achieves our goals.

The PC space is more complicated. In the Mac case, communication between the biometric sensors (be that TouchID or FaceID) occurs in a controlled communication channel where all the hardware involved knows how to talk to the other hardware. In the PC case, the typical location where we'd store secrets is in the TPM, but TPMs conform to a standardised spec that has no understanding of this sort of communication, and biometric components on PCs have no way to communicate with the TPM other than via the OS. We can generate keys in the TPM, and the TPM can attest to those keys being TPM-generated, which means an attacker can't exfiltrate those secrets and mimic the user's token on another machine. But in the absence of any explicit binding between the TPM and the physical presence indicator, the association needs to be up to code running on the CPU. If that's in the OS, an attacker who compromises the OS can simply ask the TPM to respond to an challenge it wants, skipping the biometric validation entirely.

Windows solves this problem in an interesting way. The Windows Hello Enhanced Signin doesn't add new hardware, but relies on the use of virtualisation. The agent that handles WebAuthn responses isn't running in the OS, it's running in another VM that's entirely isolated from the OS. Hardware that supports this model has a mechanism for proving its identity to the local code (eg, fingerprint readers that support this can sign their responses with a key that has a certificate that chains back to Microsoft). Additionally, the secrets that are associated with the TPM can be held in this VM rather than in the OS, meaning that the OS can't use them directly. This means we have a flow where a browser asks for a WebAuthn response, that's passed to the VM, the VM asks the biometric device for proof of user presence (including some sort of random value to prevent the OS just replaying that), receives it, and then asks the TPM to generate a response to the challenge. Compromising the OS doesn't give you the ability to forge the responses between the biometric device and the VM, and doesn't give you access to the secrets in the TPM, so again we meet all our goals.

On Linux (and other free OSes), things are less good. Projects like tpm-fido generate keys on the TPM, but there's no secure channel between that code and whatever's providing proof of physical presence. An attacker who compromises the OS may not be able to copy the keys to their own system, but while they're on the compromised system they can respond to as many challenges as they like. That's not the same security assertion we have in the other cases.

Overall, Apple's approach is the simplest - having binding between the various hardware components involved means you can just ignore the OS entirely. Windows doesn't have the luxury of having as much control over what the hardware landscape looks like, so has to rely on virtualisation to provide a security barrier against a compromised OS. And in Linux land, we're fucked. Who do I have to pay to write a lightweight hypervisor that runs on commodity hardware and provides an environment where we can run this sort of code?


[1] As I discussed recently there are scenarios where these assertions are less strong, but even so

comments

We are nearing an important milestone in the KDE Frameworks 6 development, branching and thus splitting the development of KDE Frameworks 5 and 6 is getting really close now. That’s not the only noteworthy news from the KF6 work though.

Branching plan

Following the plan from Akademy this should happen after 5.101, which is about to be released. Given we still have a few things to sort out around tooling and infrastructure as well as scheduling constraints around the upcoming holiday season we are looking at the first week of January for this now.

By then we hope to have kdesrc-build metadata prepared and new CI jobs set up. We’d then create a branch in all Frameworks repository named kf5.

From that point on, KDE Frameworks 5 is considered feature-frozen, feature work should continue to happen in the master branch, primarily targeting KF6 then. Bugfixes of course can and should be backported for quite some time to come, and KF5 releases will continue with the same pattern as previously. At some future point we will likely start to increase the interval between releases though, as the amount of changes decreases.

Ideally nothing should change for application and Plasma developers by this, both kdesrc-build and the CI should continue to provide you with KF5 unless you explicitly chose to switch to KF6. If you manage manual Frameworks builds and/or work on Frameworks itself you’ll of course have to adjust to the new branches.

Despite all the planning it is quite possible we missed something and we’ll end up with some disruptions around branching, there is no precedent for two major versions of Frameworks to exist in parallel so far.

Android builds

We were able to do Android library builds on the CI against Qt 6 since quite some time, now also executable builds work, which gives us Android CI coverage also for applications.

This however does not include APK packaging yet (that’s not run as part of the normal CI), there is more work for this needed in ECM, as well as adjustments in the Android manifest and Gradle files in each application.

Application ports

Several larger modules have meanwhile also been adapted to build against Qt 6, such as Gwenview, Zanshin, KAlarm and libkomparediff2. The latter also unblocks looking at things like Kompare and KDevelop.

Based on Nico’s statistics, close to 60% of the repositories covered there have Qt 6 CI on at least one platform. A few modules on that list are already ported but blocked by missing dependencies on the CI, but many are still to be done.

How you can help

All of the above is the result of a big team effort, and you can be part of that!

To participate, here are the most important coordination and communication channels:

• The KF6 workboard has the overview of Frameworks tasks and their status, and same for the Plasma 6 workboard.
• There’s a bi-weekly call on meet.kde.org, topics are prepared here. The next one will be on Tuesday 20th December at 17:00 CET.
• For general communication there’s the kde-frameworks-devel mailing list and the #kde-devel channel on Matrix.

There’s another way to help as well, KDE’s End of Year Fundraiser. The infrastructure we do all this work on doesn’t come for free after all.