Feeds
PyPy: Finding Simple Rewrite Rules for the JIT with Z3
In June I was at the PLDI conference in Copenhagen to present a paper I co-authored with Max Bernstein. I also finally met John Regehr, who I'd been talking on social media for ages but had never met. John has been working on compiler correctness and better techniques for building compilers and optimizers since a very long time. The blog post Finding JIT Optimizer Bugs using SMT Solvers and Fuzzing was heavily inspired by this work. We talked a lot about his and his groups work on using Z3 for superoptimization and for finding missing optimizations. I have applied some of the things John told me about to the traces of PyPy's JIT, and wanted to blog about that. However, my draft felt quite hard to understand. Therefore I have now written this current post, to at least try to provide a somewhat gentler on-ramp to the topic.
In this post we will use the Python-API to Z3 to find local peephole rewrite rules for the operations in the intermediate representation of PyPy's tracing JIT. The code for this is simple enough that we can go through all of it.
The PyPy JIT produces traces of machine level instructions, which are optimized and then turned into machine code. The optimizer uses a number of approaches to make the traces more efficient. For integer operations it applies a number of arithmetic simplification rules rules, for example int_add(x, 0) -> x. When implementing these rules in the JIT there are two problems: How do we know that the rules are correct? And how do we know that we haven't forgotten any rules? We'll try to answer both of these, but the first one in particular.
We'll be using Z3, a satisfiability module theories (SMT) solver which has good bitvector support and most importantly an excellent Python API. We can use the solver to reason about bitvectors, which are how we will model machine integers.
To find rewrite rules, we will consider the binary operations (i.e. those taking two arguments) in PyPy traces that take and produce integers. The completely general form op(x, y) is not simplifiable on its own. But if either x == y or if one of the arguments is a constant, we can potentially simplify the operation into a simpler form. The results are either the variable x, or a (potentially different) constant. We'll ignore constant-folding where both arguments of the binary operation are constants. The possible results for a simplifiable binary operation are the variable x or another constant. This leaves the following patterns as possibilities:
- op(x, x) == x
- op(x, x) == c1
- op(x, c1) == x
- op(c1, x) == x
- op(x, c1) == c2
- op(c1, x) == c2
Our approach will be to take every single supported binary integer operation, instantiate all of these patterns, and try to ask Z3 whether the resulting simplification is valid for all values of x.
Quick intro to the Z3 Python-APIHere's a terminal session showing the use of the Z3 Python API:
>>>> import z3 >>>> # construct a Z3 bitvector variable of width 8, with name x: >>>> x = z3.BitVec('x', 8) >>>> # construct a more complicated formula by using operator overloading: >>>> x + x x + x >>>> x + 1 x + 1Z3 checks the "satisfiability" of a formula. This means that it tries to find an example set of concrete values for the variables that occur in a formula, such that the formula becomes true. Examples:
>>>> solver = z3.Solver() >>>> solver.check(x * x == 3) unsat >>>> # meaning no x fulfils this property >>>> >>>> solver.check(x * x == 9) sat >>>> model = solver.model() >>>> model [x = 253] >>>> model[x].as_signed_long() -3 >>>> # 253 is the same as -3 in two's complement arithmetic with 8 bitsIn order to use Z3 to prove something, we can ask Z3 to find counterexamples for the statement, meaning concrete values that would make the negation of the statement true:
>>>> solver.check(z3.Not(x ^ -1 == ~x)) unsatThe result unsat means that we just proved that x ^ -1 == ~x is true for all x, because there is no value for x that makes not (x ^ -1 == ~x) true (this works because -1 has all the bits set).
If we try to prove something incorrect in this way, the following happens:
>>>> solver.check(z3.Not(x ^ -1 == x)) satsat shows that x ^ -1 == x is (unsurprisingly) not always true, and we can ask for a counterexample:
>>>> solver.model() [x = 0]This way of proving this works because the check calls try to solve an (implicit) "exists" quantifier, over all the Z3 variables used in the formula. check will either return z3.unsat, which means that no concrete values make the formula true; or z3.sat, which means that you can get some concrete values that make the formula true by calling solver.model().
In math terms we prove things using check by de-Morgan's rules for quantifiers:
$$ \lnot \exists x: \lnot f(x) \implies \forall x: f(x) $$
Now that we've seen the basics of using the Z3 API on a few small examples, we'll use it in a bigger program.
Encoding the integer operations of RPython's JIT into Z3 formulasNow we'll use the API to reason about the integer operations of the PyPy JIT intermediate representation (IR). The binary integer operations are:
opnames2 = [ "int_add", "int_sub", "int_mul", "int_and", "int_or", "int_xor", "int_eq", "int_ne", "int_lt", "int_le", "int_gt", "int_ge", "uint_lt", "uint_le", "uint_gt", "uint_ge", "int_lshift", "int_rshift", "uint_rshift", "uint_mul_high", "int_pydiv", "int_pymod", ]There's not much special about the integer operations. Like in LLVM, most of them are signedness-independent: int_add, int_sub, int_mul, ... work correctly for unsigned integers but also for two's-complement signed integers. Exceptions for that are order comparisons like int_lt etc. for which we have unsigned variants uint_lt etc. All operations that produce a boolean result return a full-width integer 0 or 1 (the PyPy JIT supports only word-sized integers in its intermediate representation)
In order to reason about the IR operations, some ground work:
import z3 INTEGER_WIDTH = 64 solver = z3.Solver() solver.set("timeout", 10000) # milliseconds, ie 10s xvar = z3.BitVec('x', INTEGER_WIDTH) constvar = z3.BitVec('const', INTEGER_WIDTH) constvar2 = z3.BitVec('const2', INTEGER_WIDTH) TRUEBV = z3.BitVecVal(1, INTEGER_WIDTH) FALSEBV = z3.BitVecVal(0, INTEGER_WIDTH)And here's the a function to turn an integer IR operation of PyPy's JIT into Z3 formulas:
def z3_expression(opname, arg0, arg1=None): """ computes a tuple of (result, valid_if) of Z3 formulas. `result` is the formula representing the result of the operation, given argument formulas arg0 and arg1. `valid_if` is a pre-condition that must be true for the result to be meaningful. """ result = None valid_if = True # the precondition is mostly True, with few exceptions if opname == "int_add": result = arg0 + arg1 elif opname == "int_sub": result = arg0 - arg1 elif opname == "int_mul": result = arg0 * arg1 elif opname == "int_and": result = arg0 & arg1 elif opname == "int_or": result = arg0 | arg1 elif opname == "int_xor": result = arg0 ^ arg1 elif opname == "int_eq": result = cond(arg0 == arg1) elif opname == "int_ne": result = cond(arg0 != arg1) elif opname == "int_lt": result = cond(arg0 < arg1) elif opname == "int_le": result = cond(arg0 <= arg1) elif opname == "int_gt": result = cond(arg0 > arg1) elif opname == "int_ge": result = cond(arg0 >= arg1) elif opname == "uint_lt": result = cond(z3.ULT(arg0, arg1)) elif opname == "uint_le": result = cond(z3.ULE(arg0, arg1)) elif opname == "uint_gt": result = cond(z3.UGT(arg0, arg1)) elif opname == "uint_ge": result = cond(z3.UGE(arg0, arg1)) elif opname == "int_lshift": result = arg0 << arg1 valid_if = z3.And(arg1 >= 0, arg1 < INTEGER_WIDTH) elif opname == "int_rshift": result = arg0 << arg1 valid_if = z3.And(arg1 >= 0, arg1 < INTEGER_WIDTH) elif opname == "uint_rshift": result = z3.LShR(arg0, arg1) valid_if = z3.And(arg1 >= 0, arg1 < INTEGER_WIDTH) elif opname == "uint_mul_high": # zero-extend args to 2*INTEGER_WIDTH bit, then multiply and extract # highest INTEGER_WIDTH bits zarg0 = z3.ZeroExt(INTEGER_WIDTH, arg0) zarg1 = z3.ZeroExt(INTEGER_WIDTH, arg1) result = z3.Extract(INTEGER_WIDTH * 2 - 1, INTEGER_WIDTH, zarg0 * zarg1) elif opname == "int_pydiv": valid_if = arg1 != 0 r = arg0 / arg1 psubx = r * arg1 - arg0 result = r + (z3.If(arg1 < 0, psubx, -psubx) >> (INTEGER_WIDTH - 1)) elif opname == "int_pymod": valid_if = arg1 != 0 r = arg0 % arg1 result = r + (arg1 & z3.If(arg1 < 0, -r, r) >> (INTEGER_WIDTH - 1)) elif opname == "int_is_true": result = cond(arg0 != FALSEBV) elif opname == "int_is_zero": result = cond(arg0 == FALSEBV) elif opname == "int_neg": result = -arg0 elif opname == "int_invert": result = ~arg0 else: assert 0, "unknown operation " + opname return result, valid_if def cond(z3expr): """ helper function to turn a Z3 boolean result z3expr into a 1 or 0 bitvector, using z3.If """ return z3.If(z3expr, TRUEBV, FALSEBV)We map the semantics of a PyPy JIT operation to Z3 with the z3_expression function. It takes the name of a JIT operation and its two (or one) arguments into a pair of Z3 formulas, result and valid_if. The resulting formulas are constructed with the operator overloading of Z3 variables/formulas.
The first element result of the result of z3_expression represents the result of performing the operation. valid_if is a bool that represents a condition that needs to be True in order for the result of the operation to be defined. E.g. int_pydiv(a, b) is only valid if b != 0. Most operations are always valid, so they return True as that condition (we'll ignore valid_if for a bit, but it will become more relevant further down in the post).
We can define a helper function to prove things by finding counterexamples:
def prove(cond): """ Try to prove a condition cond by searching for counterexamples of its negation. """ z3res = solver.check(z3.Not(cond)) if z3res == z3.unsat: return True elif z3res == z3.unknown: # eg on timeout return False elif z3res == z3.sat: return False assert 0, "should be unreachable" Finding rewrite rulesNow we can start finding our first rewrite rules, following the first pattern op(x, x) -> x. We do this by iterating over all the supported binary operation names, getting the z3 expression for op(x, x) and then asking Z3 to prove op(x, x) == x.
for opname in opnames2: result, valid_if = z3_expression(opname, xvar, xvar) if prove(result == xvar): print(f"{opname}(x, x) -> x, {result}")This yields the simplifications:
int_and(x, x) -> x int_or(x, x) -> x Synthesizing constantsSupporting the next patterns is harder: op(x, x) == c1, op(x, c1) == x, and op(x, c1) == x. We don't know which constants to pick to try to get Z3 to prove the equality. We could iterate over common constants like 0, 1, MAXINT, etc, or even over all the 256 values for a bitvector of length 8. However, we will instead ask Z3 to find the constants for us too.
This can be done by using quantifiers, in this case z3.ForAll. The query we pose to Z3 is "does there exist a constant c1 such that for all x the following is true: op(x, c1) == x? Note that the constant c1 is not necessarily unique, there could be many of them. We generate several matching constant, and add that they must be different to the condition of the second and further queries.
We can express this in a helper function:
def find_constant(z3expr, number_of_results=5): condition = z3.ForAll( [xvar], z3expr ) for i in range(number_of_results): checkres = solver.check(condition) if checkres == z3.sat: # if a solver check succeeds, we can ask for a model, which is # concrete values for the variables constvar model = solver.model() const = model[constvar].as_signed_long() yield const # make sure we don't generate the same constant again on the # next call condition = z3.And(constvar != const, condition) else: # no (more) constants found breakWe can use this new function for the three mentioned patterns:
# try to find constants for op(x, x) == c for opname in opnames2: result, valid_if = z3_expression(opname, xvar, xvar) for const in find_constant(result == constvar): print(f"{opname}(x, x) -> {const}") # try to find constants for op(x, c) == x and op(c, x) == x for opname in opnames2: result, valid_if = z3_expression(opname, xvar, constvar) for const in find_constant(result == xvar): print(f"{opname}(x, {const}) -> x") result, valid_if = z3_expression(opname, constvar, xvar) for const in find_constant(result == xvar): print(f"{opname}({const}, x) -> x") # this code is not quite correct, we'll correct it laterTogether this yields the following new simplifications:
# careful, these are not all correct! int_sub(x, x) -> 0 int_xor(x, x) -> 0 int_eq(x, x) -> 1 int_ne(x, x) -> 0 int_lt(x, x) -> 0 int_le(x, x) -> 1 int_gt(x, x) -> 0 int_ge(x, x) -> 1 uint_lt(x, x) -> 0 uint_le(x, x) -> 1 uint_gt(x, x) -> 0 uint_ge(x, x) -> 1 uint_rshift(x, x) -> 0 int_pymod(x, x) -> 0 int_add(x, 0) -> x int_add(0, x) -> x int_sub(x, 0) -> x int_mul(x, 1) -> x int_mul(1, x) -> x int_and(x, -1) -> x int_and(-1, x) -> x int_or(x, 0) -> x int_or(0, x) -> x int_xor(x, 0) -> x int_xor(0, x) -> x int_lshift(x, 0) -> x int_rshift(x, 0) -> x uint_rshift(x, 0) -> x int_pydiv(x, 1) -> x int_pymod(x, 0) -> xMost of these look good at first glance, but the last one reveals a problem: we've been ignoring the valid_if expression up to now. We can stop doing that by changing the code like this, which adds z3.And(valid_if, ...) to the argument of the calls to find_constant:
# try to find constants for op(x, x) == c, op(x, c) == x and op(c, x) == x for opname in opnames2: result, valid_if = z3_expression(opname, xvar, xvar) for const in find_constant(z3.And(valid_if, result == constvar)): print(f"{opname}(x, x) -> {const}") # try to find constants for op(x, c) == x and op(c, x) == x for opname in opnames2: result, valid_if = z3_expression(opname, xvar, constvar) for const in find_constant(z3.And(result == xvar, valid_if)): print(f"{opname}(x, {const}) -> x") result, valid_if = z3_expression(opname, constvar, xvar) for const in find_constant(z3.And(result == xvar, valid_if)): print(f"{opname}({const}, x) -> x")And we get this list instead:
int_sub(x, x) -> 0 int_xor(x, x) -> 0 int_eq(x, x) -> 1 int_ne(x, x) -> 0 int_lt(x, x) -> 0 int_le(x, x) -> 1 int_gt(x, x) -> 0 int_ge(x, x) -> 1 uint_lt(x, x) -> 0 uint_le(x, x) -> 1 uint_gt(x, x) -> 0 uint_ge(x, x) -> 1 int_add(x, 0) -> x int_add(0, x) -> x int_sub(x, 0) -> x int_mul(x, 1) -> x int_mul(1, x) -> x int_and(x, -1) -> x int_and(-1, x) -> x int_or(x, 0) -> x int_or(0, x) -> x int_xor(x, 0) -> x int_xor(0, x) -> x int_lshift(x, 0) -> x int_rshift(x, 0) -> x uint_rshift(x, 0) -> x int_pydiv(x, 1) -> x Synthesizing two constantsFor the patterns op(x, c1) == c2 and op(c1, x) == c2 we need to synthesize two constants. We can again write a helper method for that:
def find_2consts(z3expr, number_of_results=5): condition = z3.ForAll( [xvar], z3expr ) for i in range(number_of_results): checkres = solver.check(condition) if checkres == z3.sat: model = solver.model() const = model[constvar].as_signed_long() const2 = model[constvar2].as_signed_long() yield const, const2 condition = z3.And(z3.Or(constvar != const, constvar2 != const2), condition) else: returnAnd then use it like this:
for opname in opnames2: # try to find constants c1, c2 such that op(c1, x) -> c2 result, valid_if = z3_expression(opname, constvar, xvar) consts = find_2consts(z3.And(valid_if, result == constvar2)) for const, const2 in consts: print(f"{opname}({const}, x) -> {const2}") # try to find constants c1, c2 such that op(x, c1) -> c2 result, valid_if = z3_expression(opname, xvar, constvar) consts = find_2consts(z3.And(valid_if, result == constvar2)) for const, const2 in consts: print("%s(x, %s) -> %s" % (opname, const, const2))Which yields some straightforward simplifications:
int_mul(0, x) -> 0 int_mul(x, 0) -> 0 int_and(0, x) -> 0 int_and(x, 0) -> 0 uint_lt(x, 0) -> 0 uint_le(0, x) -> 1 uint_gt(0, x) -> 0 uint_ge(x, 0) -> 1 int_lshift(0, x) -> 0 int_rshift(0, x) -> 0 uint_rshift(0, x) -> 0 uint_mul_high(0, x) -> 0 uint_mul_high(1, x) -> 0 uint_mul_high(x, 0) -> 0 uint_mul_high(x, 1) -> 0 int_pymod(x, 1) -> 0 int_pymod(x, -1) -> 0A few require a bit more thinking:
int_or(-1, x) -> -1 int_or(x, -1) -> -1The are true because in two's complement, -1 has all bits set.
The following ones require recognizing that -9223372036854775808 == -2**63 is the most negative signed 64-bit integer, and 9223372036854775807 == 2 ** 63 - 1 is the most positive one:
int_lt(9223372036854775807, x) -> 0 int_lt(x, -9223372036854775808) -> 0 int_le(-9223372036854775808, x) -> 1 int_le(x, 9223372036854775807) -> 1 int_gt(-9223372036854775808, x) -> 0 int_gt(x, 9223372036854775807) -> 0 int_ge(9223372036854775807, x) -> 1 int_ge(x, -9223372036854775808) -> 1The following ones are true because the bitpattern for -1 is the largest unsigned number:
uint_lt(-1, x) -> 0 uint_le(x, -1) -> 1 uint_gt(x, -1) -> 0 uint_ge(-1, x) -> 1 Strength ReductionsAll the patterns so far only had a variable or a constant on the target of the rewrite. We can also use the machinery to do strengh-reductions where we generate a single-argument operation op1(x) for input operations op(x, c1) or op(c1, x). To achieve this, we try all combinations of binary and unary operations. (We won't consider strength reductions where a binary operation gets turned into a "cheaper" other binary operation here.)
opnames1 = [ "int_is_true", "int_is_zero", "int_neg", "int_invert", ] for opname in opnames2: for opname1 in opnames1: result, valid_if = z3_expression(opname, xvar, constvar) # try to find a constant op(x, c) == g(x) result1, valid_if1 = z3_expression(opname1, xvar) consts = find_constant(z3.And(valid_if, valid_if1, result == result1)) for const in consts: print(f"{opname}(x, {const}) -> {opname1}(x)") # try to find a constant op(c, x) == g(x) result, valid_if = z3_expression(opname, constvar, xvar) result1, valid_if1 = z3_expression(opname1, xvar) consts = find_constant(z3.And(valid_if, valid_if1, result == result1)) for const in consts: print(f"{opname}({const}, x) -> {opname1}(x)")Which yields the following new simplifications:
int_sub(0, x) -> int_neg(x) int_sub(-1, x) -> int_invert(x) int_mul(x, -1) -> int_neg(x) int_mul(-1, x) -> int_neg(x) int_xor(x, -1) -> int_invert(x) int_xor(-1, x) -> int_invert(x) int_eq(x, 0) -> int_is_zero(x) int_eq(0, x) -> int_is_zero(x) int_ne(x, 0) -> int_is_true(x) int_ne(0, x) -> int_is_true(x) uint_lt(0, x) -> int_is_true(x) uint_lt(x, 1) -> int_is_zero(x) uint_le(1, x) -> int_is_true(x) uint_le(x, 0) -> int_is_zero(x) uint_gt(x, 0) -> int_is_true(x) uint_gt(1, x) -> int_is_zero(x) uint_ge(x, 1) -> int_is_true(x) uint_ge(0, x) -> int_is_zero(x) int_pydiv(x, -1) -> int_neg(x) ConclusionsWith not very little code we managed to generate a whole lot of local simplifications for integer operations in the IR of PyPy's JIT. The rules discovered that way are "simple", in the sense that they only require looking at a single instruction, and not where the arguments of that instruction came from. They also don't require any knowledge about the properties of the arguments of the instructions (e.g. that they are positive).
The rewrites in this post have mostly been in PyPy's JIT already. But now we mechanically confirmed that they are correct. I've also added the remaining useful looking ones, in particular int_eq(x, 0) -> int_is_zero(x) etc.
If we wanted to scale this approach up, we would have to work much harder! There are a bunch of problems that come with generalizing the approach to looking at sequences of instructions:
-
Combinatorial explosion: if we look at sequences of instructions, we very quickly get a combinatorial explosion and it becomes untractable to try all combinations.
-
Finding non-minimal patterns: Some complicated simplifications can be instances of simpler ones. For example, because int_add(x, 0) -> x, it's also true that int_add(int_sub(x, y), 0) -> int_sub(x, y). If we simply generate all possible sequences, we will find the latter simplification rule, which we would usually not care about.
-
Unclear usefulness: if we simply generate all rewrites up to a certain number of instructions, we will get a lot of patterns that are useless in the sense that they typically aren't found in realistic programs. It would be much better to somehow focus on the patterns that real benchmarks are using.
In the next blog post I'll discuss an alternative approach to simply generating all possible sequences of instructions, that tries to address these problems. This works by analyzing the real traces of benchmarks and mining those for inefficiencies, which only shows problems that occur in actual programs.
SourcesI've been re-reading a lot of blog posts from John's blog:
- Let’s Work on an LLVM Superoptimizer
- Early Superoptimizer Results
- A Few Synthesizing Superoptimizer Results
- Synthesizing Constants
but also papers:
Another of my favorite blogs has been Philipp Zucker's blog in the last year or two, lots of excellent posts about/using Z3 on there.
Wim Leers: XB week 7: Drupal Dev Days Burgas
It’s Drupal Dev Days week! With Lauri presenting a Keynote there and several people out this week (myself included), it was a quiet week for Experience Builder.
Consequently, it was the first week since the 0.x branch was opened that zero merge requests landed…
Even the weekly meeting was fairly quiet.
Missed a prior week? See all posts tagged Experience Builder.
Goal: make it possible to follow high-level progress by reading ~5 minutes/week. I hope this empowers more people to contribute when their unique skills can best be put to use!
For more detail, join the #experience-builder Slack channel. Check out the pinned items at the top!
But on the other side of the planet, undisturbed by things such as vacation or Drupal Dev Days, Lee “larowlan” made a big push forward on [META] Support component types other than SDC, in his MR where he proposes to already remove the current tight coupling to Single-Directory Components (SDC). That tight coupling obviously needs to be removed, but we needed to start somewhere — SDCs were merely the first step.
Still, even at Drupal Dev Days Burgas, important Experience Builder discussions happened: catch, Alex Pott and Dave “longwave” Long discussed the JSON-based data storage model in depth (that was proposed by Alex “effulgentsia” Bronstein). Some important steps forward were made!
Thanks to Lauri for reviewing this!
The Savvy Few: How to dynamically add the view id as a body class for every views page
Here’s a quick guide to add custom CSS classes to the body tag of your page based on the views name for targeted styling of views pages.
Read morePython Morsels: What are lists in Python?
Lists are used to store and manipulate an ordered collection of things.
Table of contents
- Lists are ordered collections
- Containment checking
- Length
- Modifying the contents of a list
- Indexing: looking up items by their position
- Lists are the first data structure to learn
This is a list:
>>> colors = ["purple", "green", "blue", "yellow"]We can prove that to ourselves by passing that object to Python's built-in type function:
>>> type(colors) <class 'list'>Lists are ordered collections of things.
We can create a new list by using square brackets ([]), and inside those square brackets, we put each of the items that we'd like our list to contain, separated by commas:
>>> numbers = [2, 1, 3, 4, 7, 11] >>> numbers [2, 1, 3, 4, 7, 11]Lists can contain any type of object. Each item in a list doesn't need to be of the same type, but in practice, they typically are.
So we might refer to this as a list of strings:
>>> colors = ["purple", "green", "blue", "yellow"]While this is a list of numbers:
>>> numbers = [2, 1, 3, 4, 7, 11] Containment checkingWe can check whether a …
Read the full article: https://www.pythonmorsels.com/what-are-lists/Peter Bengtsson: Converting Celsius to Fahrenheit with Python
The Drop is Always Moving: 📢 Drupal 11.0.0-rc1 is now available! Release candidates are not for production sites, they are intended for widespread testing in preparation for the upcoming stable release. Test now! 11.0.0 is planned for the week of July...
📢 Drupal 11.0.0-rc1 is now available! Release candidates are not for production sites, they are intended for widespread testing in preparation for the upcoming stable release. Test now! 11.0.0 is planned for the week of July 29, 2024. https://www.drupal.org/project/drupal/releases/11.0.0-rc1
The Drop is Always Moving: Open source Drupal Starshot slide deck from Lauri Timmanee and Gábor Hojtsy with live recording from Drupal Developer Days is now available at https://www.drupal.org/about/starshot/blog/open-source-drupal-starshot-slide-deck...
Open source Drupal Starshot slide deck from Lauri Timmanee and Gábor Hojtsy with live recording from Drupal Developer Days is now available at https://www.drupal.org/about/starshot/blog/open-source-drupal-starshot-slide-deck-with-recording-is-now-available Present this at your company, meetup, Drupal 11 launch party, and so on!
Drupal Starshot blog: Open source Drupal Starshot slide deck with recording is now available
Lauri Timmanee and myself presented a keynote on Drupal Starshot at Drupal Developer Days 2024 two weeks ago in Burgas, Bulgaria. We got requests from many people that more content on Starshot would be great, so we decided to open source our slide deck! The organizers just published the video recording as well, so while we did not put in speaker notes to these slides, you can get the details from the recording.
Feel free to make a copy of the slides and present this at your company, meetup, Drupal 11 launch party, and so on!
Check out the Drupal Starshot marketing page for further marketing resources, logos and backgrounds.
Promet Source: Scaling Government: Open Source vs Proprietary CMS
Python Software Foundation: Announcing Our New PyPI Support Specialist!
We are thrilled to announce that our first-ever search for a dedicated PyPI Support Specialist has concluded with the hire of Maria Ashna, the newest member of the Python Software Foundation (PSF) staff. Reporting to Ee Durbin, Director of Infrastructure, Maria joins us from a background in academic research, technical consulting, and theatre.
Maria will help the PSF to support one of our most critical services, the Python Package Index (PyPI). Over the past 23 years, PyPI has seen essentially exponential growth in traffic and users, relying for the most part on volunteers to support it. With the addition of requirements to keep all Python maintainers and users safe, our support load has outstretched our support resources for some time now. The Python Software Foundation committed to hiring to increase this capacity in April and we’re excited to have Maria on board to begin providing crucially needed support.
From Maria, “I am a firm believer in democratizing tech. The Open Source community is the lifeblood of such democratization, which is why I am excited to be part of PSF and to serve this community.”
As you see Maria around the PyPI support inbox, issue tracker, and discuss.python.org in the future we hope that you’ll extend a warm welcome! We’re eager to get her up and running to reduce the stress that users have been experiencing around PyPI support and further our work to improve and extend PyPI sustainably.
The Drop is Always Moving: For the first time, Drupal Starshot adds a team of advisors to the leadership team. This council will provide strategic input and feedback to help ensure Starshot meets the needs of key stakeholders and end-users. Members are...
For the first time, Drupal Starshot adds a team of advisors to the leadership team. This council will provide strategic input and feedback to help ensure Starshot meets the needs of key stakeholders and end-users. Members are announced now at https://www.drupal.org/about/starshot/blog/announcing-the-drupal-starshot-advisory-council
Drupal Starshot blog: Announcing the Drupal Starshot Advisory Council
I'm excited to announce the formation of the Drupal Starshot Advisory Council. When I announced Starshot's Leadership Team, I explained that we are innovating on the leadership model by adding a team of advisors. This council will provide strategic input and feedback to help ensure Drupal Starshot meets the needs of key stakeholders and end-users.
The Drupal Starshot initiative represents an ambitious effort to expand Drupal's reach and impact. To guide this effort, we've established a diverse Advisory Council that includes members of the Drupal Starshot project team, Drupal Association staff and Board of Directors, representatives from Drupal Certified Partners, Drupal Core Committers, and last but not least, individuals representing the target end-users for Drupal Starshot. This ensures a wide range of perspectives and expertise to inform the project's direction and decision-making.
The initial members include:
- Imre Gmelig Meijling, React Online / Drupal Association Board - Fundraising Committee
- Suzanne Dergachev, Evolving Web / Drupal Association Board - Marketing Committee
- Mike Herchel, Agileana / Drupal Association Board - Innovation Committee
- Tim Doyle, CEO at the Drupal Association
- Tim Lehnen, CTO at the Drupal Association
- Kristen Pol, Salsa Digital - Drupal Certified Partner representative
- Chris Yates, Pantheon - Drupal Certified Partner representative
- Luis Ribeiro, CI&T - Drupal Certified Partner representative
- Kathryn Carruthers, alakasam - End User representative
- Emma Horrell, University of Edinburgh - End User representative
- Nathaniel Catchpole (catch), Third and Grove / Tag1 Consulting - Drupal Core Committer
- Théodore Biadala (nod_), Très Bien Tech - Drupal Core Committer
The council has been meeting monthly to receive updates from myself and the Drupal Starshot Leadership Team. Members will provide feedback on project initiatives, offer recommendations, and share insights based on their diverse experiences and areas of expertise.
In addition to guiding the strategic direction of Drupal Starshot, the Advisory Council will play a vital role in communication and alignment between the Drupal Starshot team, the Drupal Association, Drupal Core, and the broader Drupal community.
I'm excited to be working with this accomplished group to make the Drupal Starshot vision a reality. Together we can expand the reach and impact of Drupal, and continue advancing our mission to make the web a better place.
This blog has been re-posted and edited with permission from Dries Buytaert's blog.
File attachments: starshot-council-1920w.jpgReal Python: The Real Python Podcast – Episode #212: Digging Into Graph Theory in Python With David Amos
Have you wondered about graph theory and how to start exploring it in Python? What resources and Python libraries can you use to experiment and learn more? This week on the show, former co-host David Amos returns to talk about what he's been up to and share his knowledge about graph theory in Python.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
The Drop is Always Moving: Announced in https://www.drupal.org/drupalorg/blog/ending-packagesdrupalorg-support-for-composer-1, Drupal Composer 1 support is being phased out! 1️⃣ New Drupal.org packages/releases will not be available for Composer 1...
Announced in https://www.drupal.org/drupalorg/blog/ending-packagesdrupalorg-support-for-composer-1, Drupal Composer 1 support is being phased out!
1️⃣ New Drupal.org packages/releases will not be available for Composer 1 after Aug 12, 2024.
2️⃣ Composer 1 support for older packages will be dropped after Oct 1, 2024.
Akademy 2024 T-Shirt orders open
Pre-orders are now open for the Akademy 2024 T-shirt, this is only for those who will be attending Akademy, in person, in Würzburg
Pre-orders will close on 31st July
We will be opening in a few weeks separate online orders for t-shirts for those who aren't attending in person
Web Review, Week 2024-28
Let’s go for my web review for the week 2024-28.
Take action to stop chat control now!Tags: tech, politics, law, privacy
It’s time to push European governments to abandon this nonsense.
https://www.patrick-breyer.de/en/take-action-to-stop-chat-control-now/
Tags: tech, ai, machine-learning, gpt, computer-vision
Those brand new models keep failing at surprisingly simple tasks.
https://vlmsareblind.github.io/
Tags: tech, browser, security
This is a concerning finding. One can escape from the browser to the system with such chaining.
https://spaceraccoon.dev/universal-code-execution-browser-extensions/
Tags: tech, security, supply-chain, dependencies
Good tour of all the way dependencies might get compromised in your supply chain. Getting this easy to detect is needed.
https://kerkour.com/supply-chain-attacks-and-backdoored-dependencies
Tags: tech, ubuntu, security
The title is a bit pushing it. Still, I didn’t realize some of the fine prints of the Ubuntu support schemes.
https://gld.mcphail.uk/posts/ubuntu-security-updates-are-a-confusing-mess/
Tags: tech, linux, kernel, rust
It’s nice to have a balanced view on the matter. It’s not just roses and rainbows. This gives a good overview of the current limitations and where Rust can give most benefits in the kernel.
https://www.usenix.org/conference/atc24/presentation/li-hongyu
Tags: tech, linux, system, systemd
Interesting approach to test system changes. Especially welcome on immutable systems.
https://www.codethink.co.uk/articles/2024/A-new-way-to-develop-on-Linux/
Tags: tech, linux, uefi, boot, system
Looks like GRUB days as the standard bootloader are counted. Booting straight using the Linux kernel could bring interesting benefits. Probably not doable on every hardware platform though.
https://fizuxchyk.wordpress.com/2024/06/13/nmbl-we-dont-need-a-bootloader/
Tags: tech, linux, packaging
An interesting puzzle to pursue. Is it possible to rebuild exactly the same binary distribution packages?
https://blog.josefsson.org/2024/07/10/towards-idempotent-rebuilds/
Tags: tech, simd, performance
Another interesting algorithm to handle using SIMD.
https://clement-jean.github.io/simd_binary_search_tree/
Tags: tech, linux, system
It’s really a good reminder of how powerful ptrace is. You can nicely intercept and change the behavior of syscalls with it.
https://healeycodes.com/making-python-less-random
Tags: tech, cloud, storage, cryptography, security, tools
Looks like a nice tool indeed. Might be handy.
https://www.andreagrandi.it/posts/cryptomator-end-to-end-encrypt-files-in-cloud/
Tags: tech, databases
A simple explanation about dirty writes during database transactions.
https://surfingcomplexity.blog/2024/07/05/dirty-writes/
Tags: tech, databases, uuid, performance
Forced to use UUID as primary key in a table? Then make sure to use them properly to not kill the performance more than necessary. Ideally use something else though.
https://maciejwalkowiak.com/blog/postgres-uuid-primary-key/
Tags: tech, programming, python
Ever wondered how attributes work in Python under the hood? Here is how.
https://snarky.ca/unravelling-attribute-access-in-python/
Tags: tech, programming, python
An interesting Python construct to make iterators based on a simple function.
https://mathspp.com/blog/til/making-an-iterator-out-of-a-function
Tags: tech, gui, fonts
Very long read but will be an essential resource to have a fine understanding of text rendering in its current form.
Tags: tech, quality, craftsmanship, engineering, complexity, history
Interesting musing about the “software crisis” which was declared in the late 60s. We’re coping with it by piling levels of abstractions but clearly we’re still not out of it. Our craft still needs to grow.
https://wryl.tech/log/2024/the-software-crisis.html
Tags: tech, team, organization, agile
Good reminder that teams are made out of people. It’s good to look at the daily standups less as a technical management tool and more as a need to get into the work.
https://tidyfirst.substack.com/p/standups-individual-teammate
Bye for now!
Talk Python to Me: #470: Python in Medicine and Patient Care
Russ Allbery: Review: The Splinter in the Sky
Review: The Splinter in the Sky, by Kemi Ashing-Giwa
Publisher: Saga Press Copyright: July 2023 ISBN: 1-6680-0849-1 Format: Kindle Pages: 372The Splinter in the Sky is a stand-alone science fiction political thriller. It is Kemi Ashing-Giwa's first novel.
Enitan is from Koriko, a vegetation-heavy moon colonized by the Vaalbaran empire. She lives in the Ijebu community with her sibling Xiang and has an on-again, off-again relationship with Ajana, the Vaalbaran-appointed governor. Xiang is studying to be an architect, which requires passing stringent entrance exams to be allowed to attend an ancillary imperial school intended for "primitives." Enitan works as a scribe and translator, one of the few Korikese allowed to use the sacred Orin language of Vaalbara. In her free time, she grows and processes tea.
When Xiang mysteriously disappears while she's at work, Enitan goes to Ajana for help. Then Ajana dies, supposedly from suicide. The Vaalbaran government demands a local hostage while the death is investigated, someone who will be held as a diplomatic "guest" on the home world and executed if there is any local unrest. This hostage is supposed to be the child of the local headwoman, a concept that the Korikese do not have. Seeing a chance to search for Xiang, Enitan volunteers, heading into the heart of imperial power with nothing but desperate determination and a tea set.
The empire doesn't stand a chance.
Admittedly, a lot of the reason why the empire doesn't stand a chance is because the author is thoroughly on Enitan's side. Before she even arrives on Gondwana, Vaalbara's home world, Enitan is recruited as a spy by the other Gondwana power and Vaalbara's long-standing enemy. Her arrival in the Splinter, the floating arcology that serves as the center of Vaalbaran government, is followed by a startlingly meteoric rise in access. Some of this is explained by being a cultural curiosity for bored nobles, and some is explained by political factors Enitan is not yet aware of, but one can see the author's thumb resting on the scales.
This was the sort of book that was great fun to read, but whose political implausibility provoked "wait, that didn't make sense" thoughts afterwards. I think one has to assume that the total population of Vaalbara is much less than first comes to mind when considering an interplanetary empire, which would help explain the odd lack of bureaucracy. Enitan is also living in, effectively, the palace complex, for reasonably well-explained political reasons, and that could grant her a surprising amount of access. But there are other things that are harder to explain away: the lack of surveillance, the relative lack of guards, and the odd political structure that's required for the plot to work.
It's tricky to talk about this without spoilers, but the plot rests heavily on a conspiratorial view of how government power is wielded that I think strains plausibility. I'm not naive enough to think that the true power structure of a society matches the formal power structure, but I don't think they diverge as much as people think they do. It's one thing to say that the true power brokers of society can be largely unknown to the general population. In a repressive society with a weak media, that's believable. It's quite another matter for the people inside the palace to be in the dark about who is running what.
I thought that was the biggest problem with this book. Its greatest feature is the characters, and particularly the character relationships. Enitan is an excellent protagonist: fascinating, sympathetic, determined, and daring in ways that make her success more believable. Early in the book, she forms an uneasy partnership that becomes the heart of the book, and I loved everything about that relationship. The politics of her situation might be a bit too simple, but the emotions were extremely well-done.
This is a book about colonialism. Specifically, it's a book about cultural looting, appropriation, and racist superiority. The Vaalbarans consider Enitan barely better than an animal, and in her home they're merciless and repressive. Taken out of that context into their imperial capital, they see her as a harmless curiosity and novelty. Enitan exploits this in ways that are entirely believable. She is also driven to incandescent fury in ways that are entirely believable, and which she only rarely can allow herself to act on. Ashing-Giwa drives home the sheer uselessness of even the more sympathetic Vaalbarans more forthrightly than science fiction is usually willing to be. It's not a subtle point, but it is an accurate one.
The first two thirds of this book had me thoroughly engrossed and unable to put it down. The last third unfortunately turns into a Pokémon hunt of antagonists, which I found less satisfying and somewhat less believable. I wish there had been more need for Enitan to build political alliances and go deeper into the social maneuverings of the first part of the book, rather than gaining some deus ex machina allies who trivially solve some otherwise-tricky plot problems. The setup is amazing; the resolution felt a bit like escaping a maze by blasting through the walls, which I don't think played to the strengths of the characters and relationships that Ashing-Giwa had constructed. The advantage of that approach is that we do get a satisfying resolution and a standalone novel.
The central relationship of the book is unfortunately too much of a spoiler to talk about in a review, but I thought it was the best part of the story. This is a political thriller on the surface, but I think it's heart is an unexpected political alliance with a fascinatingly tricky balance of power. I was delighted that Ashing-Giwa never allows the tension in that relationship to collapse into one of the stock patterns it so easily could have become.
The Splinter in the Sky reminded me a little of Arkady Martine's A Memory Called Empire. It's not as assured or as adroitly balanced as that book, and the characters are not quite as memorable, but that's a very high bar. The political point is even sharper, and it has some of the same appeal.
I had so much fun reading this book. You may need to suspend your disbelief about some of the politics, and I wish the conclusion had been a bit less brute-force, but this is great stuff. Recommended when you're in the mood for a character story in the trappings of a political thriller.
Rating: 8 out of 10
Freexian Collaborators: Monthly report about Debian Long Term Support, June 2024 (by Roberto C. Sánchez)
Like each month, have a look at the work funded by Freexian’s Debian LTS offering.
Debian LTS contributorsIn June, 18 contributors have been paid to work on Debian LTS, their reports are available:
- Adrian Bunk did 47.0h (out of 74.25h assigned and 11.75h from previous period), thus carrying over 39.0h to the next month.
- Arturo Borrero Gonzalez did 6.0h (out of 6.0h assigned).
- Bastien Roucariès did 20.0h (out of 20.0h assigned).
- Ben Hutchings did 15.5h (out of 16.0h assigned and 8.0h from previous period), thus carrying over 8.5h to the next month.
- Chris Lamb did 18.0h (out of 18.0h assigned).
- Daniel Leidert did 4.0h (out of 8.0h assigned and 2.0h from previous period), thus carrying over 6.0h to the next month.
- Emilio Pozuelo Monfort did 23.25h (out of 49.5h assigned and 10.5h from previous period), thus carrying over 36.75h to the next month.
- Guilhem Moulin did 4.5h (out of 13.0h assigned and 7.0h from previous period), thus carrying over 15.5h to the next month.
- Lee Garrett did 17.0h (out of 25.0h assigned and 35.0h from previous period), thus carrying over 43.0h to the next month.
- Lucas Kanashiro did 5.0h (out of 10.0h assigned and 10.0h from previous period), thus carrying over 15.0h to the next month.
- Markus Koschany did 40.0h (out of 40.0h assigned).
- Ola Lundqvist did 10.0h (out of 6.5h assigned and 17.5h from previous period), thus carrying over 14.0h to the next month.
- Roberto C. Sánchez did 5.25h (out of 7.75h assigned and 4.25h from previous period), thus carrying over 6.75h to the next month.
- Santiago Ruano Rincón did 22.5h (out of 14.5h assigned and 8.0h from previous period).
- Sean Whitton did 6.5h (out of 6.0h assigned and 0.5h from previous period).
- Stefano Rivera did 0.5h (out of 0.0h assigned and 10.0h from previous period), thus carrying over 9.5h to the next month.
- Sylvain Beucler did 9.0h (out of 24.5h assigned and 35.5h from previous period), thus carrying over 51.0h to the next month.
- Thorsten Alteholz did 14.0h (out of 14.0h assigned).
In June, we have released 31 DLAs.
Notable security updates in June included:
- git: multiple vulnerabilities, which may result in privilege escalation, denial of service, and arbitrary code execution
- sendmail: SMTP smuggling allowed remote attackers bypass SPF protection checks
- cups: arbitrary remote code execution
Looking further afield to the broader Debian ecosystem, LTS contributor Bastien Roucariès also patched sendmail in Debian 12 (bookworm) and 11 (bullseye) in order to fix the previously mentioned SMTP smuggling vulnerability. Furthermore, LTS contributor Thorsten Alteholz provided fixes for the cups packages in Debian 12 (bookworm) and 11 (bullseye) in order to fix the aforementioned arbitrary remote code execution vulnerability.
Additionally, LTS contributor Ben Hutchings has commenced work on an updated backport of Linux kernel 6.1 to Debian 11 (bullseye), in preparation for bullseye transitioning to the responsibility of the LTS (and the associated closure of the bullseye-backports repository). LTS Lucas Kanashiro also began the preparatory work of backporting parts of the rust/cargo toolchain to Debian 11 (bullseye) in order to make future updates of the clamav virus scanner possible.
June was the final month of LTS for Debian 10 (as announced on the debian-lts-announce mailing list). No additional Debian 10 security updates will be made available on security.debian.org.
However, Freexian and its team of paid Debian contributors will continue to maintain Debian 10 going forward for the customers of the Extended LTS offer. Subscribe right away if you sill have Debian 10 which must be kept secure (and which cannot yet be upgraded).
Thanks to our sponsorsSponsors that joined recently are in bold.
- Platinum sponsors:
- TOSHIBA (for 105 months)
- Civil Infrastructure Platform (CIP) (for 73 months)
- VyOS Inc (for 37 months)
- Gold sponsors:
- Roche Diagnostics International AG (for 115 months)
- Linode (for 109 months)
- Babiel GmbH (for 99 months)
- Plat’Home (for 98 months)
- CINECA (for 73 months)
- University of Oxford (for 55 months)
- Deveryware (for 42 months)
- EDF SA (for 27 months)
- Dataport AöR
- Silver sponsors:
- Domeneshop AS (for 120 months)
- Nantes Métropole (for 114 months)
- Univention GmbH (for 106 months)
- Université Jean Monnet de St Etienne (for 106 months)
- Ribbon Communications, Inc. (for 100 months)
- Exonet B.V. (for 90 months)
- Leibniz Rechenzentrum (for 84 months)
- Ministère de l’Europe et des Affaires Étrangères (for 68 months)
- Cloudways by DigitalOcean (for 57 months)
- Dinahosting SL (for 55 months)
- Bauer Xcel Media Deutschland KG (for 49 months)
- Platform.sh SAS (for 49 months)
- Moxa Inc. (for 43 months)
- sipgate GmbH (for 41 months)
- OVH US LLC (for 39 months)
- Tilburg University (for 39 months)
- GSI Helmholtzzentrum für Schwerionenforschung GmbH (for 30 months)
- Soliton Systems K.K. (for 27 months)
- THINline s.r.o. (for 3 months)
- Bronze sponsors:
- Evolix (for 120 months)
- Seznam.cz, a.s. (for 120 months)
- Intevation GmbH (for 117 months)
- Linuxhotel GmbH (for 117 months)
- Daevel SARL (for 116 months)
- Bitfolk LTD (for 115 months)
- Megaspace Internet Services GmbH (for 115 months)
- Greenbone AG (for 114 months)
- NUMLOG (for 114 months)
- WinGo AG (for 113 months)
- Ecole Centrale de Nantes - LHEEA (for 109 months)
- Entr’ouvert (for 104 months)
- Adfinis AG (for 102 months)
- Tesorion (for 97 months)
- GNI MEDIA (for 96 months)
- Laboratoire LEGI - UMR 5519 / CNRS (for 96 months)
- Bearstech (for 88 months)
- LiHAS (for 88 months)
- Catalyst IT Ltd (for 83 months)
- Supagro (for 78 months)
- Demarcq SAS (for 77 months)
- Université Grenoble Alpes (for 63 months)
- TouchWeb SAS (for 55 months)
- SPiN AG (for 52 months)
- CoreFiling (for 48 months)
- Institut des sciences cognitives Marc Jeannerod (for 43 months)
- Observatoire des Sciences de l’Univers de Grenoble (for 39 months)
- Tem Innovations GmbH (for 34 months)
- WordFinder.pro (for 33 months)
- CNRS DT INSU Résif (for 32 months)
- Alter Way (for 25 months)
- Institut Camille Jordan (for 15 months)