Feeds

The Drupal Association’s individual membership program has always played a crucial role in supporting the Drupal community and ensuring the ongoing success of the Drupal project. The program was initially set up as a transactional vehicle: aside from the badge and voting rights, members received access to discounts from Drupal services providers.

The individual membership program stayed on autopilot during the turmoil of the Covid pandemic as we made the difficult decision to cancel DrupalCon North America 2020. During this time, our members and other Drupal community supporters donated unprecedented unrestricted funds using the hashtag #DrupalCares.

I joined the Drupal Association about two years ago as the Development & Membership Manager. My role split my time between Drupal Certified partners and the individual membership program, however it was clear from the beginning that the individual membership program would need a lot more attention.

The membership program underwent significant transformation from 2019 through May 2023, overcoming challenges and celebrating successes along the way. Initially, we faced a decline in numbers, but through consistent effort and unprecedented generosity, we've made remarkable strides. Today, we proudly recognize 1,747 members, with 70% of them providing recurring support.

Ripple Makers: Celebrating Changemakers in our Community 

The individual membership program rebranded as Ripple Makers in 2024. With this new name, the Drupal Association increases focus on communication, transparency, and engagement within the community. The ‘new’ program encourages sustaining donors to make monthly recurring gifts, which provide a reliable source of funding. This financial support allows the Drupal Association to plan for the future, support essential projects, and foster a dynamic and responsive community​.

Membership Programs, Sustainable Giving, and Nonprofits 

Why does a nonprofit organization such as the Drupal Association need a sustaining giving program? This program is vital for the sustainability and growth of the Drupal Association, and the benefit it brings to the community. It provides a stable foundation of support, ensuring that we can continue to innovate and grow. In addition, unrestricted giving in particular allows nonprofits to allocate resources where they are needed most, supporting the overall health of the Drupal project. Importantly. It also opens up direct lines of communication with the community.

Positive Impact on the Drupal Community

The Drupal community thrives because of several factors: open source collaboration, supportive environment, diverse participation, commitment to quality, and others. In my opinion, a supportive environment is the most important one.

By becoming a Ripple Maker, you directly support a vibrant and inclusive community of people who care for the Drupal project. Your contributions empower us to foster a sustainable ecosystem for Drupal by harnessing the collective generosity and commitment to the future of Drupal. Learn more about the program and join the wave on our sign up page.

Thank you for your ongoing support and dedication. Let's make the next chapter of our sustainable giving program the best one yet!

Functional programming is a programming paradigm in which the primary method of computation is the evaluation of functions. But how does Python support functional programming?

In this tutorial, you’ll learn:

• What the functional programming paradigm entails
• What it means to say that functions are first-class citizens in Python
• How to define anonymous functions with the lambda keyword
• How to implement functional code using map(), filter(), and reduce()

Functional programming typically plays a minor role in Python code, but it’s still good to be familiar with it. You’ll probably encounter it from time to time when reading code written by others. And you may even find situations where it’s advantageous to use Python’s functional programming capabilities in your own code.

Get Your Code: Click here to download the free sample code that shows you when and how to use functional programming in Python.

What Is Functional Programming?

A pure function is a function whose output value follows solely from its input values without any observable side effects. In functional programming, a program consists primarily of the evaluation of pure functions. Computation proceeds by nested or composed function calls without changes to state or mutable data.

The functional paradigm is popular because it offers several advantages over other programming paradigms. Functional code is:

• High level: You describe the result you want rather than explicitly specifying the steps required to get there. Single statements tend to be concise but pack a lot of punch.
• Transparent: The behavior of a pure function can be described by its inputs and outputs, without intermediary values. This eliminates the possibility of side effects and facilitates debugging.
• Parallelizable: Routines that don’t cause side effects can more easily run in parallel with one another.

Many programming languages support some degree of functional programming. In some languages, virtually all code follows the functional paradigm. Haskell is one such example. Python, by contrast, does support functional programming but contains features of other programming models as well.

While it’s true that an in-depth description of functional programming is somewhat complex, the goal here isn’t to present a rigorous definition but to show you what you can do by way of functional programming in Python.

How Well Does Python Support Functional Programming?

To support functional programming, it’s beneficial if a function in a given programming language can do these two things:

1. Take another function as an argument
2. Return another function to its caller

Python plays nicely in both respects. Everything in Python is an object, and all objects in Python have more or less equal stature. Functions are no exception.

In Python, functions are first-class citizens. This means that functions have the same characteristics as values like strings and numbers. Anything you would expect to be able to do with a string or number, you can also do with a function.

For example, you can assign a function to a variable. You can then use that variable the same way you would use the function itself:

Python 1>>> def func(): 2... print("I am function func()!") 3... 4 5>>> func() 6I am function func()! 7 8>>> another_name = func 9>>> another_name() 10I am function func()! Copied!

The assignment another_name = func on line 8 creates a new reference to func() named another_name. You can then call the function by either of the two names, func or another_name, as shown on lines 5 and 9.

You can display a function to the console with print(), include it as an element in a composite data object like a list, or even use it as a dictionary key:

Python >>> def func(): ... print("I am function func()!") ... >>> print("cat", func, 42) cat <function func at 0x7f81b4d29bf8> 42 >>> objects = ["cat", func, 42] >>> objects[1] <function func at 0x7f81b4d29bf8> >>> objects[1]() I am function func()! >>> d = {"cat": 1, func: 2, 42: 3} >>> d[func] 2 Copied!

In this example, func() appears in all the same contexts as the values "cat" and 42, and the interpreter handles it just fine.

Note: What you can or can’t do with an object in Python depends to some extent on context. Some operations work for certain object types but not for others.

For example, you can add two integer objects or concatenate two string objects with the plus operator (+), but the plus operator isn’t defined for function objects.

For present purposes, what matters is that functions in Python satisfy the two criteria beneficial for functional programming listed above. You can pass a function to another function as an argument:

Python 1>>> def inner(): 2... print("I am function inner()!") 3... 4 5>>> def outer(function): 6... function() 7... 8 9>>> outer(inner) 10I am function inner()! Copied! Read the full article at https://realpython.com/python-functional-programming/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Here's the relevant part of the new code that I have been working on in kis_tool_freehandhelper::paint : if (m_d->smoothingOptions->smoothingType() == KisSmoothingOptions::SIMPLE_SMOOTHING || m_d->smoothingOptions->smoothingType() == KisS...

Dear Readers, 

Imagine a bustling workshop filled with developers, designers, and enthusiasts collaborating to build something extraordinary. This is the scene as Drupal 11 emerges, packed with features designed to make web development more intuitive and efficient.

"In recent years, we've seen an uptick in innovation in Drupal. Drupal 11 continues this trend with many new and exciting features."

notes Dries Buytaert, Founder and Lead of Drupal.

He emphasises that Drupal 11 is designed to empower ambitious site builders to create exceptional websites and accelerate Drupal's innovation. With this release, Drupal has become more intuitive, powerful, and flexible, ensuring it remains a leader in web development and digital experience creation.

Key among these innovations are Recipes and Single-Directory Components (SDCs). Recipes act like pre-assembled kits, allowing developers to add features to their websites with ease. Meanwhile, SDCs gather all necessary code for each component into one tidy package, simplifying the workflow.

Drupal 11 boasts superior performance, running up to 50% faster on PHP 8.3 compared to its predecessors. This improvement ensures swift page loading and an overall enhanced user experience. Accessibility remains a key focus, with Drupal continuing to support over 100 languages, ensuring inclusivity and usability for a global audience. This new release is the product of a vibrant community effort, with 1,858 individuals from 590 organizations contributing their expertise and passion. It’s a shining example of what can be achieved when people come together with a common goal: to push the boundaries of what’s possible with Drupal.

But the excitement doesn’t end with Drupal 11. The community is already buzzing about the upcoming Drupal Starshot project. Starshot aims to make Drupal more accessible than ever, especially for newcomers. By integrating user-friendly tools like the Project Browser and automatic updates, Starshot promises a smooth journey from installation to launching a fully functional website. With 148 days left for the year, the community is eagerly anticipating the arrival of the initial version of Starshot.

These developments are more than just updates; they’re part of an ongoing story of innovation and collaboration. With Drupal 11 and the forthcoming Starshot project, the Drupal community is not just keeping pace with the future—they're shaping it.

Moving on to stand-out stories of the past week.

The most important and currently happening update from the Drupal Community is the announcement of candidates for Drupal Association Board Elections. This year's election will fill one at-large board seat, with candidates Albert Hughes, Will Huggins, Alejandro Moreno, Janna Malikova, Kevin Quillen, Matthew Saunders, and Dominique De Cooman vying for the position. Voting will open on 15 August, requiring active Drupal Association memberships by 14 August to participate. The election results will be ratified between 6-13 September, with the new board member announced at DrupalCon Barcelona.

Last week, Daniel Cothran, in a conversation with Kazima Abbas, sub-editor of The DropTimes, shared his journey into web development and the creation of Views CSV Source. He explained how this module not only simplifies the data presentation process but also improves the efficiency and performance of Drupal sites, making it especially valuable for projects requiring reliable and streamlined data handling. Read the full article here.

In an email conversation, I had the pleasure of interviewing Jürgen Haas, Co-Founder of LakeDrops, and the creative mind behind the ECA module. During our discussion, Jürgen delved into the development of the ECA (Event, Condition, Action) module, which he designed to modernize workflow automation within Drupal. He shared the story behind the ECA module's inception, its development path, and its potential integration with future Drupal core updates, emphasizing its value in enhancing the user experience through intuitive tools.

The second part of the Thoughts on Starshot feature revealed widespread excitement within the Drupal community, highlighting the potential of the Starshot initiative to transform the Drupal platform. With contributions from seasoned community members like Kristen Pol, Murray Woodman, Nicolas Loye, Martin Anderson-Clutz, and Tim Hestenes Lehnen, the consensus is clear: Drupal Starshot promises to streamline the user experience, foster greater collaboration, and lower the barriers to entry, making Drupal more accessible to a wider audience.

DrupalCamp Ottawa 2024, held on August 2, brought together web development enthusiasts of all skill levels for a day of learning and networking centered around the Drupal platform. Highlighting key speakers like Martin Anderson-Clutz, the event emphasized community, inclusivity, and the latest advancements in Drupal, ensuring a successful and collaborative experience for all attendees. Read here.

The Pacific Northwest Drupal Summit is set to return for its 10th event, taking place from October 11 to 13, 2024, in Seattle, Washington. Since 2009, this summit has been a key regional event for Drupal professionals in the Pacific Northwest.

The Acquia 2024 Digital Freedom Tour will make its next stop in New York City on October 24, 2024. The event aims to advance a safer, more inclusive, and accessible digital world. It will bring together prominent digital leaders who will share their expertise in creating impactful digital experiences.

Anoop Singh, Tech Lead at Valuebound, announced on LinkedIn the upcoming release of the FlexiStyle Bootstrap SCSS theme on Drupal.org. This follows the success of the original FlexiStyle Bootstrap theme and promises even greater customization and flexibility for Drupal projects.

Additionally, amazee.io has released Lagoon V2.20, a significant update to its open-source application delivery platform. This release includes enhancements in user management, security, and onboarding efficiency designed to better support business needs.

Backdrop CMS 1.28.0 has been released, bringing significant enhancements to the platform, including new options for configuration storage. Laryn Kragt Bakker, Senior Developer at Aten Design Group, detailed the update, which allows users to choose between storing their configuration data in the file system or the database.

We acknowledge that there are more stories to share. However, due to selection constraints, we must pause further exploration for now.

To get timely updates, follow us on LinkedIn, Twitter and Facebook. You can also, join us on Drupal Slack at #thedroptimes.

Thank you,
Sincerely
Alka Elizabeth
Sub-editor, The DropTimes.

In this quiz, you’ll test your understanding of Functional Programming in Python.

By working through this quiz, you’ll revisit the functional programming paradigm, the concept of functions as first-class citizens in Python, the use of the lambda keyword, and how to implement functional code using map(), filter(), and reduce().

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this digital world, it is absolutely necessary to stay up-to-date with the latest upgrades and trends. Drupal 11 is here, bringing you better security, performance, and user experience. As a pioneer in 360 Drupal services, LN Webworks is excited to help you upgrade 

In this blog, we will navigate more about the intricacies of Drupal 11’s core attributes, the benefits of embracing the latest update, and how LN Webworks stands as your steadfast Drupal Certification Migration Partner for a successful migration.

<p>How we fixed a buggy custom Drupal module for setting up a <a href="https://www.adcisolutions.com/work/product-catalog-import?utm_source=planetdrupal%26utm_medium=rss_feed%26utm_campaign=catalog_import">product catalog</a> and got faulty catalog filters working with Ajax modules.</p><img data-entity-uuid="89483bc1-60e8-4538-8747-32b5bb9d1100" data-entity-type="file" src="https://www.adcisolutions.com/sites/default/files/inline-images/catalog-data-import.png" width="1986" height="1263" alt="catalog import">

The European Accessibility Act (EAA) aims to enhance the accessibility of products and services for people with disabilities and the elderly throughout the EU. With less than a year remaining to meet compliance requirements, both the public and private sectors must act now.

API development workflow with Zato 2024-08-05, by Dariusz Suchojad

Zato is an integration platform and backend application server which means that, during most of their projects, developers using Zato are interested in a few specific matters.

The platform concentrates on answering these key, everyday questions that Python backend developers routinely ask:

• How do I connect to external systems or databases to store or extract data?
• How do I map messages from one format to another?
• How do I automate my work so that it is repeatable across environments?
• How can I focus on my job instead of thinking of trivial low-level details?
• How do I debug my code?
• How can I reuse the skills and knowledge that I obtained elsewhere and how can I better myself?
• How do I manage the complexity of having to integrate at least several systems and dozens or hundreds of APIs?

The typical workflow

Dashboard

• Dashboard is a web-based administration console that lets you define all the various connection pools, security types, scheduler jobs or other integration assets that you later make use of in your Python-based API services

• When you install Zato, it is comfortable to use its Dashboard to define resources like REST, MongoDB or SOAP connections pools because it does not require any programming, you just fill out a form and that resource is created

Python

• All of your code is in Python - that makes it easy to manipulate any kind of data that you receive or may need to access

• In your code, you only deal with the business logic - if your service is invoked, it means that all the security layers, serializers and other low-level parts have already completed their job and you are given a nice Python-based business object that represents your input. Based on this input information, you create output that your business logic dictates, still working purely on a high-level of Python objects, without any low-level details.

• Static typing is encouraged through the use of dataclasses for your data models

• Your code is hot-deployed from your IDE, no matter where your servers are, e.g. you work on Mac but the server is a remote Linux instance

• You can debug your services from your IDE as well and, again, the server can be a remote one

# -*- coding: utf-8 -*- # stdlib from dataclasses import dataclass # Zato from zato.server.service import Model, Service @dataclass(init=False) class GetClientRequest(Model): client_id: int @dataclass(init=False) class GetClientResponse(Model): name: str email: str segment: str class GetClient(Service): class SimpleIO: input = GetClientRequest output = GetClientResponse def handle(self): # Enable type checking and type completion request = self.request.input # type: GetClientRequest # Log details of our request self.logger.info('Processing client `%s`', request.client_id) # Produce our response - in a full service this information # will be obtained from one or more databases or systems. response = GetClientResponse() response.name = 'Jane Doe' response.email = 'hello@example.com' response.segment = 'RNZ' # Return the response to our caller self.response.payload = response

Such a service can be assigned to one or more REST channels and we can invoke it now:

$ curl localhost:17010/api/v1/client -d '{"client_id":123}' {"name":"Jane Doe","email":"hello@example.com","segment":"RNZ"} $ Automation

• Using Dashboard is very handy, and, once you are satisfied with what you created using it, the resources can be exported from command line using a built-in tool called enmasse - the result is a regular YAML file that describes every resource that you created using Dashboard

• In the next step, an emasse file can be imported in a new environment, also from command line. For instance, you can keep adding new connection definitions to the export file each day and they can be imported in a separate testing environment by CI/CD regression testing processes.

• Thanks to the enmasse export and import cycle, you do not need to manually recreate any resources that you initially created in your development environment. Moreover, because enmasse files are simple YAML files, you can script their usage in any way required when they are being import, e.g. by filling out credentials or other details that each environment may differ by.

• Similarly, with your Python code, you can commit your code to git and configure each environment to hot-deploy it directly from a git checkout. Alternatively, Python code can be deployed statically, which requires a server restart. The latter approach is useful when you, for instance, prepare an AWS AMI with your solution that you create new runtime instance from, because it lets you embed your entire enmasse definition and code directly in your AMIs.

OpenAPI

• Definitions of your services can be exported to OpenAPI

• Note that, by default, your services can be multi-protocol - it means that you may have a service that only uses WebSockets, or perhaps is a background one that has no REST endpoint at all, and yet, you still will be able to export it to OpenAPI

• Taken together, it means that you can use any OpenAPI-compatible tool, such as Postman, for accessing and testing your API services, no matter if they use REST or not

• You can also generate an entire static HTML site that will include OpenAPI as well as static pages describing your APIs - this is useful, for instance when you need to share with your tech partners not only an OpenAPI definition but a pretty-looking API documentation site as well

Let's generate an OpenAPI definition for the GetClient service above:

$ zato openapi /path/to/server/server1 \ --include "client*" \ --file /tmp/openapi.yaml

Now, we can import the OpenAPI definition to Postman and invoke our services from it:

Python once more

• Python is at the core of Zato and the platform allows you to leverage your already existing skills and knowledge of third-party libraries

• For instance, the code below is a stand-alone program that connects to Redis to insert a key and read it back:

# -*- coding: utf-8 -*- # Redis import redis # Connect to the database conn = redis.Redis(host='localhost', port=6379, db=0) # Set a key .. conn.set('key', 'value') # .. read it back .. result = conn.get('key') # .. and log what we received. print('Result', result)

• Now, the same as a Zato-based service:

# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class MyService(Service): def handle(self): # Set a key .. self.kvdb.conn.set('key', 'value') # .. read it back .. result = self.kvdb.conn.get('key') # .. and log what we received. self.logger.info('Result %s', result)

• In both cases, it is exactly the same amount of code yet a couple of differences are notable.

  • First, because the configuration was created earlier using Dashboard or enmasse, your code does not need to be concerned with where Redis is, it just makes use of it.
  • Second, already at this point, your Zato code is a scalable API service that can be hot-deployed to clusters for other systems to invoke it.
  • Yet, the actual business logic, the usage of Redis, is the same, which means that you can reuse what you already know and build upon it while developing with Zato.

• Redis was but one example and the same principle applies to everything else. Be it SQL, MongoDB, ElasticSearch or any other connection type, you can take your existing code and simply embed it in Zato services.

• This also means that it is easy to find programming examples specific to a particular system because, in the end, when you connect to such an external system through Zato, you issue requests using a library which already is widely popular among Python programmers, like redis-py in this example above.

• Similarly, as a data scientist you may already have some Pandas or Spark code developed earlier - the same principle applies, you can embed it in your services or you can import it from your libraries, just like with regular Python code.

More resources

➤ Python API integration tutorial
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

We talk with Jamie Abrahams from FreelyGive about the new AI module and what it means for the future of AI modules in the Drupal ecosystem.

URLs mentioned

• The new AI module 
• Introductory video on the new AI module
• Press release announcing AI module
• Detailed introduction of AI module 
• #AI channel in Drupal Slack workspace 
• Issue queue discussion about generic "ai()" method
• LangChain  
• LlamaIndex 
• LM Studio 
• Ollama 

DrupalEasy News

• Professional module development - 15 weeks, 90 hours, live, online course.  
• Drupal Career Online - 12 weeks, 77 hours, live online, beginner-focused course.

Audio transcript

We're using the machine-driven Amazon Transcribe service to provide an audio transcript of this episode.

Subscribe

Subscribe to our podcast on iTunes, iHeart, Amazon, YouTube, or Spotify. 

If you'd like to leave us a voicemail, call 321-396-2340. Please keep in mind that we might play your voicemail during one of our future podcasts. Feel free to call in with suggestions, rants, questions, or corrections. If you'd rather just send us an email, please use our contact page.

Credits

Podcast edited by Amelia Anello.
 

The Freedesktop.org Specifications directory contains a list of common specifications that have accumulated over the decades and define how common desktop environment functionality works. The specifications are designed to increase interoperability between desktops. Common specifications make the life of both desktop-environment developers and especially application developers (who will almost always want to maximize the amount of Linux DEs their app can run on and behave as expected, to increase their apps target audience) a lot easier.

Unfortunately, building the HTML specifications and maintaining the directory of available specs has become a bit of a difficult chore, as the pipeline for building the site has become fairly old and unmaintained (parts of it still depended on Python 2). In order to make my life of maintaining this part of Freedesktop easier, I aimed to carefully modernize the website. I do have bigger plans to maybe eventually restructure the site to make it easier to navigate and not just a plain alphabetical list of specifications, and to integrate it with the Wiki, but in the interest of backwards compatibility and to get anything done in time (rather than taking on a mega-project that can’t be finished), I decided to just do the minimum modernization first to get a viable website, and do the rest later.

So, long story short: Most Freedesktop specs are written in DocBook XML. Some were plain HTML documents, some were DocBook SGML, a few were plaintext files. To make things easier to maintain, almost every specification is written in DocBook now. This also simplifies the review process and we may be able to switch to something else like AsciiDoc later if we want to. Of course, one could have switched to something else than DocBook, but that would have been a much bigger chore with a lot more broken links, and I did not want this to become an even bigger project than it already was and keep its scope somewhat narrow.

DocBook is a markup language for documentation which has been around for a very long time, and therefore has older tooling around it. But fortunately our friends at openSUSE created DAPS (DocBook Authoring and Publishing Suite) as a modern way to render DocBook documents to HTML and other file formats. DAPS is now used to generate all Freedesktop specifications on our website. The website index and the specification revisions are also now defined in structured TOML files, to make them easier to read and to extend. A bunch of specifications that had been missing from the original website are also added to the index and rendered on the website now.

Originally, I wanted to put the website live in a temporary location and solicit feedback, especially since some links have changed and not everything may have redirects. However, due to how GitLab Pages worked (and due to me not knowing GitLab CI well enough…) the changes went live before their MR was actually merged. Rather than reverting the change, I decided to keep it (as the old website did not build properly anymore) and to see if anything breaks. So far, no dead links or bad side effects have been observed, but:

If you notice any broken link to specifications.fd.o or anything else weird, please file a bug so that we can fix it!

Thank you, and I hope you enjoy reading the specifications in better rendering and more coherent look!

The Freedesktop.org Specifications directory contains a list of common specifications that have accumulated over the decades and define how common desktop environment functionality works. The specifications are designed to increase interoperability between desktops. Common specifications make the life of both desktop-environment developers and especially application developers (who will almost always want to maximize the amount of Linux DEs their app can run on and behave as expected, to increase their apps target audience) a lot easier.

Unfortunately, building the HTML specifications and maintaining the directory of available specs has become a bit of a difficult chore, as the pipeline for building the site has become fairly old and unmaintained (parts of it still depended on Python 2). In order to make my life of maintaining this part of Freedesktop easier, I aimed to carefully modernize the website. I do have bigger plans to maybe eventually restructure the site to make it easier to navigate and not just a plain alphabetical list of specifications, and to integrate it with the Wiki, but in the interest of backwards compatibility and to get anything done in time (rather than taking on a mega-project that can’t be finished), I decided to just do the minimum modernization first to get a viable website, and do the rest later.

So, long story short: Most Freedesktop specs are written in DocBook XML. Some were plain HTML documents, some were DocBook SGML, a few were plaintext files. To make things easier to maintain, almost every specification is written in DocBook now. This also simplifies the review process and we may be able to switch to something else like AsciiDoc later if we want to. Of course, one could have switched to something else than DocBook, but that would have been a much bigger chore with a lot more broken links, and I did not want this to become an even bigger project than it already was and keep its scope somewhat narrow.

DocBook is a markup language for documentation which has been around for a very long time, and therefore has older tooling around it. But fortunately our friends at openSUSE created DAPS (DocBook Authoring and Publishing Suite) as a modern way to render DocBook documents to HTML and other file formats. DAPS is now used to generate all Freedesktop specifications on our website. The website index and the specification revisions are also now defined in structured TOML files, to make them easier to read and to extend. A bunch of specifications that had been missing from the original website are also added to the index and rendered on the website now.

Originally, I wanted to put the website live in a temporary location and solicit feedback, especially since some links have changed and not everything may have redirects. However, due to how GitLab Pages worked (and due to me not knowing GitLab CI well enough…) the changes went live before their MR was actually merged. Rather than reverting the change, I decided to keep it (as the old website did not build properly anymore) and to see if anything breaks. So far, no dead links or bad side effects have been observed, but:

If you notice any broken link to specifications.fd.o or anything else weird, please file a bug so that we can fix it!

Thank you, and I hope you enjoy reading the specifications in better rendering and more coherent look!

Thankfully no tragedies to report this week! I thank each and everyone of you that has donated to my car fund. I still have a ways to go and could use some more help so that we can go to the funeral. https://gofund.me/033eb25d I am between contracts and work packages, so all of my work is currently for free. Thanks for your consideration.

Another very busy week getting qt6 updates in Debian, Kubuntu, and KDE snaps.

Kubuntu:

• Merkuro and Neochat SRUs have made progress.
• See Debian for the qt6 Plasma / applications work.

Debian:

• qtmpv – in NEW
• arianna – in NEW
• kamera – experimental
• libkdegames – experimental
• kdenetwork-filesharing – experimental
• xwaylandvideobridge – NEW
• futuresql – NEW
• kpat WIP
• Tokodon – Done, but needs qtmpv to pass NEW
• Gwenview – WIP needs kamera, kio-extras
• kio-extras – Blocked on kdsoap in which the maintainer is not responding to bug reports or emails. Will likely fork in Kubuntu as our freeze quickly approaches.

KDE Snaps:

Updated QT to 6.7.2 which required a rebuild of all our snaps. Also found an issue with mismatched ffmpeg libraries, we have to bundle them for now until versioning issues are resolved.

Made new theme snaps for KDE breeze: gtk-theme-breeze, icon-theme-breeze so if you use the plasma theme breeze please install these and run

for PLUG in $(snap connections | grep gtk-common-themes:icon-themes | awk '{print $2}'); do sudo snap connect ${PLUG} icon-theme-breeze:icon-themes; done for PLUG in $(snap connections | grep gtk-common-themes:gtk-3-themes | awk '{print $2}'); do sudo snap connect ${PLUG} gtk-theme-breeze:gtk-3-themes; done for PLUG in $(snap connections | grep gtk-common-themes:gtk-2-themes | awk '{print $2}'); do sudo snap connect ${PLUG} gtk-theme-breeze:gtk-2-themes; done

This should resolve most theming issues. We are still waiting for kdeglobals to be merged in snapd to fix colorscheme issues, it is set for next release. I am still working on qt6 themes and working out how to implement them in snaps as they are more complex than gtk themes with shared libraries and file structures.

Please note: Please help test the –edge snaps so I can promote them to stable.

• Elisa – stable https://snapcraft.io/elisa
• Okular – stable https://snapcraft.io/okular
• Konsole ( please note this is a confined terminal for the ‘project’ and not very useful except to ssh to the host system ) – stable
• Kwrite – stable https://snapcraft.io/kwrite
• Gwenview – stable https://snapcraft.io/gwenview
• Kate ( –classic –edge ) https://snapcraft.io/kate
• Gcompris –edge https://snapcraft.io/gcompris
• Alligator stable https://snapcraft.io/alligator
• Angelfish –edge https://snapcraft.io/angelfish ( Crashes on first run, but runs fine after that.. looking into it)
• Arianna –edge ( Just pushed fix for required browser-support plug)
• Ark stable https://snapcraft.io/ark
• Blinken stable https://snapcraft.io/blinken
• Bomber stable https://snapcraft.io/bomber
• Bovo stable https://snapcraft.io/bovo
• Calindori stable https://snapcraft.io/calindori
• Digikam stable https://snapcraft.io/digikam
• Dragon –edge ( just pushed dbus fix )
• Falkon stable https://snapcraft.io/falkon
• Filelight stable https://snapcraft.io/filelight
• GCompris –edge https://snapcraft.io/gcompris ( Will remain in edge until official release )
• Ghostwriter –edge ( Broken, need to workout Qt webengine obscure way of handling hunspell dictionaries.)
• Granatier stable https://snapcraft.io/granatier
• Kalzium stable https://snapcraft.io/kalzium
• Kapman stable https://snapcraft.io/kapman
• Kasts –edge ( Broken, portal failure, testing some plugs)
• Kate stable –classic https://snapcraft.io/kate
• Killbots stable https://snapcraft.io/killbots
• Katomic stable https://snapcraft.io/katomic
• Kbackup –edge ( Needs auto-connect udisks2, added home plug)
• Kblackbox stable https://snapcraft.io/kblackbox
• Kblocks stable https://snapcraft.io/kblocks
• Kbreakout stable https://snapcraft.io/kbreakout
• Kbruch stable https://snapcraft.io/kbruch
• Kcharselect stable https://snapcraft.io/kcharselect
• Kcolorchooser stable https://snapcraft.io/kcolorchooser
• Kdebugsettings –edge ( Added missing personal-files plug, will need approval)
• Kdf stable https://snapcraft.io/kdf
• KDiamond –edge ( sound issues )
• Kfind –edge ( request in for udisks2)
• Kfourinline stable https://snapcraft.io/kfourinline
• Kgeography stable https://snapcraft.io/kgeography
• Kgoldrunner stable https://snapcraft.io/kgoldrunner
• Qrca –edge ( needs snap connect qrca:camera camera until auto-connect approved, will remain in –edge until official release)
• Kclock –edge https://snapcraft.io/kclock
• Kigo –edge https://snapcraft.io/kigo
• Kimagemapeditor –edge https://snapcraft.io/kimagemapeditor
• Kspacedual –edge https://snapcraft.io/kspaceduel

WIP Snaps or MR’s made

• Juk (WIP)
• Kajongg (WIP problem with pyqt)
• Kalgebra (in store review)
• Kdevelop (WIP)
• Kdenlive (MR)
• KHangman (WIP)
• Ruqola (WIP)
• Picmi (building)
• Kubrick (WIP)
• lskat (building)
• Palapeli (MR)
• Kanagram (WIP)
• Labplot (WIP)
• Ktuberling (building)
• Ksudoku (building)
• Ksquares (MR)

Thankfully no tragedies to report this week! I thank each and everyone of you that has donated to my car fund. I still have a ways to go and could use some more help so that we can go to the funeral. https://gofund.me/033eb25d I am between contracts and work packages, so all of my work is currently for free. Thanks for your consideration.

Another very busy week getting qt6 updates in Debian, Kubuntu, and KDE snaps.

Kubuntu:

• Merkuro and Neochat SRUs have made progress.
• See Debian for the qt6 Plasma / applications work.

Debian:

• qtmpv – in NEW
• arianna – in NEW
• kamera – experimental
• libkdegames – experimental
• kdenetwork-filesharing – experimental
• xwaylandvideobridge – NEW
• futuresql – NEW
• kpat WIP
• Tokodon – Done, but needs qtmpv to pass NEW
• Gwenview – WIP needs kamera, kio-extras
• kio-extras – Blocked on kdsoap in which the maintainer is not responding to bug reports or emails. Will likely fork in Kubuntu as our freeze quickly approaches.

KDE Snaps:

Updated QT to 6.7.2 which required a rebuild of all our snaps. Also found an issue with mismatched ffmpeg libraries, we have to bundle them for now until versioning issues are resolved.

Made new theme snaps for KDE breeze: gtk-theme-breeze, icon-theme-breeze so if you use the plasma theme breeze please install these and run

for PLUG in $(snap connections | grep gtk-common-themes:icon-themes | awk '{print $2}'); do sudo snap connect ${PLUG} icon-theme-breeze:icon-themes; done for PLUG in $(snap connections | grep gtk-common-themes:gtk-3-themes | awk '{print $2}'); do sudo snap connect ${PLUG} gtk-theme-breeze:gtk-3-themes; done for PLUG in $(snap connections | grep gtk-common-themes:gtk-2-themes | awk '{print $2}'); do sudo snap connect ${PLUG} gtk-theme-breeze:gtk-2-themes; done

This should resolve most theming issues. We are still waiting for kdeglobals to be merged in snapd to fix colorscheme issues, it is set for next release. I am still working on qt6 themes and working out how to implement them in snaps as they are more complex than gtk themes with shared libraries and file structures.

Please note: Please help test the –edge snaps so I can promote them to stable.

• Elisa – stable https://snapcraft.io/elisa
• Okular – stable https://snapcraft.io/okular
• Konsole ( please note this is a confined terminal for the ‘project’ and not very useful except to ssh to the host system ) – stable
• Kwrite – stable https://snapcraft.io/kwrite
• Gwenview – stable https://snapcraft.io/gwenview
• Kate ( –classic –edge ) https://snapcraft.io/kate
• Gcompris –edge https://snapcraft.io/gcompris
• Alligator stable https://snapcraft.io/alligator
• Angelfish –edge https://snapcraft.io/angelfish ( Crashes on first run, but runs fine after that.. looking into it)
• Arianna –edge ( Just pushed fix for required browser-support plug)
• Ark stable https://snapcraft.io/ark
• Blinken stable https://snapcraft.io/blinken
• Bomber stable https://snapcraft.io/bomber
• Bovo stable https://snapcraft.io/bovo
• Calindori stable https://snapcraft.io/calindori
• Digikam stable https://snapcraft.io/digikam
• Dragon –edge ( just pushed dbus fix )
• Falkon stable https://snapcraft.io/falkon
• Filelight stable https://snapcraft.io/filelight
• GCompris –edge https://snapcraft.io/gcompris ( Will remain in edge until official release )
• Ghostwriter –edge ( Broken, need to workout Qt webengine obscure way of handling hunspell dictionaries.)
• Granatier stable https://snapcraft.io/granatier
• Kalzium stable https://snapcraft.io/kalzium
• Kapman stable https://snapcraft.io/kapman
• Kasts –edge ( Broken, portal failure, testing some plugs)
• Kate stable –classic https://snapcraft.io/kate
• Killbots stable https://snapcraft.io/killbots
• Katomic stable https://snapcraft.io/katomic
• Kbackup –edge ( Needs auto-connect udisks2, added home plug)
• Kblackbox stable https://snapcraft.io/kblackbox
• Kblocks stable https://snapcraft.io/kblocks
• Kbreakout stable https://snapcraft.io/kbreakout
• Kbruch stable https://snapcraft.io/kbruch
• Kcharselect stable https://snapcraft.io/kcharselect
• Kcolorchooser stable https://snapcraft.io/kcolorchooser
• Kdebugsettings –edge ( Added missing personal-files plug, will need approval)
• Kdf stable https://snapcraft.io/kdf
• KDiamond –edge ( sound issues )
• Kfind –edge ( request in for udisks2)
• Kfourinline stable https://snapcraft.io/kfourinline
• Kgeography stable https://snapcraft.io/kgeography
• Kgoldrunner stable https://snapcraft.io/kgoldrunner
• Qrca –edge ( needs snap connect qrca:camera camera until auto-connect approved, will remain in –edge until official release)
• Kclock –edge https://snapcraft.io/kclock
• Kigo –edge https://snapcraft.io/kigo
• Kimagemapeditor –edge https://snapcraft.io/kimagemapeditor
• Kspacedual –edge https://snapcraft.io/kspaceduel

WIP Snaps or MR’s made

• Juk (WIP)
• Kajongg (WIP problem with pyqt)
• Kalgebra (in store review)
• Kdevelop (WIP)
• Kdenlive (MR)
• KHangman (WIP)
• Ruqola (WIP)
• Picmi (building)
• Kubrick (WIP)
• lskat (building)
• Palapeli (MR)
• Kanagram (WIP)
• Labplot (WIP)
• Ktuberling (building)
• Ksudoku (building)
• Ksquares (MR)

I like to call myself git “expert”, but I failed pretty badly few weeks ago I needed to bisect the kernel source code to figure out one bug. General git bisect process is, You have two known commits, one is good and one is bad, and you keep searching for bad commit by splitting history in two. Lets say you have following changelog (not the actual changelog I was debugging, but example):

KStars v3.7.2 is released on 2024.08.03 for Windows, MacOS & Linux. It's a bi-monthly bug-fix release with a couple of exciting features.
A few members of the KStars development team were enjoying their summer holidays over the past few weeks, but we still have a couple of exciting features in this release!
Multi-Camera SupportWolfgang Reissenberger devoted countless hours to bring this complex feature that required lots of architectural changes in Ekos. If you have double rigs (or even more), we have a new great feature: multi camera support! If you have more than one optical train on your mount, you now will be able to run capture sequences for each of your optical trains in parallel from the same KStars instance. You simply need to create an optical train for each of your telescopes on your mount, create camera tabs for each optical train, create their own sequence and let them run in parallel.


This new feature may also be used in combination with the scheduler: in this release, the scheduler itself will control the first camera. On top of that, you could configure additional cameras in the capture tab which could execute their own capture sequences. Enhancing the scheduler such that it could control multiple optical trains in parallel is already under development and will be part of one of the next releases. So stay tuned!
Focus Advisor v4
John Evans refactored the Focus Advisor to make it simpler while still offering a lot of insight on parameters tuning. It is now more accessible to new users and has added functionality to optimize values for 2 of the more difficult Focus parameters: Step Size and Backlash (or AF Overscan).



In addition, Focus Advisor contains a convenience tool to locate stars by searching the range of motion of the focuser and another convenience tool to highlight differences between current Focus parameter settings and those recommended by Focus Advisor.

After Max' introduction to abstract interpretation for the toy optimizer in the last post, I want to present a more complicated abstract domain in this post. This abstract domain reasons about the individual bits of a variable in a trace. Every bit can be either "known zero", "known one" or "unknown". The abstract domain is useful for optimizing integer operations, particularly the bitwise operations. The abstract domain follows quite closely the tristate abstract domain of the eBPF verifier in the Linux Kernel, as described by the paper Sound, Precise, and Fast Abstract Interpretation with Tristate Numbers by Harishankar Vishwanathan, Matan Shachnai, Srinivas Narayana, and Santosh Nagarakatte.

The presentation in this post will still be in the context of the toy optimizer. We'll spend a significant part of the post convincing ourselves that the abstract domain transfer functions that we're writing are really correct, using both property-based testing and automated proofs (again using Z3).

PyPy has implemented and merged a more complicated version of the same abstract domain for the "real" PyPy JIT. A more thorough explanation of that real world implementation will follow.

I'd like to thank Max Bernstein and Armin Rigo for lots of great feedback on drafts of this post. The PyPy implementation was mainly done by Nico Rittinghaus and me.

Contents:

• Motivation
• The Knownbits Abstract Domain
• Transfer Functions
• Property-based Tests with Hypothesis
• When are Transfer Functions Correct? How do we test them?
• Implementing Binary Transfer Functions
• Addition and Subtraction
• Proving correctness of the transfer functions with Z3
• Cases where this style of Z3 proof doesn't work
• Making Statements about Precision
• Using the Abstract Domain in the Toy Optimizer for Generalized Constant Folding
• Using the KnownBits Domain for Conditional Peephole Rewrites
• Conclusion

Motivation

In many programs that do bit-manipulation of integers, some of the bits of the integer variables of the program can be statically known. Here's a simple example:

x = a | 1 ... if x & 1: ... else: ...

After the assignment x = a | 1, we know that the lowest bit of x must be 1 (the other bits are unknown) and an optimizer could remove the condition x & 1 by constant-folding it to 1.

Another (more complicated) example is:

assert i & 0b111 == 0 # check that i is a multiple of 8 j = i + 16 assert j & 0b111 == 0

This kind of code could e.g. happen in a CPU emulator, where i and j are integers that represent emulated pointers, and the asserts are alignment checks. The first assert implies that the lowest three bits of i must be 0. Adding 16 to such a number produces a result where the lowest three bits are again all 0, therefore the second assert is always true. So we would like a compiler to remove the second assert.

Both of these will optimizations are doable with the help of the knownbits abstract domain that we'll discuss in the rest of the post.

The Knownbits Abstract Domain

An abstract value of the knownbits domain needs to be able to store, for every bit of an integer variable in a program, whether it is known 0, known 1, or unknown. To represent three different states, we need 2 bits, which we will call one and unknown. Here's the encoding:

one unknown knownbit 0 0 0 1 0 1 0 1 ? 1 1 illegal

The unknown bit is set if we don't know the value of the bit ("?"), the one bit is set if the bit is known to be a 1. Since two bits are enough to encode four different states, but we only need three, the combination of a set one bit and a set unknown is not allowed.

We don't just want to encode a single bit, however. Instead, we want to do this for all the bits of an integer variable. Therefore the instances of the abstract domain get two integer fields ones and unknowns, where each pair of corresponding bits encodes the knowledge about the corresponding bit of the integer variable in the program.

We can start implementing a Python class that works like this:

from dataclasses import dataclass @dataclass(eq=False) class KnownBits: ones : int unknowns : int def __post_init__(self): if isinstance(self.ones, int): assert self.is_well_formed() def is_well_formed(self): # a bit cannot be both 1 and unknown return self.ones & self.unknowns == 0 @staticmethod def from_constant(const : int): """ Construct a KnownBits corresponding to a constant, where all bits are known.""" return KnownBits(const, 0) def is_constant(self): """ Check if the KnownBits instance represents a constant. """ # it's a constant if there are no unknowns return self.unknowns == 0

We can also add some convenience properties. Sometimes it is easier to work with an integer where all the known bits are set, or one where the positions of all the known zeros have a set bit:

class KnownBits: ... @property def knowns(self): """ return an integer where the known bits are set. """ # the knowns are just the unknowns, inverted return ~self.unknowns @property def zeros(self): """ return an integer where the places that are known zeros have a bit set. """ # it's a 0 if it is known, but not 1 return self.knowns & ~self.ones

Also, for debugging and for writing tests we want a way to print the known bits in a human-readable form, and also to have a way to construct a KnownBits instance from a string. It's not important to understand the details of __str__ or from_str for the rest of the post, so I'm putting them into a fold:

KnownBits from and to string conversions class KnownBits: ... def __repr__(self): if self.is_constant(): return f"KnownBits.from_constant({self.ones})" return f"KnownBits({self.ones}, {self.unknowns})" def __str__(self): res = [] ones, unknowns = self.ones, self.unknowns # construct the string representation right to left while 1: if not ones and not unknowns: break # we leave off the leading known 0s if ones == -1 and not unknowns: # -1 has all bits set in two's complement, so the leading # bits are all 1 res.append('1') res.append("...") break if unknowns == -1: # -1 has all bits set in two's complement, so the leading bits # are all ? assert not ones res.append("?") res.append("...") break if unknowns & 1: res.append('?') elif ones & 1: res.append('1') else: res.append('0') ones >>= 1 unknowns >>= 1 if not res: res.append('0') res.reverse() return "".join(res) @staticmethod def from_str(s): """ Construct a KnownBits instance that from a string. String can start with ...1 to mean that all higher bits are 1, or ...? to mean that all higher bits are unknown. Otherwise it is assumed that the higher bits are all 0. """ ones, unknowns = 0, 0 startindex = 0 if s.startswith("...?"): unknowns = -1 startindex = 4 elif s.startswith("...1"): ones = -1 startindex = 4 for index in range(startindex, len(s)): ones <<= 1 unknowns <<= 1 c = s[index] if c == '1': ones |= 1 elif c == '?': unknowns |= 1 return KnownBits(ones, unknowns) @staticmethod def all_unknown(): """ convenience constructor for the "all bits unknown" abstract value """ return KnownBits.from_str("...?")

And here's a pytest-style unit test for str:

def test_str(): assert str(KnownBits.from_constant(0)) == '0' assert str(KnownBits.from_constant(5)) == '101' assert str(KnownBits(5, 0b10)) == '1?1' assert str(KnownBits(~0b1111, 0b10)) == '...100?0' assert str(KnownBits(1, ~0b1)) == '...?1'

An instance of KnownBits represents a set of integers, namely those that match the known bits stored in the instance. We can write a method contains that takes a concrete int value and returns True if the value matches the pattern of the known bits:

class KnownBits: ... def contains(self, value : int): """ Check whether the KnownBits instance contains the concrete integer `value`. """ # check whether value matches the bit pattern. in the places where we # know the bits, the value must agree with ones. return value & self.knowns == self.ones

and a test:

def test_contains(): k1 = KnownBits.from_str('1?1') assert k1.contains(0b111) assert k1.contains(0b101) assert not k1.contains(0b110) assert not k1.contains(0b011) k2 = KnownBits.from_str('...?1') # all odd numbers for i in range(-101, 100): assert k2.contains(i) == (i & 1) Transfer Functions

Now that we have implemented the basics of the KnownBits class, we need to start implementing the transfer functions. They are for computing what we know about the results of an operation, given the knownledge we have about the bits of the arguments.

We'll start with a simple unary operation, invert(x) (which is ~x in Python and C syntax), which flips all the bits of at integer. If we know some bits of the arguments, we can compute the corresponding bits of the result. The unknown bits remain unknown.

Here's the code:

class KnownBits: ... def abstract_invert(self): # self.zeros has bits set where the known 0s are in self return KnownBits(self.zeros, self.unknowns)

And a unit-test:

def test_invert(): k1 = KnownBits.from_str('01?01?01?') k2 = k1.abstract_invert() assert str(k2) == '...10?10?10?' k1 = KnownBits.from_str('...?') k2 = k1.abstract_invert() assert str(k2) == '...?'

Before we continue with further transfer functions, we'll think about correctness of the transfer functions and build up some test infrastructure. To test transfer functions, it's quite important to move being simple example-style unit tests. The state-space for more complicated binary transfer functions is extremely large and it's too easy to do something wrong in a corner case. Therefore we'll look at property-based-test for KnownBits next.

Property-based Tests with Hypothesis

We want to do property-based tests of KnownBits, to try make it less likely that we'll get a corner-case in the implementation wrong. We'll use Hypothesis for that.

I can't give a decent introduction to Hypothesis here, but want to give a few hints about the API. Hypothesis is a way to run unit tests with randomly generated input. It provides strategies to describe the data that the test functions expects. Hypothesis provides primitive strategies (for things like integers, strings, floats, etc) and ways to build composite strategies out of the primitive ones.

To be able to write the tests, we need to generate random KnownBits instances, and we also want an int instance that is a member of the KnownBits instance. We generate tuples of (KnownBits, int) together, to ensure this property. We'll ask Hypothesis to generate us a random concrete int as the concrete value, and then we'll also generate a second random int to use as the unknown masks (i.e. which bits of the concrete int we don't know in the KnownBits instance). Here's a function that takes two such ints and builds the tuple:

def build_knownbits_and_contained_number(concrete_value : int, unknowns : int): # to construct a valid KnownBits instance, we need to mask off the unknown # bits ones = concrete_value & ~unknowns return KnownBits(ones, unknowns), concrete_value

We can turn this function into a hypothesis strategy to generate input data using the strategies.builds function:

from hypothesis import strategies, given, settings ints = strategies.integers() random_knownbits_and_contained_number = strategies.builds( build_knownbits_and_contained_number, ints, ints )

One important special case of KnownBits are the constants, which contain only a single concrete value. We'll also generate some of those specifically, and then combine the random_knownbits_and_contained_number strategy with it:

constant_knownbits = strategies.builds( lambda value: (KnownBits.from_constant(value), value), ints ) knownbits_and_contained_number = constant_knownbits | random_knownbits_and_contained_number

Now we can write the first property-based tests, for the KnownBits.contains method:

@given(knownbits_and_contained_number) def test_contains(t): k, n = t assert k.contains(t)

The @given decorator is used to tell Hypothesis which strategy to use to generate random data for the test function. Hypothesis will run the test with a number of random examples (100 by default). If it finds an error, it will try to minimize the example needed that demonstrates the problem, to try to make it easier to understand what is going wrong. It also saves all failing cases into an example database and tries them again on subsequent runs.

This test is as much a check for whether we got the strategies right as it is for the logic in KnownBits.contains. Here's an example output of random concrete and abstract values that we are getting here:

110000011001101 ...?0???1 ...1011011 ...1011011 ...1001101110101000010010011111011 ...1001101110101000010010011111011 ...1001101110101000010010011111011 ...100110111010100001?010?1??1??11 1000001101111101001011010011111101000011000111011001011111101 1000001101111101001011010011111101000011000111011001011111101 1000001101111101001011010011111101000011000111011001011111101 1000001101111101001011010011111101000011000111????01?11?????1 1111100000010 1111100000010 1111100000010 ...?11111?00000?? 110110 110110 110110 ...?00?00????11??10 110110 ??0??0 ...100010111011111 ...?100?10111??111? ...1000100000110001 ...?000?00000??000? 110000001110 ...?0?0??000?00?0?0000000?00???0000?????00???000?0?00?01?000?0??1?? 110000001110 ??000000???0 1011011010000001110101001111000010001001011101010010010001000000010101010010001101110101111111010101010010101100110000011110000 1011011010000001110101001111000010001001011101010010010001000000010101010010001101110101111111010101010010101100110000011110000 ...1011010010010100 ...1011010010010100 ...1011111110110011 ...1011111110110011 101000011110110 101000011?10?1? 100101 ?00?0?

That looks suitably random, but we might want to bias our random numbers a little bit towards common error values like small constants, powers of two, etc. Like this:

INTEGER_WIDTH = 64 # some small integers ints_special = set(range(100)) # powers of two ints_special = ints_special.union(1 << i for i in range(INTEGER_WIDTH - 2)) # powers of two - 1 ints_special = ints_special.union((1 << i) - 1 for i in range(INTEGER_WIDTH - 2)) # negative versions of what we have so far ints_special = ints_special.union(-x for x in ints_special) # bit-flipped versions of what we have so far ints_special = ints_special.union(~x for x in ints_special) ints_special = list(ints_special) # sort them (because hypothesis simplifies towards earlier elements in the list) ints_special.sort(key=lambda element: (abs(element), element < 0)) ints = strategies.sampled_from(ints_special) | strategies.integers()

Now we get data like this:

1110 1110 ...10000000000000000001 ...10000??0??0000??00?1 1 ??0??0000??00?1 1 ? ...10101100 ...10101100 110000000011001010111011111111111111011110010001001100110001011 ...?0?101? 110000000011001010111011111111111111011110010001001100110001011 ??00000000??00?0?0???0??????????????0????00?000?00??00??000?0?? ...1011111111111111111111111111 ...?11?11?? ...1011111111111111111111111111 ...?0?????????????????????????? 0 ...?0?????????????????????????? 101101 101101 111111111111111111111111111111111111111111111 111111111111111111111111111111111111111111111 10111 10111 ...101100 ...1?111011?0 101000 ?001010?0 101000 ?0?000 110010 110010 ...100111 ...100111 1111011010010 1111011010010 ...1000000000000000000000000000000000000 ...1000000000000000000000000000000000000

We can also write a test that checks that the somewhat tricky logic in __str__ and from_str is correct, by making sure that the two functions round-trip (ie converting a KnownBits to a string and then back to a KnownBits instance produces the same abstract value).

@given(knownbits_and_contained_number) def test_hypothesis_str_roundtrips(t1): k1, n1 = t1 s = str(k1) k2 = KnownBits.from_str(s) assert k1.ones == k2.ones assert k1.unknowns == k2.unknowns

Now let's actually apply this infrastructure to test abstract_invert.

When are Transfer Functions Correct? How do we test them?

Abstract values, i.e. instances of KnownBits represent sets of concrete values. We want the transfer functions to compute overapproximations of the concrete values. So if we have an arbitrary abstract value k, with a concrete number n that is a member of the abstract values (i.e. k.contains(n) == True) then the result of the concrete operation op(n) must be a member of the result of the abstract operation k.abstract_op() (i.e. k.abstract_op().contains(op(n)) == True).

Checking the correctness/overapproximation property is a good match for hypothesis. Here's what the test for abstract_invert looks like:

@given(knownbits_and_contained_number) def test_hypothesis_invert(t): k1, n1 = t1 n2 = ~n1 # compute the real result k2 = k1.abstract_invert() # compute the abstract result assert k2.contains(n2) # the abstract result must contain the real result

This is the only condition needed for abstract_invert to be correct. If abstract_invert fulfils this property for every combination of abstract and concrete value then abstract_invert is correct. Note however, that this test does not actually check whether abstract_invert gives us precise results. A correct (but imprecise) implementation of abstract_invert would simply return a completely unknown result, regardless of what is known about the input KnownBits.

The "proper" CS term for this notion of correctness is called soundness. The correctness condition on the transfer functions is called a Galois connection. I won't go into any mathematical/technical details here, but wanted to at least mention the terms. I found Martin Kellogg's slides to be quite an approachable introduction to the Galois connection and how to show soundness.

Implementing Binary Transfer Functions

Now we have infrastructure in place for testing transfer functions with random inputs. With that we can start thinking about the more complicated case, that of binary operations. Let's start with the simpler ones, and and or. For and, we can know a 0 bit in the result if either of the input bits are known 0; or we can know a 1 bit in the result if both input bits are known 1. Otherwise the resulting bit is unknown. Let's look at all the combinations:

and input1: 000111??? input2: 01?01?01? result: 00001?0?? class KnownBits: ... def abstract_and(self, other): ones = self.ones & other.ones # known ones knowns = self.zeros | other.zeros | ones return KnownBits(ones, ~knowns)

Here's an example unit-test and a property-based test for and:

def test_and(): # test all combinations of 0, 1, ? in one example k1 = KnownBits.from_str('01?01?01?') k2 = KnownBits.from_str('000111???') res = k1.abstract_and(k2) # should be: 0...00001?0?? assert str(res) == "1?0??" @given(knownbits_and_contained_number, knownbits_and_contained_number) def test_hypothesis_and(t1, t2): k1, n1 = t1 k2, n2 = t2 k3 = k1.abstract_and(k2) n3 = n1 & n2 assert k3.contains(n3)

To implement or is pretty similar. The result is known 1 where either of the inputs is 1. The result is known 0 where both inputs are known 0, and ? otherwise.

or input1: 000111??? input2: 01?01?01? result: 01?111?1? class KnownBits: ... def abstract_or(self, other): ones = self.ones | other.ones zeros = self.zeros & other.zeros knowns = ones | zeros return KnownBits(ones, ~knowns)

Here's an example unit-test and a property-based test for or:

def test_or(): k1 = KnownBits.from_str('01?01?01?') k2 = KnownBits.from_str('000111???') res = k1.abstract_or(k2) # should be: 0...01?111?1? assert str(res) == "1?111?1?" @given(knownbits_and_contained_number, knownbits_and_contained_number) def test_hypothesis_or(t1, t2): k1, n1 = t1 k2, n2 = t2 k3 = k1.abstract_or(k2) n3 = n1 | n2 assert k3.contains(n3)

Implementing support for abstract_xor is relatively simple, and left as an exercise :-).

Addition and Subtraction

invert, and, and or are relatively simple transfer functions to write, because they compose over the individual bits of the integers. The arithmetic functions add and sub are significantly harder, because of carries and borrows. Coming up with the formulas for them and gaining an intuitive understanding is quite tricky and involves carefully going through a few examples with pen and paper. When implementing this in PyPy, Nico and I didn't come up with the implementation ourselves, but instead took them from the Tristate Numbers paper. Here's the code, with example tests and hypothesis tests:

class KnownBits: ... def abstract_add(self, other): sum_ones = self.ones + other.ones sum_unknowns = self.unknowns + other.unknowns all_carries = sum_ones + sum_unknowns ones_carries = all_carries ^ sum_ones unknowns = self.unknowns | other.unknowns | ones_carries ones = sum_ones & ~unknowns return KnownBits(ones, unknowns) def abstract_sub(self, other): diff_ones = self.ones - other.ones val_borrows = (diff_ones + self.unknowns) ^ (diff_ones - other.unknowns) unknowns = self.unknowns | other.unknowns | val_borrows ones = diff_ones & ~unknowns return KnownBits(ones, unknowns) def test_add(): k1 = KnownBits.from_str('0?10?10?10') k2 = KnownBits.from_str('0???111000') res = k1.abstract_add(k2) assert str(res) == "?????01?10" def test_sub(): k1 = KnownBits.from_str('0?10?10?10') k2 = KnownBits.from_str('0???111000') res = k1.abstract_sub(k2) assert str(res) == "...?11?10" k1 = KnownBits.from_str( '...1?10?10?10') k2 = KnownBits.from_str('...10000???111000') res = k1.abstract_sub(k2) assert str(res) == "111?????11?10" @given(knownbits_and_contained_number, knownbits_and_contained_number) def test_hypothesis_add(t1, t2): k1, n1 = t1 k2, n2 = t2 k3 = k1.abstract_add(k2) n3 = n1 + n2 assert k3.contains(n3) @given(knownbits_and_contained_number, knownbits_and_contained_number) def test_hypothesis_sub(t1, t2): k1, n1 = t1 k2, n2 = t2 k3 = k1.abstract_sub(k2) n3 = n1 - n2 assert k3.contains(n3)

Now we are in a pretty good situation, and have implemented abstract versions for a bunch of important arithmetic and binary functions. What's also surprising is that the implementation of all of the transfer functions is quite efficient. We didn't have to write loops over the individual bits at all, instead we found closed form expressions using primitive operations on the underlying integers ones and unknowns. This means that computing the results of abstract operations is quite efficient, which is important when using the abstract domain in the context of a JIT compiler.

Proving correctness of the transfer functions with Z3

As one can probably tell from my recent posts, I've been thinking about compiler correctness a lot. Getting the transfer functions absolutely correct is really crucial, because a bug in them would lead to miscompilation of Python code when the abstract domain is added to the JIT. While the randomized tests are great, it's still entirely possible for them to miss bugs. The state space for the arguments of a binary transfer function is 3**64 * 3**64, and if only a small part of that contains wrong behaviour it would be really unlikely for us to find it with random tests by chance. Therefore I was reluctant to merge the PyPy branch that contained the new abstract domain for a long time.

To increase our confidence in the correctness of the transfer functions further, we can use Z3 to prove their correctness, which gives us much stronger guarantees (not 100%, obviously). In this subsection I will show how to do that.

Here's an attempt to do this manually in the Python repl:

>>>> import z3 >>>> solver = z3.Solver() >>>> # like last blog post, proof by failing to find counterexamples >>>> def prove(cond): assert solver.check(z3.Not(cond)) == z3.unsat >>>> >>>> # let's set up a z3 bitvector variable for an abitrary concrete value >>>> n1 = z3.BitVec('concrete_value', 64) >>>> n1 concrete_value >>>> # due to operator overloading we can manipulate z3 formulas >>>> n2 = ~n1 >>>> n2 ~concrete_value >>>> >>>> # now z3 bitvector variables for the ones and zeros fields >>>> ones = z3.BitVec('abstract_ones', 64) >>>> unknowns = z3.BitVec('abstract_unknowns', 64) >>>> # we construct a KnownBits instance with the z3 variables >>>> k1 = KnownBits(ones, unknowns) >>>> # due to operator overloading we can call the methods on k1: >>>> k2 = k1.abstract_invert() >>>> k2.ones ~abstract_unknowns & ~abstract_ones >>>> k2.unknowns abstract_unknowns >>>> # here's the correctness condition that we want to prove: >>>> k2.contains(n2) ~concrete_value & ~abstract_unknowns == ~abstract_unknowns & ~abstract_ones >>>> # let's try >>>> prove(k2.contains(n2)) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<stdin>", line 1, in prove AssertionError >>>> # it doesn't work! let's look at the counterexample to see why: >>>> solver.model() [abstract_unknowns = 0, abstract_ones = 0, concrete_value = 1] >>>> # we can build a KnownBits instance with the values in the >>>> # counterexample: >>>> ~1 # concrete result -2 >>>> counter_example_k1 = KnownBits(0, 0) >>>> counter_example_k1 KnownBits.from_constant(0) >>>> counter_example_k2 = counter_example_k1.abstract_invert() >>>> counter_example_k2 KnownBits.from_constant(-1) >>>> # let's check the failing condition >>>> counter_example_k2.contains(~1) False

What is the problem here? We didn't tell Z3 that n1 was supposed to be a member of k1. We can add this as a precondition to the solver, and then the prove works:

>>>> solver.add(k1.contains(n1)) >>>> prove(k2.contains(n2)) # works!

This is super cool! It's really a proof about the actual implementation, because we call the implementation methods directly, and due to the operator overloading that Z3 does we can be sure that we are actually checking a formula that corresponds to the Python code. This eliminates one source of errors in formal methods.

Doing the proof manually on the Python REPL is kind of annoying though, and we also would like to make sure that the proofs are re-done when we change the code. What we would really like to do is writing the proofs as a unit-test that we can run while developing and in CI. Doing this is possible, and the unit tests that really perform proofs look pleasingly similar to the Hypothesis-based ones.

First we need to set up a bit of infrastructure:

INTEGER_WIDTH = 64 def BitVec(name): return z3.BitVec(name, INTEGER_WIDTH) def BitVecVal(val): return z3.BitVecVal(val, INTEGER_WIDTH) def z3_setup_variables(): # instantiate a solver solver = z3.Solver() # a Z3 variable for the first concrete value n1 = BitVec("n1") # a KnownBits instances that uses Z3 variables as its ones and unknowns, # representing the first abstract value k1 = KnownBits(BitVec("n1_ones"), BitVec("n1_unkowns")) # add the precondition to the solver that the concrete value n1 must be a # member of the abstract value k1 solver.add(k1.contains(n1)) # a Z3 variable for the second concrete value n2 = BitVec("n2") # a KnownBits instances for the second abstract value k2 = KnownBits(BitVec("n2_ones"), BitVec("n2_unkowns")) # add the precondition linking n2 and k2 to the solver solver.add(k2.contains(n2)) return solver, k1, n1, k2, n2 def prove(cond, solver): z3res = solver.check(z3.Not(cond)) if z3res != z3.unsat: assert z3res == z3.sat # can't be timeout, we set no timeout # make the model with the counterexample global, to make inspecting the # bug easier when running pytest --pdb global model model = solver.model() print(f"n1={model.eval(n1)}, n2={model.eval(n2)}") counter_example_k1 = KnownBits(model.eval(k1.ones).as_signed_long(), model.eval(k1.unknowns).as_signed_long()) counter_example_k2 = KnownBits(model.eval(k2.ones).as_signed_long(), model.eval(k2.unknowns).as_signed_long()) print(f"k1={counter_example_k1}, k2={counter_example_k2}") print(f"but {cond=} evaluates to {model.eval(cond)}") raise ValueError(solver.model())

And then we can write proof-unit-tests like this:

def test_z3_abstract_invert(): solver, k1, n1, _, _ = z3_setup_variables() k2 = k1.abstract_invert() n2 = ~n1 prove(k2.contains(n2), solver) def test_z3_abstract_and(): solver, k1, n1, k2, n2 = z3_setup_variables() k3 = k1.abstract_and(k2) n3 = n1 & n2 prove(k3.contains(n3), solver) def test_z3_abstract_or(): solver, k1, n1, k2, n2 = z3_setup_variables() k3 = k1.abstract_or(k2) n3 = n1 | n2 prove(k3.contains(n3), solver) def test_z3_abstract_add(): solver, k1, n1, k2, n2 = z3_setup_variables() k3 = k1.abstract_add(k2) n3 = n1 + n2 prove(k3.contains(n3), solver) def test_z3_abstract_sub(): solver, k1, n1, k2, n2 = z3_setup_variables() k3 = k1.abstract_sub(k2) n3 = n1 - n2 prove(k3.contains(n3), solver)

It's possible to write a bit more Python-metaprogramming-magic and unify the Hypothesis and Z3 tests into the same test definition.1

Cases where this style of Z3 proof doesn't work

Unfortunately the approach described in the previous section only works for a very small number of cases. It breaks down as soon as the KnownBits methods that we're calling contain any if conditions (including hidden ones like the short-circuiting and and or in Python). Let's look at an example and implement abstract_eq. eq is supposed to be an operation that compares two integers and returns 0 or 1 if they are different or equal, respectively. Implementing this in knownbits looks like this (with example and hypothesis tests):

class KnownBits: ... def abstract_eq(self, other): # the result is a 0, 1, or ? # if they are both the same constant, they must be equal if self.is_constant() and other.is_constant() and self.ones == other.ones: return KnownBits.from_constant(1) # check whether we have known disagreeing bits, then we know the result # is 0 if self._disagrees(other): return KnownBits.from_constant(0) return KnownBits(0, 1) # an unknown boolean def _disagrees(self, other): # check whether the bits disagree in any place where both are known both_known = self.knowns & other.knowns return self.ones & both_known != other.ones & both_known def test_eq(): k1 = KnownBits.from_str('...?') k2 = KnownBits.from_str('...?') assert str(k1.abstract_eq(k2)) == '?' k1 = KnownBits.from_constant(10) assert str(k1.abstract_eq(k1)) == '1' k1 = KnownBits.from_constant(10) k2 = KnownBits.from_constant(20) assert str(k1.abstract_eq(k2)) == '0' @given(knownbits_and_contained_number, knownbits_and_contained_number) def test_hypothesis_eq(t1, t2): k1, n1 = t1 k2, n2 = t2 k3 = k1.abstract_eq(k2) assert k3.contains(int(n1 == n2))

Trying to do the proof in the same style as before breaks:

>>>> k3 = k1.abstract_eq(k2) Traceback (most recent call last): File "<stdin>", line 1, in <module> File "knownbits.py", line 246, in abstract_eq if self._disagrees(other): File "venv/site-packages/z3/z3.py", line 381, in __bool__ raise Z3Exception("Symbolic expressions cannot be cast to concrete Boolean values.") z3.z3types.Z3Exception: Symbolic expressions cannot be cast to concrete Boolean values.

We cannot call abstract_eq on a KnownBits with Z3 variables as fields, because once we hit an if statement, the whole approach of relying on the operator overloading breaks down. Z3 doesn't actually parse the Python code or anything advanced like that, we rather build an expression only by running the code and letting the Z3 formulas build up.

To still prove the correctness of abstract_eq we need to manually transform the control flow logic of the function into a Z3 formula that uses the z3.If expression, using a small helper function:

def z3_cond(b, trueval=1, falseval=0): return z3.If(b, BitVecVal(trueval), BitVecVal(falseval)) def z3_abstract_eq(k1, k2): # follow the *logic* of abstract_eq, we can't call it due to the ifs in it case1cond = z3.And(k1.is_constant(), k2.is_constant(), k1.ones == k2.ones) case2cond = k1._disagrees(k2) # ones is 1 in the first case, 0 otherwise ones = z3_cond(case1cond, 1, 0) # in the first two cases, unknowns is 0, 1 otherwise unknowns = z3_cond(z3.Or(case1cond, case2cond), 0, 1) return KnownBits(ones, unknowns) def test_z3_abstract_eq_logic(): solver, k1, n1, k2, n2 = z3_setup_variables() n3 = z3_cond(n1 == n2) # concrete result k3 = z3_abstract_eq(k1, k2) prove(k3.contains(n3), solver)

This proof works. It is a lot less satisfying than the previous ones though, because we could have done an error in the manual transcription from Python code to Z3 formulas (there are possibly more heavy-handed approaches where we do this transformation more automatically using e.g. the ast module to analyze the source code, but that's a much more complicated researchy project). To lessen this problem somewhat we can factor out the parts of the logic that don't have any conditions into small helper methods (like _disagrees in this example) and use them in the manual conversion of the code to Z3 formulas.2

The final condition that Z3 checks, btw, is this one:

If(n1 == n2, 1, 0) & ~If(Or(And(n1_unkowns == 0, n2_unkowns == 0, n1_ones == n2_ones), n1_ones & ~n1_unkowns & ~n2_unkowns != n2_ones & ~n1_unkowns & ~n2_unkowns), 0, 1) == If(And(n1_unkowns == 0, n2_unkowns == 0, n1_ones == n2_ones), 1, 0) Making Statements about Precision

So far we have only used Z3 to prove statements about correctness, i.e. that our abstract operations overapproximate what can happen with concrete values. While proving this property is essential if we want to avoid miscompilation, correctness alone is not a very strong constraint on the implementation of our abstract transfer functions. We could simply return Knownbits.unknowns() for every abstract_* method and the resulting overapproximation would be correct, but useless in practice.

It's much harder to make statements about whether the transfer functions are maximally precise. There are two aspects of precision I want to discuss in this section, however.

The first aspect is that we would really like it if the transfer functions compute the maximally precise results for singleton sets. If all abstract arguments of an operations are constants, i.e. contain only a single concrete element, then we know that the resulting set also has only a single element. We can prove that all our transfer functions have this property:

def test_z3_prove_constant_folding(): solver, k1, n1, k2, n2 = z3_setup_variables() k3 = k1.abstract_invert() prove(z3.Implies(k1.is_constant(), k3.is_constant()), solver) k3 = k1.abstract_and(k2) prove(z3.Implies(z3.And(k1.is_constant(), k2.is_constant()), k3.is_constant()), solver) k3 = k1.abstract_or(k2) prove(z3.Implies(z3.And(k1.is_constant(), k2.is_constant()), k3.is_constant()), solver) k3 = k1.abstract_sub(k2) prove(z3.Implies(z3.And(k1.is_constant(), k2.is_constant()), k3.is_constant()), solver) k3 = z3_abstract_eq(k1, k2) prove(z3.Implies(z3.And(k1.is_constant(), k2.is_constant()), k3.is_constant()), solver)

Proving with Z3 that the transfer functions are maximally precise for non-constant arguments seems to be relatively hard. I tried a few completely rigorous approaches and failed. The paper Sound, Precise, and Fast Abstract Interpretation with Tristate Numbers contains an optimality proof for the transfer functions of addition and subtraction, so we can be certain that they are as precise as is possible.

I still want to show an approach for trying to find concrete examples of abstract values that are less precise than they could be, using a combination of Hypothesis and Z3. The idea is to use hypothesis to pick random abstract values. Then we compute the abstract result using our transfer function. Afterwards we can ask Z3 to find us an abstract result that is better than the one our transfer function produced. If Z3 finds a better abstract result, we have a concrete example of imprecision for our transfer function. Those tests aren't strict proofs, because they rely on generating random abstract values, but they can still be valuable (not for the transfer functions in this blog post, which are all optimal).

Here is what the code looks like (this is a little bit bonus content, I'll not explain the details and can only hope that the comments are somewhat helpful):

@given(random_knownbits_and_contained_number, random_knownbits_and_contained_number) @settings(deadline=None) def test_check_precision(t1, t2): k1, n1 = t1 k2, n2 = t2 # apply transfer function k3 = k1.abstract_add(k2) example_res = n1 + n2 # try to find a better version of k3 with Z3 solver = z3.Solver() solver.set("timeout", 8000) var1 = BitVec('v1') var2 = BitVec('v2') ones = BitVec('ones') unknowns = BitVec('unknowns') better_k3 = KnownBits(ones, unknowns) print(k1, k2, k3) # we're trying to find an example for a better k3, so we use check, without # negation: res = solver.check(z3.And( # better_k3 should be a valid knownbits instance better_k3.is_well_formed(), # it should be better than k3, ie there are known bits in better_k3 # that we don't have in k3 better_k3.knowns & ~k3.knowns != 0, # now encode the correctness condition for better_k3 with a ForAll: # for all concrete values var1 and var2, it must hold that if # var1 is in k1 and var2 is in k2 it follows that var1 + var2 is in # better_k3 z3.ForAll( [var1, var2], z3.Implies( z3.And(k1.contains(var1), k2.contains(var2)), better_k3.contains(var1 + var2))))) # if this query is satisfiable, we have found a better result for the # abstract_add if res == z3.sat: model = solver.model() rk3 = KnownBits(model.eval(ones).as_signed_long(), model.eval(unknowns).as_signed_long()) print("better", rk3) assert 0 if res == z3.unknown: print("timeout")

It does not actually fail for abstract_add (nor the other abstract functions). To see the test failing we can add some imprecision to the implementation of abstract_add to see Hypothesis and Z3 find examples of values that are not optimally precise (for example by setting some bits of unknowns in the implementation of abstract_add unconditionally).

Using the Abstract Domain in the Toy Optimizer for Generalized Constant Folding

Now after all this work we can finally actually use the knownbits abstract domain in the toy optimizer. The code for this follows Max' intro post about abstract interpretation quite closely.

For completeness sake, in the fold there's the basic infrastructure classes that make up the IR again (they are identical or at least extremely close to the previous toy posts).

toy infrastructure class Value: def find(self): raise NotImplementedError("abstract") @dataclass(eq=False) class Operation(Value): name : str args : list[Value] forwarded : Optional[Value] = None def find(self) -> Value: op = self while isinstance(op, Operation): next = op.forwarded if next is None: return op op = next return op def arg(self, index): return self.args[index].find() def make_equal_to(self, value : Value): self.find().forwarded = value @dataclass(eq=False) class Constant(Value): value : object def find(self): return self class Block(list): def __getattr__(self, opname): def wraparg(arg): if not isinstance(arg, Value): arg = Constant(arg) return arg def make_op(*args): op = Operation(opname, [wraparg(arg) for arg in args]) self.append(op) return op return make_op def bb_to_str(l : Block, varprefix : str = "var"): def arg_to_str(arg : Value): if isinstance(arg, Constant): return str(arg.value) else: return varnames[arg] varnames = {} res = [] for index, op in enumerate(l): # give the operation a name used while # printing: var = f"{varprefix}{index}" varnames[op] = var arguments = ", ".join( arg_to_str(op.arg(i)) for i in range(len(op.args)) ) strop = f"{var} = {op.name}({arguments})" res.append(strop) return "\n".join(res)

Now we can write some first tests, the first one simply checking constant folding:

def test_constfold_two_ops(): bb = Block() var0 = bb.getarg(0) var1 = bb.int_add(5, 4) var2 = bb.int_add(var1, 10) var3 = bb.int_add(var2, var0) opt_bb = simplify(bb) assert bb_to_str(opt_bb, "optvar") == """\ optvar0 = getarg(0) optvar1 = int_add(19, optvar0)"""

Calling the transfer functions on constant KnownBits produces a constant results, as we have seen. Therefore "regular" constant folding should hopefully be achieved by optimizing with the KnownBits abstract domain too.

The next two tests are slightly more complicated and can't be optimized by regular constant-folding. They follow the motivating examples from the start of this blog post, a hundred years ago:

def test_constfold_via_knownbits(): bb = Block() var0 = bb.getarg(0) var1 = bb.int_or(var0, 1) var2 = bb.int_and(var1, 1) var3 = bb.dummy(var2) opt_bb = simplify(bb) assert bb_to_str(opt_bb, "optvar") == """\ optvar0 = getarg(0) optvar1 = int_or(optvar0, 1) optvar2 = dummy(1)""" def test_constfold_alignment_check(): bb = Block() var0 = bb.getarg(0) var1 = bb.int_invert(0b111) # mask off the lowest three bits, thus var2 is aligned var2 = bb.int_and(var0, var1) # add 16 to aligned quantity var3 = bb.int_add(var2, 16) # check alignment of result var4 = bb.int_and(var3, 0b111) var5 = bb.int_eq(var4, 0) # var5 should be const-folded to 1 var6 = bb.dummy(var5) opt_bb = simplify(bb) assert bb_to_str(opt_bb, "optvar") == """\ optvar0 = getarg(0) optvar1 = int_and(optvar0, -8) optvar2 = int_add(optvar1, 16) optvar3 = dummy(1)"""

Here is simplify to make these tests pass:

def unknown_transfer_functions(*abstract_args): return KnownBits.all_unknown() def simplify(bb: Block) -> Block: abstract_values = {} # dict mapping Operation to KnownBits def knownbits_of(val : Value): if isinstance(val, Constant): return KnownBits.from_constant(val.value) return abstract_values[val] opt_bb = Block() for op in bb: # apply the transfer function on the abstract arguments name_without_prefix = op.name.removeprefix("int_") method_name = f"abstract_{name_without_prefix}" transfer_function = getattr(KnownBits, method_name, unknown_transfer_functions) abstract_args = [knownbits_of(arg.find()) for arg in op.args] abstract_res = abstract_values[op] = transfer_function(*abstract_args) # if the result is a constant, we optimize the operation away and make # it equal to the constant result if abstract_res.is_constant(): op.make_equal_to(Constant(abstract_res.ones)) continue # otherwise emit the op opt_bb.append(op) return opt_bb

The code follows the approach from the previous blog post very closely. The only difference is that we apply the transfer function first, to be able to detect whether the abstract domain can tell us that the result has to always be a constant. This code makes all three tests pass.

Using the KnownBits Domain for Conditional Peephole Rewrites

So far we are only using the KnownBits domain to find out that certain operations have to produce a constant. We can also use the KnownBits domain to check whether certain operation rewrites are correct. Let's use one of the examples from the Mining JIT traces for missing optimizations with Z3 post, where Z3 found the inefficiency (x << 4) & -0xf == x << 4 in PyPy JIT traces. We don't have shift operations, but we want to generalize this optimization anyway. The general form of this rewrite is that under some circumstances x & y == x, and we can use the KnownBits domain to detect situations where this must be true.

To understand when x & y == x is true, we can think about individual pairs of bits a and b. If a == 0, then a & b == 0 & b == 0 == a. If b == 1 then a & b == a & 1 == a. So if either a == 0 or b == 1 is true, a & b == a follows. And if either of these conditions is true for all the bits of x and y, we can know that x & y == x.

We can write a method on KnownBits to check for this condition:

class KnownBits: ... def is_and_identity(self, other): """ Return True if n1 & n2 == n1 for any n1 in self and n2 in other. (or, equivalently, return True if n1 | n2 == n2)""" return self.zeros | other.ones == -1

Since my reasoning about this feels ripe for errors, let's check that our understanding is correct with Z3:

def test_prove_is_and_identity(): solver, k1, n1, k2, n2 = z3_setup_variables() prove(z3.Implies(k1.is_and_identity(k2), n1 & n2 == n1), solver)

Now let's use this in the toy optimizer. Here are two tests for this rewrite:

def test_remove_redundant_and(): bb = Block() var0 = bb.getarg(0) var1 = bb.int_invert(0b1111) # mask off the lowest four bits var2 = bb.int_and(var0, var1) # applying the same mask is not redundant var3 = bb.int_and(var2, var1) var4 = bb.dummy(var3) opt_bb = simplify(bb) assert bb_to_str(opt_bb, "optvar") == """\ optvar0 = getarg(0) optvar1 = int_and(optvar0, -16) optvar2 = dummy(optvar1)""" def test_remove_redundant_and_more_complex(): bb = Block() var0 = bb.getarg(0) var1 = bb.getarg(1) # var2 has bit pattern ???? var2 = bb.int_and(var0, 0b1111) # var3 has bit pattern ...?1111 var3 = bb.int_or(var1, 0b1111) # var4 is just var2 var4 = bb.int_and(var2, var3) var5 = bb.dummy(var4) opt_bb = simplify(bb) assert bb_to_str(opt_bb, "optvar") == """\ optvar0 = getarg(0) optvar1 = getarg(1) optvar2 = int_and(optvar0, 15) optvar3 = int_or(optvar1, 15) optvar4 = dummy(optvar2)"""

The first test could also be made to pass by implementing a reassociation optimization that turns (x & c1) & c2 into x & (c1 & c2) and then constant-folds the second and. But here we want to use KnownBits and conditionally rewrite int_and to its first argument. So to make the tests pass, we can change simplify like this:

def simplify(bb: Block) -> Block: abstract_values = {} # dict mapping Operation to KnownBits def knownbits_of(val : Value): ... opt_bb = Block() for op in bb: # apply the transfer function on the abstract arguments name_without_prefix = op.name.removeprefix("int_") method_name = f"abstract_{name_without_prefix}" transfer_function = getattr(KnownBits, method_name, unknown_transfer_functions) abstract_args = [knownbits_of(arg.find()) for arg in op.args] abstract_res = abstract_values[op] = transfer_function(*abstract_args) # if the result is a constant, we optimize the operation away and make # it equal to the constant result if abstract_res.is_constant(): op.make_equal_to(Constant(abstract_res.ones)) continue # <<<< new code # conditionally rewrite int_and(x, y) to x if op.name == "int_and": k1, k2 = abstract_args if k1.is_and_identity(k2): op.make_equal_to(op.arg(0)) continue # >>>> end changes opt_bb.append(op) return opt_bb

And with that, the new tests pass as well. A real implementation would also check the other argument order, but we leave that out for the sake of brevity.

This rewrite also generalizes the rewrites int_and(0, x) -> 0 and int_and(-1, x) -> x, let's add a test for those:

def test_remove_and_simple(): bb = Block() var0 = bb.getarg(0) var1 = bb.getarg(1) var2 = bb.int_and(0, var0) # == 0 var3 = bb.int_invert(var2) # == -1 var4 = bb.int_and(var1, var3) # == var1 var5 = bb.dummy(var4) opt_bb = simplify(bb) assert bb_to_str(opt_bb, "optvar") == """\ optvar0 = getarg(0) optvar1 = getarg(1) optvar2 = dummy(optvar1)"""

This test just passes. And that's it for this post!

Conclusion

In this post we've seen the implementation, testing and proofs about a 'known bits' abstract domain, as well as its use in the toy optimizer to generalize constant folding, and to implement conditional peephole rewrites.

In the next posts I'll write about the real implementation of a knownbits domain in PyPy's JIT, its combination with the existing interval abstract domain, how to deal with gaining informations from conditions in the program, and some lose ends.

Sources:

• Known bits in LLVM
• Tristate numbers for known bits in Linux eBPF
• Sound, Precise, and Fast Abstract Interpretation with Tristate Numbers
• Verifying the Verifier: eBPF Range Analysis Verification
• Bit-Twiddling: Addition with Unknown Bits is a super readable blog post by Dougall J. I've taken the ones and unknowns naming from this post, which I find significantly clearer than value and mask, which the Linux kernel uses.
• Bits, Math and Performance(?), a fantastic blog by Harold Aptroot. There are a lot of relevant posts about known bits, range analysis etc. Harold is also the author of Haroldbot, a website that can be used for bitvector calculations, and also checks bitvector identities.
• Sharpening Constraint Programming approaches for Bit-Vector Theory
• Deriving Abstract Transfer Functions for Analyzing Embedded Software
• Synthesizing Abstract Transformers

1. There's a subtletly about the Z3 proofs that I'm sort of glossing over here. Python integers are of arbitrary width, and the KnownBits code is actually carefully written to work for integers of any size. This property is tested by the Hypothesis tests, which don't limit the sizes of the generated random integers. However, the Z3 proofs only check bitvectors of a fixed bitwidth of 64. There are various ways to deal with this situation. For most "real" compilers, the bitwidth of integers would be fixed anyway. Then the components ones and unknowns of the KnownBits class would use the number of bits the corresponding integer variable has, and the Z3 proofs would use the same width. This is what we do in the PyPy JIT. ↩

2. The less close connection between implementation and proof for abstract_eq is one of the reasons why it makes sense to do unit-testing in addition to proofs. For a more detailed explanation of why both tests and proofs are good to have, see Jeremy Siek's blog post, as well as the Knuth quote. ↩

Focus

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Issues

• Unnecessary deps in zlint

Administration

• Debian IRC: add an entry message redirecting folks from #debian-ci to #debci

Communication

• Respond to queries from folks on Debian and other IRC channels.

Sponsors

All work was done on a volunteer basis.

Early Bird Tickets Are Almost Gone! Published Date Friday, August 2nd, 2024 - 09:33 pm bcross Fri, 08/02/2024 - 21:33 Twin Cities Drupal Camp 2024 Early Bird Tickets Are Almost Sold Out! 

Now is your last chance to save before regular pricing takes affect. Twin Cities Drupal Camp is coming on September 12-13, 2024, at the University of Minnesota campus in Minneapolis.

Registration includes access to all sessions and events, including the keynote and the unconference. Meals and social events are also part of the deal, free of additional cost. 

This Early Bird rate won’t last forever, so do it now.

Register today! 

Posted In Drupal Planet

The Cypress front-end testing infrastructure clean-up landed on Monday, so this week we’ve already been seeing increased velocity! Last week, we landed the initial implementation of the primary menu, this week it was improved by Harumi “hooroomoo” Jang:

XB’s primary menu now closes after dropping a dragged component onto the canvas. Issue #3458617, image by Harumi.

We saw a huge leap forward this week, thanks to Ben “bnjmnm” Mullins! He’s taking an unchanged PHP-based form definition and is rendering it using React and TSX! :O
Why? To prove we can render existing (core/contrib/custom) Field Widgets, because a goal for XB is to keep existing functionality working. Here’s what that looks like:

A React+TSX-rendered Drupal form to edit the props of the selected component. Incredibly ugly, because no CSS/JS is loaded yet, but mostly because the original Drupal form also is incredibly ugly — you’re seeing part of TwoTerribleTextAreasWidget (written by me to get us started)! Expect significant improvements in the coming weeks.

That screenshot does not do Ben’s monumental work justice. We’re of course already planning to improve that, starting with … no longer using TwoTerribleTextAreasWidget: #3461422: Evolve component instance edit form to become simpler: generate a Field Widget directly. Expect future screenshots and GIFs to be far more convincing :)

This leap also made it became critically important that the Cypress end-to-end tests (tests/src/Cypress/cypress/e2e/xb-general.cy.js) would actually test both the client and server, with the client actually talking to the server. (Until now, xb-general.cy.js was talking to the mock server!) So, Ben also made that happen. (Long overdue ever since the client and server first got connected … 4 weeks ago.)

Back end

On the back-end side, Ben also improved DX and velocity by removing XB’s dependency on the sdc_test module, which was kinda tricky to install (due to it being a test-only module). This simplifies the onboarding/contribution experience, and makes it easier to try the 0.x branch of the XB module too!

Ted “tedbow” Bowman landed the thorough validation for the component tree list: #3456024: Lift most logic out of ComponentTreeItem::preSave() and into a new validation constraint — yay!

What’s cool is that this one validation constraint is being used to validate both an Experience Builder (XB) field instance on a content entity and the default value of the XB field in config (yes, config schema validation at work!). This means slower progress, but means more reliable foundations, because there’s no separate code paths, no distinct semantics. This is crucial to meet the 14. Configuration management and 1.2 Design system (foundations), 45. Content type template variants and many other product requirements.

In other words: it is an important stepping stone towards both #3446722: Introduce an example set of representative SDC components; transition from “component list” to “component tree” and #3455629: [later phase] [META] 7. Content type templates — aka “default layouts” — affects the tree+props data model

Now that that is in, Ted will begin work next on #3460856: Create validation constraint for ComponentTreeStructure, which is a hard blocker for #3446722.

In progress

So many high-impact MRs having landed this week — I’ve simply omitted the smaller ones that are more housekeeping-esque. In closing, there are also some interesting things in progress:

• Lee “larowlan” Rowlands and I met to dive into his draft/proposal MR on #3454519: [META] Support component types other than SDC. Lee provided context for the 5 different parts that his MR consists of and … I liked the direction each one :) We can’t merge as-is, because A) upstream changed quite a bit since then, B) some things we agreed should change. I’ve summarized our conversation, Lee will be extracting MRs for the different pieces :)
• Jesse “jessebaker” Baker and Lee have been making progress on #3453690: [META] Real-time preview: supporting back-end infrastructure
• #3454257: Allow Experience Builder fields to support Asymmetric and Symmetric translations is on the verge of being finished by Ted
• I revived #3450308: Create an Open API spec for the current mock HTTP requests, because that is growing ever more important

… and some notable new issues:

• Lauri created #3461431: Improve client side error handling, where I suggested to use the technique that we used in Acquia Migrate: Accelerate’s React UI, which worked out very well
• by yours truly: #3461499: Introduce new PropShapeInput config entity type — with strong +1 from Lee :)
• by yours truly: #3461490: Document the current JSON-based data model, and describe in an ADR, but will be impacted by the above, so holding off until that lands
• finally: Gauravvvv created #3462074: Option to change the icon for components on the page hierarchy display — which if we end up doing, would require additions to Single-Directory Components’ metadata

I’m pretty confident next week will be more exciting still (well, has been … because I’m a few weeks behind on writing these posts!)

Week 10 was July 15–21, 2024.

• Drupal
• Acquia
• Experience Builder

• XB's primary menu now closes after dropping a dragged component onto the canvas.
• The first React+TSX-rendered Drupal form!