FLOSS Project Planets

Modern web development relies heavily on automation to stay productive, validate code, and perform repetitive tasks that could slow developers down. Front-end development in particular has evolved, and it can be a daunting task to configure effective automation. In this post, I'll try to walk you through basic automation for your Drupal theme, which uses Storybook as its design system.

Recently I worked on a large Drupal project that needed to migrate its design system from Patternlab to Storybook. I knew switching design systems also meant switching front-end build tools. The obvious choice seemed to be Webpack, but as I looked deeper into build tools, I discovered ViteJS.

Vite is considered the Next Generation Frontend Tooling, and when tested, we were extremely impressed not only with how fast Vite is, but also with its plugin's ecosystem and its community support. Vite is relatively new, but it is solid and very well maintained. Learn more about Vite.

The topics covered in this post can be broken down in two categories:

1. Preparing the Front-end environment

   • Build the front-end environment with Vite & Storybook
   • Restructure the project
   • Configure TwigJS
   • Configure postCSS
   • Storybook's CSS configuration

2. Automating the environment

   • Copying static assets
   • The Watch task
   • Linting CSS and JavaScript

1. Build the front-end environment with Vite & Storybook

In a previous post, I wrote in detail how to build a front-end environment with Vite and Storybook, I am going to spare you those details here but you can reference them from the original post.

1. In your command line, navigate to the directory where you wish to build your environment. If you're building a new Drupal theme, navigate to your site's web/themes/custom/
2. Run the following commands (Storybook should launch at the end):

npm create vite@latest storybook cd storybook npx storybook@latest init --type react

Fig. 1: The first command builds the Vite project, and the last one integrates Storybook into it.

Reviewing Vite's and Storybook's out of the box build scripts

Vite and Storybook ship with a handful of useful scripts. We may find some of them already do what we want or may only need minor tweaks to make them our own.

• In your code editor, open package.json from the root of your newly built project.
• Look in the scripts section and you should see something like this:

"scripts": { "dev": "vite", "build": "vite build", "lint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0", "preview": "vite preview", "storybook": "storybook dev -p 6006", "build-storybook": "storybook build" },

Fig. 2: Example of default Vite and Storybook scripts out of the box.

To run any of those scripts, prefix them with npm run. For example: npm run build, npm run lint, etc. Let's review the scripts.

• dev: This is a Vite-specific command which runs the Vite app we just build for local development
• build: This is the "do it all" command. Running npm run build on a project runs every task defined in the build configuration we will do later. CI/CD runners run this command to build your app for production.
• lint: Will lint your JavaScript code inside .js or .jsx files.
• preview: This is also another Vite-specific command which runs your app in preview mode.
• storybook: This is the command you run to launch and keep Storybook running while you code.
• build-storybook: To build a static version of Storybook to package it or share it, or to run it as a static version of your project.

Building your app for the first time Getting a consistent environment

In front-end development, it is important everyone in your team use the same version of NodeJS while working in the same project. This ensures consistency in your project's behavior for everyone in your team. Differences in the node version your team uses can lead to inconsistencies when the project is built. One way to ensure your team is using the same node version when working in the same project, is by adding a .nvmrc file in the root of your project. This file specifies the node version your project uses. The node version is unique to each project, which means different projects can use different node versions.

• In the root of your theme, create a file called .nvmrc (mind the dot)
• Inside .nvmrc add the following: v20.14.0
• Stop Storybook by pressing Ctrl + C in your keyboard
• Build the app:

nvm install npm install npm run build

Fig. 3: Installs the node version defined in .nvmrc, then installs node packages, and finally builds the app.

NOTE: You need to have NVM installed in your system to execute nvm commands.
You only need to run nvm install once per project unless the node version changes. If you switch to a project that uses a different node version, when you return to this project, run nvm use to set your environment back to the right node version.

The output in the command line should look like this:

Fig. 4: Screenshot of files compiled by the build command.

By default, Vite names the compiled files by appending a random 8-character string to the original file name. This works fine for Vite apps, but for Drupal, the libraries we'll create expect for CSS and JS file names to stay consistent and not change. Let's change this default behavior.

• First, install the glob extension. We'll use this shortly to import multiple CSS files with a single import statement.

npm i -D glob

• Then, open vite.config.js in your code editor. This is Vite's main configuration file.
• Add these two imports around line 3 or directly after the last import in the file

import path from 'path'; import { glob } from 'glob';

• Still in vite.config.js, replace the export default... with the following snippet which adds new settings for file names:

export default defineConfig({ plugins: [ ], build: { emptyOutDir: true, outDir: 'dist', rollupOptions: { input: glob.sync(path.resolve(__dirname,'./src/**/*.{css,js}')), output: { assetFileNames: 'css/[name].css', entryFileNames: 'js/[name].js', }, }, }, })

Fig. 5: Build object to modify where files are compiled as well as their name preferences.

• First we imported path and { glob }. path is part of Vite and glob was added by the extension we installed earlier.
• Then we added a build configuration object in which we defined several settings:
  • emptyOutDir: When the build job runs, the dist directory will be emptied before the new compiled code is added.
  • outDir: Defines the App's output directory.
  • rollupOptions: This is Vite's system for bundling code and within it we can include neat configurations:
    • input: The directory where we want Vite to look for CSS and JS files. Here's where the path and glob imports we added earlier are being used. By using src/**/**/*.{css,js}, we are instructing Vite to look three levels deep into the src directory and find any file that ends with .css or .js.
    • output: The destination for where CSS and JS will be compiled into (dist/css and dist/js), respectively. And by setting assetFileNames: 'css/[name].css', and entryFileNames: 'css/[name].js', CSS and JS files will retain their original names.

Now if we run npm run build again, the output should be like this:

Fig. 6: Screenshot of compiled code using the original file names.

The random 8-character string is gone and notice that this time the build command is pulling more CSS files. Since we configured the input to go three levels deep, the src/stories directory was included as part of the input path.

2. Restructure the project

The out of the box Vite project structure is a good start for us. However, we need to make some adjustments so we can adopt the Atomic Design methodology. This is today's standards and will work well with our Component-driven Development workflow. At a high level, this is the current project structure:

> .storybook/ > dist/ > public/ > src/ |- stories/ package.json vite.config.js

Fig. 7: Basic structure of a Vite project listing only the most important parts.

• > .storybook is the main location for Storybook's configuration.
• > dist is where all compiled code is copied into and where the production app looks for all code.
• > public is where we can store images and other static assets we need to reference from our site. Equivalent to Drupal's /sites/default/files/.
• > src is the directory we work out of. We will update the structure of this directory next.
• package.json tracks all the different node packages we install for our app as well as the scripts we can run in our app.
• vite.config.js is Vite's main configuration file. This is probably where we will spend most of our time.

Adopting the Atomic Design methodology

The Atomic Design methodology was first introduced by Brad Frost a little over ten years ago. Since then it has become the standard for building web projects. Our environment needs updating to reflect the structure expected by this methodology.

• First stop Storybook from running by pressing Ctrl + C in your keyboard.
• Next, inside src, create these directories: base, components, and utilities.
• Inside components, create these directories: 01-atoms, 02-molecules, 03-organisms, 04-layouts, and 05-pages.
• While we're at it, delete the stories directory inside src, since we won't be using it.

NOTE: You don't need to use the same nomenclature as what Atomic Design suggests. I am using it here for simplicity. Update Storybook's stories with new paths

Since the project structure has changed, we need to make Storybook aware of these changes:

• Open .storybook/main.js in your code editor
• Update the stories: [] array as follows:

stories: [ "../src/components/**/*.mdx", "../src/components/**/*.stories.@(js|jsx|mjs|ts|tsx)", ],

Fig. 8: Updating stories' path after project restructure.

The Stories array above is where we tell Storybook where to find our stories and stories docs, if any. In Storybook, stories are the components and their variations.

Add pre-built components

As our environment grows, we will add components inside the new directories, but for the purpose of testing our environment's automation, I have created demo components.

• Download demo components (button, title, card), from src/components/, and save them all in their content part directories in your project.
• Feel free to add any other components you may have built yourself. We'll come back to the components shortly.

3. Configure TwigJS

Before we can see the newly added components, we need to configure Storybook to understands the Twig and YML code we are about to introduce within the demo components. To do this we need to install several node packages.

• In your command line run:

npm i -D vite-plugin-twig-drupal @modyfi/vite-plugin-yaml twig twig-drupal-filters html-react-parser

• Next, update vite.config.js with the following configuration. Add the snippet below at around line 5:

import twig from 'vite-plugin-twig-drupal'; import yml from '@modyfi/vite-plugin-yaml'; import { join } from 'node:path';

Fig. 9: TwigJS related packages and Drupal filters function.

The configuration above is critical for Storybook to understand the code in our components:

• vite-plugin-twig-drupal, is the main TwigJS extension for our project.
• Added two new imports which are used by Storybook to understand Twig:
  • vite-plugin-twig-drupal handles transforming Twig files into JavaScript functions.
  • @modyfi/vite-plugin-yaml let's us pass data and variables through YML to our Twig components.

Creating Twig namespaces

• Still in vite.config.js, add the twig and yml() plugins to add Twig namespaces for Storybook.

plugins: [ twig({ namespaces: { atoms: join(__dirname, './src/components/01-atoms'), molecules: join(__dirname, './src/components/02-molecules'), organisms: join(__dirname, './src/components/03-organisms'), layouts: join(__dirname, './src/components/04-layouts'), pages: join(__dirname, './src/components/05-pages'), }, }), yml(), ],

Fig. 10: Twig namespaces reflecting project restructure.

Since we removed the react() function by using the snippet above, we can remove import react from '@vitejs/plugin-react' from the imports list as is no longer needed.

With all the configuration updates we just made, we need to rebuild the project for all the changes to take effect. Run the following commands:

npm run build npm run storybook

The components are available but as you can see, they are not styled even though each component contains a CSS stylesheet in its directory. The reason is Storybook has not been configured to find the component's CSS. We'll address this shortly.

4. Configure postCSS

What is PostCSS? It is a JavaScript tool or transpiler that turns a special PostCSS plugin syntax into Vanilla CSS.

As we start interacting with CSS, we need to install several node packages to enable functionality we would not have otherwise. Native CSS has come a long way to the point that I no longer use Sass as a CSS preprocessor.

• Stop Storybook by pressing Ctrl + C in your keyboard
• In your command line run this command:

npm i -D postcss postcss-import postcss-import-ext-glob postcss-nested postcss-preset-env

• At the root of your theme, create a new file called postcss.config.js, and in it, add the following:

import postcssImport from 'postcss-import'; import postcssImportExtGlob from 'postcss-import-ext-glob'; import postcssNested from 'postcss-nested'; import postcssPresetEnv from 'postcss-preset-env'; export default { plugins: [ postcssImportExtGlob(), postcssImport(), postcssNested(), postcssPresetEnv({ stage: 4, }), ], };

Fig. 11: Base configuration for postCSS.

One cool thing about Vite is that it comes with postCSS functionality built in. The only requirement is that you have a postcss.config.js file in the project's root. Notice how we are not doing much configuration for those plugins except for defining them. Let's review the code above:

• postcss-import the base for importing CSS stylesheets.
• postcss-import-ext-glob to do bulk @import of all CSS content in a directory.
• postcss-nested to unwrap nested rules to make its syntax closer to Sass.
• postcss-preset-env defines the CSS browser support level we need. Stage 4 means we want the "web standards" level of support.

5. CSS and JavaScript configuration

The goal here is to ensure that every time a new CSS stylesheet or JS file is added to the project, Storybook will automatically be aware and begin consuming their code.

NOTE: This workflow is only for Storybook. In Drupal we will use Drupal libraries in which we will include any CSS and JS required for each component.

There are two types of styles to be configured in most project, global styles which apply site-wide, and components styles which are unique to each component added to the project.

Global styles

• Inside src/base, add two stylesheets: reset.css and base.css.
• Copy and paste the styles for reset.css and base.css.
• Inside src/utilities create utilities.css and in it paste these styles.
• Inside src/, create a new stylesheet called styles.css.
• Inside styles.css, add the following imports:

@import './base/reset.css'; @import './base/base.css'; @import './utilities/utilities.css';

Fig. 12: Imports to gather all global styles.

The order in which we have imported our stylesheets is important as the cascading order in which they load makes a difference. We start from reset to base, to utilities.

• reset.css: A reset stylesheet (or CSS reset) is a collection of CSS rules used to clear the browser's default formatting of HTML elements, removing potential inconsistencies between different browsers before any of our styles are applied.
• base.css: CSS Base applies a style foundation for HTML elements that is consistent for baseline styles such as typography, branding and colors, font-sizes, etc.
• utilities.css: Are a collection of pre-defined CSS rules we can apply to any HTML element. Rules such as variables for colors, font size, font color, as well as margin, sizes, z-index, animations, etc.

Component styles

Before our components can be styled with their unique and individual styles, we need to make sure all our global styles are loaded so the components can inherit all the base/global styles.

• Inside src/components create a new stylesheet, components.css. This is where we are going to gather all components styles.
• Inside components.css add glob imports for each of the component's categories:

@import-glob './01-atoms/**/*.css'; @import-glob './02-molecules/**/*.css';

Fig. 13: Glob import for all components of all categories.

NOTE: Since we only have Atoms and Molecules to work with, we are omitting imports for 03-organisms, 04-layouts, 05-pages. Feel free to add them if you have that kind of components. Updating Storybook's Preview

There are several ways in which we can make Storybook aware of our styles and javascript. We could import each component's stylesheet and javascript into each *.stories.js file, but this could result in some components with multiple sub-components having several CSS and JS imports. In addition, this is not an automated system which means we need to manually do imports as they become available. The approach we are going to take is importing the stylesheets we created above into Storybook's preview system. This provides a couple of advantages:

• The component's *.stories.js files are clean without any css imports as all CSS will already be available to Storybook.
• As we add new components with individual stylesheets, these stylesheets will automatically be recognized by Storybook.

Remember, the order in which we import the styles makes a difference. We want all global and base styles to be imported first, before we import component styles.

• In .storybook/preview.js add these imports at the top of the page around line 2.

import Twig from 'twig'; import drupalFilters from 'twig-drupal-filters'; import '../src/styles.css'; /* Contains reset, base, and utilities styles. */ import '../src/components/components.css'; /* Contains all components CSS. */ function setupFilters(twig) { twig.cache(); drupalFilters(twig); return twig; } setupFilters(Twig);

Fig. 14: Importing all styles, global and components.

In addition to importing two new extensions: twig and twig-drupal-filters, we setup a setupFilters function for Storybook to read Drupal filters we may use in our components. We are also importing two of the stylesheets we created earlier:

• styles.css contains all the CSS code from reset.css, base.css, and utilities.css (in that order)
• components.css contains all the CSS from all components. As new components are added and they have their own stylesheets, they will automatically be included in this import.

IMPORTANT: For Storybook to immediately display changes you make in your CSS, the imports above need to be from the src directory and not dist. I learned this the hard way. JavaScript compiling

On a typical project, you will find that the majority of your components don't use JavaScript, and for this reason, we don't need such an elaborate system for JS code. Importing the JS files in the component's *.stories.js should work just fine. Since the demo components dont use JS, I have commented near the top of card.stories.js how the component's JS file would be imported if JS was needed.

If the need for a more automated JavaScript processing workflow arose, we could easily repeat the same CSS workflow but for JS.

Build the project again

Now that our system for CSS and JS is in place, let's build the project to ensure everything is working as we expect it.

npm run build npm run storybook

You may notice that now the components in Storybook look styled. This tells us our new system is working as expected. However, the Card component, if you used the demo components, is missing an image. We will address this issue in the next section.

This concludes the preparation part of this post. The remaining part will focus on creating automation tasks for compiling, minifying and linting code, copying static assets such as images, and finally, watching for code changes as we code. 6. Copying images and other assets

Copying static assets like images, icons, JS, and other files from src into dist is a common practice in front-end projects. Vite comes with built-in functionality to do this. Your assets need to be placed in the public directory and Vite will automatically copy them on build. However, sometimes we may have those assets alongside our components or other directories within our project.

In Vite, there are many ways to accomplish any task, in this case, we will be using a nice plugin called vite-plugin-static-copy. Let's set it up.

• If Storybook is running, kill it with Ctrl + C in your keyboard
• Next, install the extension by running:

npm i -D vite-plugin-static-copy

• Next, right after all the existing imports in vite.config.js, import one more extension:

import { viteStaticCopy } from 'vite-plugin-static-copy';

• Lastly, still in vite.config.js, add the viteStaticCopy function configuration inside the plugins:[] array:

viteStaticCopy({ targets: [ { src: './src/components/**/*.{png,jpg,jpeg,svg,webp,mp4}', dest: 'images', }], }),

Fig. 15: Adds tasks for copying JavaScript and Images from src to dist.

The viteStaticCopy function we added allows us to copy any type of static assets anywhere within your project. We added a target array in which we included src and dest for the images we want copied. Every time we run npm run build, any images inside any of the components, will be copied into dist/images.
If you need to copy other static assets, simply create new targets for each.

• Build the project again:

npm run build npm run storybook

The missing image for the Card component should now be visible, see below. Pretty sweet! 🍰

Fig. 16: Screenshot of the Card component in Storybook.

7. The Watch task

A watch task makes it possible for developers to see the changes they are making as they code, and without being interrupted by running commands. Depending on your configuration, a watch task watches for any changes you make to CSS, JavaScript and other file types, and upon saving those changes, code is automatically compiled, and a Hard Module Reload (HMR) is evoked, making the changes visible in Storybook.

Although there are extensions to create watch tasks, we will stick with Storybook's out of the box watch functionality because it does everything we need. In fact, I have used this very approach on a project that supports over one hundred sites.

I actually learned this the hard way, I originally was importing the key stylesheets in .storybook/preview.js using the files from dist. This works to an extend because the code is compiled upon changes, but Storybook is not aware of the changes unless we restart Storybook. I spent hours debugging this issue and tried so many other options, but at the end, the simple solution was to import CSS and JS into Storybook's preview using the source files. For example, if you look in .storybook/preview.js, you will see we are importing two CSS files which contain all of the CSS code our project needs:

import '../src/styles.css'; import '../src/components/components.css';

Fig. 17: Importing source assets into Storybook's preview.

Importing source CSS or JS files into Storybook's preview allows Storybook to become aware immediately of any code changes.

The same, or kind of the same works for JavaScript. However, the difference is that for JS, we import the JS file in the component's *.stories.js, which in turn has the same effect as what we've done above for CSS. The reason for this is that typically not every component we build needs JS.

A real watch task

Currently we are running npm run storybook as a watch task. Nothing wrong with this. However, to keep up with standards and best practices, we could rename the storybook command, watch, so we can run npm run watch. Something to consider.

You could also make a copy of the storybook command and name it watch and add additional commands you wish to run with watch, while leaving the original storybook command intact. Choices, choices.

8. Linting CSS and JavaScript

Our workflow is coming along nicely. There are many other things we can do but for now, we will end with one last task: CSS and JS linting.

• Install the required packages. There are several of them.

npm i -D eslint stylelint vite-plugin-checker stylelint-config-standard stylelint-order stylelint-selector-pseudo-class-lvhfa

• Next, after the last import in vite.config.js, add one more:

import checker from 'vite-plugin-checker';

• Then, let's add one more plugin in the plugins:[] array:

checker({ eslint: { lintCommand: 'eslint "./src/components/**/*.{js,jsx}"', }, stylelint: { lintCommand: 'stylelint "./src/components/**/*.css"', }, }),

Fig. 18: Checks for linting CSS and JavaScript.

So we can execute the above checks on demand, we can add them as commands to our app.

• In package.json, within the scripts section, add the following commands:

"eslint": "eslint . --ext js,jsx --report-unused-disable-directives --max-warnings 0", "stylelint": "stylelint './src/components/**/*.css'",

Fig. 19: Two new npm commands to lint CSS and JavaScript.

• We installed a series of packages related to ESLint and Stylelint.
• vite-plugin-checker is a plugin that can run TypeScript, VLS, vue-tsc, ESLint, and Stylelint in worker thread.
• We imported vite-plugin-checker and created a new plugin with two checks, one for ESLint and the other for Stylelint.
• By default, the new checks will run when we execute npm run build, but we also added them as individual commands so we can run them on demand.

Configure rules for ESLint and Stylelint

Both ESLint and Stylelint use configuration files where we can configure the various rules we want to enforce when writing code. The files they use are eslint.config.js and .stylelintrc.yml respectively. For the purpose of this post, we are only going to add the .stylelintrc.yml in which we have defined basic CSS linting rules.

• In the root of your theme, create a new file called .stylelintrc.yml (mind the dot)
• Inside .stylelintrc.yml, add the following code:

extends: - stylelint-config-standard plugins: - stylelint-order - stylelint-selector-pseudo-class-lvhfa ignoreFiles: - './dist/**' rules: at-rule-no-unknown: null alpha-value-notation: number color-function-notation: null declaration-empty-line-before: never declaration-block-no-redundant-longhand-properties: null hue-degree-notation: number import-notation: string no-descending-specificity: null no-duplicate-selectors: true order/order: - - type: at-rule hasBlock: false - custom-properties - declarations - unspecified: ignore disableFix: true order/properties-alphabetical-order: error plugin/selector-pseudo-class-lvhfa: true property-no-vendor-prefix: null selector-class-pattern: null value-keyword-case: - lower - camelCaseSvgKeywords: true ignoreProperties: - /^--font/

Fig. 20: Basic CSS Stylelint rules.

The CSS rules above are only a starting point, but should be able to check for the most common CSS errors.

Test the rules we've defined by running either npm run build or npm run stylelint. Either command will alert you of a couple of errors our current code contains. This tells us the linting process is working as expected. You could test JS linting by creating a dummy JS file inside a component and writing bad JS in it.

9. One last thing

It goes without saying that we need to add storybook.info.yml and storybook.libraries.yml files for this to be a true Drupal theme. In addition, we need to create the templates directory somewhere within our theme.

storybook.info.yml

The same way we did for Storybook, we need to create namespaces for Drupal. This requires the Components module and storybook.info.yml configuration is like this:

components: namespaces: atoms: - src/components/01-atoms molecules: - src/components/02-molecules organisms: - src/components/03-organisms layouts: - src/components/04-layouts pages: - src/components/05-pages templates: - src/templates

Fig. 21: Drupal namespaces for nesting components.

storybook.libraries.yml

The recommended method for adding CSS and JS to components or a theme in Drupal is by using Drupal libraries. In our project we would create a library for each component in which we will include any CSS or JS the component needs. In addition, we need to create a global library which includes all the global and utilities styles. Here are examples of libraries we can add in storybook.libraries.yml.

global: version: VERSION css: base: dist/css/reset.css: {} dist/css/base.css: {} dist/css/utilities.css: {} button: css: component: dist/css/button.css: {} card: css: component: dist/css/card.css: {} title: css: component: dist/css/title.css: {}

Fig. 22: Drupal libraries for global styles and component's styles.

/templates

Drupal's templates' directory can be created anywhere within the theme. I typically like to create it inside the src directory. Go ahead and create it now.

• Inside storybook.info.yml, add a new Twig namespace for the templates directory. See example above. Update your path accordingly based on where you created your templates directory.

P.S: When the Vite project was originally created at the begining of the post, Vite created files such as App.css, App.js, main.js, and index.html. All these files are in the root of the project and can be deleted. It won't affect any of the work we've done, but Vite will no longer run on its own, which we don't need it to anyway.

In closing

I realize this is a very long post, but there is really no way around it when covering these many topics in a single post. I hope you found the content useful and can apply it to your next Drupal project. There are different ways to do what I've covered in this post, and I challenge you to find better and more efficient ways. For now, thanks for visiting.

Download the theme

A full version of the Drupal theme built with this post can be downloaded.

Download the theme

Make sure you are using the theme branch from the repo.

A new minor release 0.1.5 of RApiSerialize arrived on CRAN today. The RApiSerialize package is used by both my RcppRedis as well as by Travers excellent qs package. This release adds an optional C++ namespace, available when the API header file is included in a C++ source file. And as one often does, the release also brings a few small updates to different aspects of the packaging.

Changes in version 0.1.4 (2024-09-28)

• Add C++ namespace in API header (Dirk in #9 closing #8)

• Several packaging updates: switched to Authors@R, README.md badge updates, added .editorconfig and cleanup

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More details are at the RApiSerialize page; code, issue tickets etc at the GitHub repositoryrapiserializerepo.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

The Reproducible Builds project relies on several projects, supporters and sponsors for financial support, but they are also valued as ambassadors who spread the word about our project and the work that we do.

This is the eighth installment in a series featuring the projects, companies and individuals who support the Reproducible Builds project. We started this series by featuring the Civil Infrastructure Platform project, and followed this up with a post about the Ford Foundation as well as recent ones about ARDC, the Google Open Source Security Team (GOSST), Bootstrappable Builds, the F-Droid project, David A. Wheeler and Simon Butler.

Today, however, we will be talking with Kees Cook, founder of the Kernel Self-Protection Project.

Vagrant Cascadian: Could you tell me a bit about yourself? What sort of things do you work on?

Kees Cook: I’m a Free Software junkie living in Portland, Oregon, USA. I have been focusing on the upstream Linux kernel’s protection of itself. There is a lot of support that the kernel provides userspace to defend itself, but when I first started focusing on this there was not as much attention given to the kernel protecting itself. As userspace got more hardened the kernel itself became a bigger target. Almost 9 years ago I formally announced the Kernel Self-Protection Project because the work necessary was way more than my time and expertise could do alone. So I just try to get people to help as much as possible; people who understand the ARM architecture, people who understand the memory management subsystem to help, people who understand how to make the kernel less buggy.

Vagrant: Could you describe the path that lead you to working on this sort of thing?

Kees: I have always been interested in security through the aspect of exploitable flaws. I always thought it was like a magic trick to make a computer do something that it was very much not designed to do and seeing how easy it is to subvert bugs. I wanted to improve that fragility. In 2006, I started working at Canonical on Ubuntu and was mainly focusing on bringing Debian and Ubuntu up to what was the state of the art for Fedora and Gentoo’s security hardening efforts. Both had really pioneered a lot of userspace hardening with compiler flags and ELF stuff and many other things for hardened binaries. On the whole, Debian had not really paid attention to it. Debian’s packaging building process at the time was sort of a chaotic free-for-all as there wasn’t centralized build methodology for defining things. Luckily that did slowly change over the years. In Ubuntu we had the opportunity to apply top down build rules for hardening all the packages. In 2011 Chrome OS was following along and took advantage of a bunch of the security hardening work as they were based on ebuild out of Gentoo and when they looked for someone to help out they reached out to me. We recognized the Linux kernel was pretty much the weakest link in the Chrome OS security posture and I joined them to help solve that. Their userspace was pretty well handled but the kernel had a lot of weaknesses, so focusing on hardening was the next place to go. When I compared notes with other users of the Linux kernel within Google there were a number of common concerns and desires. Chrome OS already had an “upstream first” requirement, so I tried to consolidate the concerns and solve them upstream. It was challenging to land anything in other kernel team repos at Google, as they (correctly) wanted to minimize their delta from upstream, so I needed to work on any major improvements entirely in upstream and had a lot of support from Google to do that. As such, my focus shifted further from working directly on Chrome OS into being entirely upstream and being more of a consultant to internal teams, helping with integration or sometimes backporting. Since the volume of needed work was so gigantic I needed to find ways to inspire other developers (both inside and outside of Google) to help. Once I had a budget I tried to get folks paid (or hired) to work on these areas when it wasn’t already their job.

Vagrant: So my understanding of some of your recent work is basically defining undefined behavior in the language or compiler?

Kees: I’ve found the term “undefined behavior” to have a really strict meaning within the compiler community, so I have tried to redefine my goal as eliminating “unexpected behavior” or “ambiguous language constructs”. At the end of the day ambiguity leads to bugs, and bugs lead to exploitable security flaws. I’ve been taking a four-pronged approach: supporting the work people are doing to get rid of ambiguity, identify new areas where ambiguity needs to be removed, actually removing that ambiguity from the C language, and then dealing with any needed refactoring in the Linux kernel source to adapt to the new constraints.

None of this is particularly novel; people have recognized how dangerous some of these language constructs are for decades and decades but I think it is a combination of hard problems and a lot of refactoring that nobody has the interest/resources to do. So, we have been incrementally going after the lowest hanging fruit. One clear example in recent years was the elimination of C’s “implicit fall-through” in switch statements. The language would just fall through between adjacent cases if a break (or other code flow directive) wasn’t present. But this is ambiguous: is the code meant to fall-through, or did the author just forget a break statement? By defining the “[[fallthrough]]” statement, and requiring its use in Linux, all switch statements now have explicit code flow, and the entire class of bugs disappeared. During our refactoring we actually found that 1 in 10 added “[[fallthrough]]” statements were actually missing break statements. This was an extraordinarily common bug!

So getting rid of that ambiguity is where we have been. Another area I’ve been spending a bit of time on lately is looking at how defensive security work has challenges associated with metrics. How do you measure your defensive security impact? You can’t say “because we installed locks on the doors, 20% fewer break-ins have happened.” Much of our signal is always secondary or retrospective, which is frustrating: “This class of flaw was used X much over the last decade so, and if we have eliminated that class of flaw and will never see it again, what is the impact?” Is the impact infinity? Attackers will just move to the next easiest thing. But it means that exploitation gets incrementally more difficult. As attack surfaces are reduced, the expense of exploitation goes up.

Vagrant: So it is hard to identify how effective this is… how bad would it be if people just gave up?

Kees: I think it would be pretty bad, because as we have seen, using secondary factors, the work we have done in the industry at large, not just the Linux kernel, has had an impact. What we, Microsoft, Apple, and everyone else is doing for their respective software ecosystems, has shown that the price of functional exploits in the black market has gone up. Especially for really egregious stuff like a zero-click remote code execution.

If those were cheap then obviously we are not doing something right, and it becomes clear that it’s trivial for anyone to attack the infrastructure that our lives depend on. But thankfully we have seen over the last two decades that prices for exploits keep going up and up into millions of dollars. I think it is important to keep working on that because, as a central piece of modern computer infrastructure, the Linux kernel has a giant target painted on it. If we give up, we have to accept that our computers are not doing what they were designed to do, which I can’t accept. The safety of my grandparents shouldn’t be any different from the safety of journalists, and political activists, and anyone else who might be the target of attacks. We need to be able to trust our devices otherwise why use them at all?

Vagrant: What has been your biggest success in recent years?

Kees: I think with all these things I am not the only actor. Almost everything that we have been successful at has been because of a lot of people’s work, and one of the big ones that has been coordinated across the ecosystem and across compilers was initializing stack variables to 0 by default. This feature was added in Clang, GCC, and MSVC across the board even though there were a lot of fears about forking the C language.

The worry was that developers would come to depend on zero-initialized stack variables, but this hasn’t been the case because we still warn about uninitialized variables when the compiler can figure that out. So you still still get the warnings at compile time but now you can count on the contents of your stack at run-time and we drop an entire class of uninitialized variable flaws. While the exploitation of this class has mostly been around memory content exposure, it has also been used for control flow attacks. So that was politically and technically a large challenge: convincing people it was necessary, showing its utility, and implementing it in a way that everyone would be happy with, resulting in the elimination of a large and persistent class of flaws in C.

Vagrant: In a world where things are generally Reproducible do you see ways in which that might affect your work?

Kees: One of the questions I frequently get is, “What version of the Linux kernel has feature $foo?” If I know how things are built, I can answer with just a version number. In a Reproducible Builds scenario I can count on the compiler version, compiler flags, kernel configuration, etc. all those things are known, so I can actually answer definitively that a certain feature exists. So that is an area where Reproducible Builds affects me most directly. Indirectly, it is just being able to trust the binaries you are running are going to behave the same for the same build environment is critical for sane testing.

Vagrant: Have you used diffoscope?

Kees: I have! One subset of tree-wide refactoring that we do when getting rid of ambiguous language usage in the kernel is when we have to make source level changes to satisfy some new compiler requirement but where the binary output is not expected to change at all. It is mostly about getting the compiler to understand what is happening, what is intended in the cases where the old ambiguity does actually match the new unambiguous description of what is intended. The binary shouldn’t change. We have used diffoscope to compare the before and after binaries to confirm that “yep, there is no change in binary”.

Vagrant: You cannot just use checksums for that?

Kees: For the most part, we need to only compare the text segments. We try to hold as much stable as we can, following the Reproducible Builds documentation for the kernel, but there are macros in the kernel that are sensitive to source line numbers and as a result those will change the layout of the data segment (and sometimes the text segment too). With diffoscope there’s flexibility where I can exclude or include different comparisons. Sometimes I just go look at what diffoscope is doing and do that manually, because I can tweak that a little harder, but diffoscope is definitely the default. Diffoscope is awesome!

Vagrant: Where has reproducible builds affected you?

Kees: One of the notable wins of reproducible builds lately was dealing with the fallout of the XZ backdoor and just being able to ask the question “is my build environment running the expected code?” and to be able to compare the output generated from one install that never had a vulnerable XZ and one that did have a vulnerable XZ and compare the results of what you get. That was important for kernel builds because the XZ threat actor was working to expand their influence and capabilities to include Linux kernel builds, but they didn’t finish their work before they were noticed. I think what happened with Debian proving the build infrastructure was not affected is an important example of how people would have needed to verify the kernel builds too.

Vagrant: What do you want to see for the near or distant future in security work?

Kees: For reproducible builds in the kernel, in the work that has been going on in the ClangBuiltLinux project, one of the driving forces of code and usability quality has been the continuous integration work. As soon as something breaks, on the kernel side, the Clang side, or something in between the two, we get a fast signal and can chase it and fix the bugs quickly. I would like to see someone with funding to maintain a reproducible kernel build CI. There have been places where there are certain architecture configurations or certain build configuration where we lose reproducibility and right now we have sort of a standard open source development feedback loop where those things get fixed but the time in between introduction and fix can be large. Getting a CI for reproducible kernels would give us the opportunity to shorten that time.

Vagrant: Well, thanks for that! Any last closing thoughts?

Kees: I am a big fan of reproducible builds, thank you for all your work. The world is a safer place because of it.

Vagrant: Likewise for your work!

For more information about the Reproducible Builds project, please see our website at reproducible-builds.org. If you are interested in ensuring the ongoing security of the software that underpins our civilisation and wish to sponsor the Reproducible Builds project, please reach out to the project by emailing contact@reproducible-builds.org.

The Python Coding Stack has a new look. Here it is:

The Stack has been growing steadily as a standalone publication, separate from The Python Coding Book and The Python Coding Place, and now it has its own identity.

It offers a very different and unique perspective on Python programming, often with a narrative style.

If you’ve not read articles on The Stack yet, have a look at some of the most recent ones, or the top 5.

The post The Python Coding Stack’s New Look appeared first on The Python Coding Book.

I playfully quipped about changelogs, and Sumana Harihareswara thoughtfully responded with Changelogs and Release Notes. I agree with her on some things, and disagree on others.

My point with the meme was that people should put effort into a hand-crafted description of what has changed in each release of their product. It should be focused on what users need to know, and not include internal changes, which can be found in the git commits or pull requests. It’s easy to publish a list of commits or pull requests and call it a changelog, but it’s not that helpful to your users trying to understand what has changed for them. That was the point of the meme.

But Sumana raised the stakes, explaining why projects should produce two hand-crafted descriptions. The first is a changelog which mentions every non-trivial change. The second are release notes which should be user-focused with more details.

I liked the reasons Sumana gave:

• Release notes can include project-level information that doesn’t correspond to a particular change in a release. Maybe you started a new discussion forum, or there’s a shift in maintainer attention, plans for upcoming work, and so on.
• If the release notes are user-focused, then the changelog can be more comprehensive, giving people a fuller picture of the work that goes into producing the project. This can pull back the curtain, helping people understand the inner workings of the project and perhaps find a way to help out.

My problem with separating the changelog and release notes is that I have limited energy to produce them, and perhaps more importantly, people have limited attention to read them. For my projects, I opt instead for a middle ground: my changelogs lean more toward Sumana’s ideal of release notes. They are hand-written, focused on what users of the project need to know, and do not include things like build changes and refactorings.

For large projects like Python and Linux, there are many maintainers and many types of information, so it makes sense to have multiple views of “what’s changed.” For single-maintainer projects, it feels like too much. I applaud people who can do it, but I don’t think I can, and I won’t expect it from others.

Ultimately, each project has to decide for themselves how to balance the effort and the benefit. They know their audience(s), and what resources they have to do the work. Open source is already difficult, the last thing I want to do is add a giant SHOULD to a project.

There’s an inexact nested ratio at work in projects: Most users (say 90%) will only consume, you will never hear from them. You hear from the remaining 10%, but only 10% of those will do something you consider a contribution. For widely used projects like coverage.py, I think the ratio might be more like 1% of 1% instead of 10% of 10%. How does this affect your communication approach? You could look at it two ways: either write for the audience you have (focus on the 90%), or write for the audience you want (focus on the 10%).

In my changelogs now, for fixes I try to describe the bad thing that used to happen and any important changes in behavior. For features, I link to the new docs. I include links to issues and pull requests, and I name the contributors who helped.

So I guess my approach is to write changelogs for the 90%. But I like Sumana’s idea of making the full picture of maintainence more visible to people, so I’m thinking about how to add that without changing the essential character of my changelog. Perhaps something at the end summarizing the changes that aren’t yet mentioned, with a link to the git history? I’m not sure I can automate collecting that information, but I’ll have to play with it.

This year I attended EuroBSDCon 2024 in Dublin. I always appreciate an excuse to head over to Ireland, and this seemed like a great chance to spend some time in Dublin and learn new things. Due to constraints on my time I didn’t go to the 2 day devsummit that precedes the conference, only the main event itself. The Event EuroBSDCon was attended by about 200-250 people, the hardcore of the BSD community!

Test your understanding of Python’s most common pieces of syntactic sugar and how they make your code more Pythonic and readable.

Take this quiz after reading our Syntactic Sugar: Why Python is Sweet and Pythonic tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of Python 3.13: Cool New Features for You to Try. By working through this quiz, you’ll review the key updates and improvements in this version of Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Every great product needs a great supporting documentation - this rule is as simple and well known as it is hard to stick to, especially if we are talking about the continuously growing Open Source system that Drupal is. A while ago we came to realise that Drupal documentation needs a revamp. Together with ever amazing community members, the DA engineering team has been looking for a solution that will bring world class documentation to Drupal and that joint effort did not go in vain as such a solution has been defined!

As each of you can imagine, the goal of overhauling documentation tooling to a modern docs-as-code system is a big challenge so it is logical that we decided to define bite-sized deliverables and today we are happy to announce the first phase of the project - delivering clear and easy to follow user guide for Drupal CMS. And we need your help!

The Drupal Association is kicking off an initiative to bring Drupal.org documentation and accompanying software to world-class level. For the first phase dedicated to the Drupal CMS we are partnering with Drupalize.me: they will create the gold-standard user guide for Drupal CMS. 

But that is just the beginning! As a next step Drupal Association will onboard the Documentation Lead who, in close collaboration with the community, will help to ensure that Drupal.org documentation is clear, comprehensive and current. 

Last but not the least, we will implement a docs-as-code system that will improve the creation and maintenance process of documentation. 

Now the question is how you can help? And the answer is simple - by “Adopting a Document”!

Inspired by the “Adopt a Highway” program that is becoming more and more popular in North America, we created a way for the partners to get involved in creating great documentation by taking care of one of the Drupal CMS milestones and sponsoring creation and further maintenance of the documentation for one or multiple Drupal CMS working tracks.

Thanks to your support, future users of Drupal CMS will be able to easily find answers to the questions that might arise as they discover Drupal CMS capabilities.

But not only that, as we find it fair for the partners who will decide to contribute to the initiative to receive some benefits:

• Your logo will be placed in the sidebar of the adopted documentation section—which Your logo will be placed on the documentation page(s) you adopt, visible to the thousands of new users we expect to try Drupal CMS

• Your logo will be highlighted on Drupal.org as a Drupal CMS sponsor, as well as at DrupalCons for the next year!

• For each $100 of contribution you will receive 1 credit that you will be able to use within 1 year.

We’ve got 30 documentation sections available for sponsorship so far. You can adopt one of those and by donating $2400 you will ensure that future users will be able to easily find answers to the questions that might arise as they discover Drupal CMS as well as will enable for the next phase of the project to get started. 

Sounds exciting, right? So if you are keen to be part of Adopt a document - do reach out to me and let’s work together to bring Drupal Documentation to the new level!

This months has been so densely packed with conferences that I’m lagging behind on reporting on them here. So you get two in one post now, the Nextcloud Community Conference 2024 which followed almost back-to-back on Akademy, and the Matrix conference 2024 last weekend.

Nextcloud Photo by Nextcloud

The Nextcloud conference is a week-long event, I’ve however only managed to attend parts of the first two conference days. Still enough for a few interesting conversations.

The most exciting development from a KDE (PIM) point of view would be the progress on CalDav push notifications. While this was still mostly a theoretical concept when we discussed it last year with the DAVx⁵ team, there is now a working prototype consisting of the DAV Push Nextcloud (server) app and the DAVx⁵ Android sync client.

With a working server to test against, getting this implemented for the KDE clients has become a lot more interesting and feasible.

We also discussed Wayland support of the Nextcloud desktop client, and I learned an interesting detail in Björn Lundell’s keynote, the fact that the official EU translations of laws can have (unintentional) semantic differences, as can be observed apparently in the definition of “FOSS” in the new EU Cyber Resilience Act (CRA).

Matrix

For the Matrix conference I also only managed to attend half of it, but KDE overall had a slightly bigger presence there, as KDE also is an Associate Member of the Matrix foundation.

My main interest here was advancing the Matrix-based trip synchronization in Itinerary. I described the basic idea here previously, but of course actually implementing this encounters many more challenges. And those are much easier to solve when you have physical access to a bunch of people with extensive experience on that subject.

Most parts for interacting with Matrix directly should be done now, but there’s still a lot of things to be sorted out to make the synchronization robust in all kinds of scenarios. Overall this looks promising though and might get done in time for 24.12.

Beyond Itinerary I also was involved a bit with reviving and improving the Android CD builds of NeoChat. Some of the discussed changes would also benefit other KDE applications on Android.

What’s next

October will be a bit more relaxed regarding events, next I’ll be at the bi-annual OSM Hack Weekend in Karlruhe again.

One of the major reason of using static blogging for me is to less worry about how the site will look like. Instead the focus was to just write (which of course I did not do well this year). I did not change my blog's theme for many many years.

But, I noticed Oskar Wickström created a monospace based site and kindly released it under MIT license. I liked the theme, so decided to start using it. I still don't know HTML/CSS but managed to change the template for my website.

You can let me know over mastodon what do you think :)

The core Plasma team remains deep in bug-fixing mode until Plasma 6.2.1, with lots of bugs fixed this week! This is the second-to-last week of development before the repos are frozen, and we’re cranking away like mad to get 6.2 in great shape. And it is indeed in very good shape so far. The worst issues we’re still seeing are related to notifications freezing and being mis-rendered, caused by recent changes made to fix another significantly less severe issue. So in the worst-case scenario, we can simply revert the changes before the final 6.2 release if we don’t manage to fix the regressions in time.

Something I hope we can prove to the world is that we’re capable of keeping Plasma stable over the long haul at the same time that we add features and refine the UI. Plasma 6.2 offers us a good opportunity for it!

Notable UI Improvements

Kickoff’s category icons have been made symbolic and monochrome (where the active icon theme supports it), which conforms better to the HIG and other apps, and mirrors a similar change done for Discover recently (me: Nate Graham, Plasma 6.2.0. Link):

On System Settings’ Region and Language page, the list of languages you can add to your system is now alphabetized by first letter (rather than by the hidden language code), and all the languages are properly capitalized (David Edmundson, Plasma 6.2.0. Link 1 and link 2)

In Plasma’s Digital Clock popup, calendar dates are now perfectly horizontally aligned even when some of them have text for events under them, and content can no longer sometimes overflow the header when using the combination of an alternate calendar plugin, certain third-party Plasma themes, and a large font size (Tusooa Windy, Plasma 6.2.0. Link 1 and link 2)

It’s now possible to get a standard context menu for the text field that appears when renaming files or folders on the desktop (Akseli Lahtinen, Plasma 6.2.0. Link)

System Settings’ Legacy X11 App Support page now supports the non-default settings highlighting feature (David Edmundson, Plasma 6.2.0. Link)

With the “Switch virtual desktops on screen edge” setting turned on, screen edges with no virtual desktop on the other side of them will no longer inappropriately show a glow anyway (Xaver Hugl, Plasma 6.2.0. Link)

Plasma notifications that show job progress no longer include a “Details” button if there are no extra details to show (Kai Uwe Broulik, Plasma 6.2.0. Link)

Windows no longer snap to the invisible edge where an auto-hidden panel would be when it’s visible (Vlad Zahorodnii, Plasma 6.2.0. Link)

Improved the margins and paddings for the “Add Widgets” sidebar (me: Nate Graham, Plasma 6.2.0. Link 1 and link 2):

When dragging an abstract representation of an app (from e.g. Kickoff, KRunner, or Task Manager) to the desktop, you’ll no longer be prompted to create an Icon widget out of it; you’ll now only have the “Copy” and “Link” options that create an actual file. The Icon Widget option was found to be confusing on the desktop because it doesn’t follow the normal semantics for desktop icons. This is part of a larger project to improve the usability of dragging apps to the desktop; expect more similar patches in coming weeks (me: Nate Graham, Plasma 6.3.0. Link)

Improved how System Settings’ Default Applications page communicates the situation where you’ve forced it to use an app that doesn’t actually advertise support for the file formats you want it to open (Marco Martin, Plasma 6.3.0. Link):

Don’t do this, it’s silly!

Every KWin effect listed on System Settings’ Desktop effects page that needs to be activated using a keyboard shortcut now mentions this in its caption (me: Nate Graham, Plasma 6.3.0. Link)

Plasma’s Sticky Note widget now has a symbolic monochrome widget when placed on a panel while using the Breeze icon theme. This completes the project to support symbolic panel icons for all of the widgets we ship by default! (Martin Frueh, Frameworks 6.7. Link 1 and link 2):

The “sleep and screen locking are inhibited” icon has gotten a redesign to hopefully make its meaning clearer (Andy Betts and Natalie Clarius, Frameworks 6.7. Link)

Notable Bug Fixes

Fixed a case where KWin would crash while you’re using the Khronkite tiling script (Vlad Zahorodnii, Plasma 6.2.0. Link)

Fixed a case where KWin could crash under certain circumstances while the Sheet effect is active (Vlad Zahorodnii, Plasma 6.2.0. Link)

Fixed a case where Plasma could crash and KWin could hang when you drag a layer from GIMP onto the desktop for some reason (David Edmundson, Plasma 6.2.0. Link)

Fixed a case where Plasma could crash when you chicken out of applying a Global Theme and its associated desktop layout after starting the process (Marco Martin, Plasma 6.2.0. Link)

Fixed a case where Powerdevil could crash on login (Alessandro Astone, Plasma 6.2.0. Link)

Fixed a way that System Settings’ KWin Rules and Device Automount pages could crash on close due to the use of nested event loops. Nested event loops are evil; get rid of them all! (Nicolas Fella, Plasma 6.2.0. Link 1 and link 2)

XWayland-using apps now have their accessibility properties exposed to screen readers as expected (David Edmundson, Plasma 6.2.0. Link)

When Flatpak has an oopsie and throws the dreaded “Aborted due to failure” message while you’re updating Flatpaks, Discover now wraps it in a nicer message telling you to try again later, which is usually enough to make it work the next time. This also fixes a related issue with Discover’s error dialogs that could cause them to not be large enough to show their content in some cases. Unfortunately we have not been able to actually fix the error itself or improve its wording yet, since it’s a bug in Flatpak itself (Akseli Lahtinen, Plasma 6.2.0. Link 1 and link 2)

Fixed an annoying bug that could cause some (not all) tiled CSD-using apps to become un-tiled when their headers are clicked. This affected VSCode specifically, but for other affected apps (e.g. Firefox) it can also be an app-specific issue (Vlad Zahorodnii, Plasma 6.2.0. Link)

On System Settings’ Shortcuts page, extremely long labels for shortcuts no longer sometimes overflow the layout (Akseli Lahtinen, Plasma 6.2.0. Link)

Fixed a bug that could cause maximized windows in multi-screen setups to be restored to the wrong screen after un-maximizing them with certain methods (Xaver Hugl, Plasma 6.2.0. Link)

Setting up the Meta key to toggle KWin’s Overview effect now works consistently after a reboot (Xaver Hugl, Plasma 6.2.0. Link)

Fixed an issue that prevented newly installed or deleted third-party splash screens from being shown or removed (respectively) from the relevant System Settings page at the right times (Marco Martin, Plasma 6.2.0. Link)

Fixed an issue that made it hard to trigger edges and hotcorners on screen edges that also have a Plasma panel in auto-hide mode (Xaver Hugl, Plasma 6.2.0. Link)

Fixed a graphical glitch affecting people using AMD and NVIDIA GPUs who maximize windows on a screen with a floating panel (Vlad Zahorodnii, Plasma 6.2.0. Link)

Fixed a color bug in Kirigami that caused the text of disabled buttons in various Kirigami-based apps to not look visually disabled, and also caused caused some pieces of text to inappropriately have a disabled appearance on System Settings’ Screen Locking page (Marco Martin and Arjen Hiemstra, Frameworks 6.7. Link 1 and link 2)

Fixed a series of sizing bugs affecting Kirigami.Dialog and its subclasses that could cause it to not be wide enough when assigned very long footer buttons (Akseli Lahtinen, Frameworks 6.7. Link 1 and link 2)

Fixed the ugly new Qt font selector dialog to at least not be completely visually broken when using a dark color scheme (Kai Uwe Broulik, Qt 6.8.0. Link)

Setting the GTK_USE_PORTAL=1 environment variable on your system to make GTK apps use the portal system (and hence use the superior KDE file dialog) no longer breaks font rendering in GTK apps quite horribly unless the GTK portal is also installed (Ilya Fedin, GTK 3.24.44, Link)

Other bug information of note:

• 2 Very high priority Plasma bug (up from 1 last week). Current list of bugs
• 33 15-minute Plasma bugs (down from 36 last week). Current list of bugs
• 138 KDE bugs of all kinds fixed over the last week. Full list of bugs

Notable in Performance & Technical

Improved the speed with which the Plasma Task Manager widget’s context menu appears when recent document tracking is globally disabled, especially when using a networked home directory (Kai Uwe Broulik, Plasma 5.27.12. Link)

Fixed the binding loops affecting Kirigami.Dialog and its subclasses. These components are widely used, so this should make a difference (Akseli Lahtinen, Frameworks 6.7. Link)

How You Can Help

Please continue to test the Plasma 6.2 beta release! We’ve focused a lot on stability for this release and want to make sure we haven’t missed anything big before the final release in two weeks. Your bug reports do not go into a black hole; we triage every one! So enthusiastic testing and bug reporting is encouraged.

Otherwise, visit https://community.kde.org/Get_Involved to discover additional ways to be part of a project that really matters. Each contributor makes a huge difference in KDE; you are not a number or a cog in a machine! You don’t have to already be a programmer, either. I wasn’t when I got started. Try it, you’ll like it! We don’t bite! Or consider donating instead! That helps too.

58th issue of Andrei Neagoie's must-read monthly Python Newsletter: itertools Guide, Become an Open-Source God, and much more. Read the full newsletter to get up-to-date with everything you need to know from last month.

Three weeks ago, I attended KDE Akademy 2024 in Würzburg, Germany. It was pretty exciting to meet my KDE friends after one year since last Akademy 2023! Travel drama Ideally whole trip should’ve taken just ~18 hours door-to-door but thanks to Lufthansa whole travel turned out to be of 48 hours in total including layovers. Flight cancellation and rebooking caused by travel to start way earlier than planned (Thursday 5:00 AM instead of planned 07:00 PM) and had to spend insane amount of time in layover.

Strings can be split by a substring separator. Usually the string split is called without any arguments, which splits on any whitespace.

Table of contents

1. Breaking apart a string by a separator
2. Splitting a specific number of times
3. Splitting from the end of a string
4. The string split method versus regular expressions

Breaking apart a string by a separator

If you need to break a string into smaller strings based on a separator, you can use the string split method:

>>> time = "1:19:48" >>> time.split(":") ['1', '19', '48']

The separator you split by can be any string. It doesn't need to be just one character:

>>> graph = "A->B->C->D" >>> graph.split("->") ['A', 'B', 'C', 'D']

Note that it's a little bit unusual to call the string split method on a single space character:

>>> langston = "Does it dry up\nlike a raisin in the sun?\n" >>> langston.split(" ") ['Does', 'it', 'dry', 'up\nlike', 'a', 'raisin', 'in', 'the', 'sun?\n']

It's usually preferable to call split without any arguments at all:

>>> langston = "Does it dry up\nlike a raisin in the sun?\n" >>> langston.split() ['Does', 'it', 'dry', 'up', 'like', 'a', 'raisin', 'in', 'the', 'sun?']

Calling the split with no arguments will split on any consecutive whitespace characters. So we're even splitting on a new line here in between up and like (up\nlike).

Also note that the split method without any arguments removes leading and trailing whitespace (note that the last element in the list is sun? rather than sun?\n).

There's one more split feature that's often overlooked: maxsplit.

Splitting a specific number of times

When calling split with a …

Read the full article: https://www.pythonmorsels.com/string-split-method/

Through the kindness of the DAG Hamilton project team, I was able to secure an official svg version of the DAG Hamilton logo. It looks significantly better than the one I had generated with an online image to svg converter and is much smaller and easy to work with (4 kilobytes versus 200 kb). The DAG Hamilton graphviz graph now shows up in Blogger; it is unlikely to show up on the planet(python) feed. Blogger is not liking the code and svg I have included (complaints of malformed html). In the interest of preserving the rendering of the graph(s), I am constraining the text here to a few paragraphs

The first graph has the code provided. This graph is from a previous post.

The second graph represents the DAG Hamilton workflow for the production of the first graph. This is in keeping with the "Eat your own dogfood" mantra. I happen to like the DAG Hamilton dogfood as I've mentioned in previous posts. It allows me to visualize my workflows and track complexity and areas for improvement in the code.

The third one I did with a scaled down version of the code presented (no logos). I hand pasted the DAG Hamilton official logo into the third one. It is not subtle (the logo is huge), but it provides an idea of what one can do creatively with the logo or any svg element. Also, it shows the DAG Hamilton workflow for the graph with the respective two logos.

All the code is a work in progress. Ideally I would like to keep reducing this to the most simple svg implementation possible to get it to show up or "work." Realistically, I'm afraid to sneeze for fear Blogger will protest. For now, I'm leaving good enough alone. Links and thoughts on svg (there is at least one python library (orsinium-labs/svg.py) out there that is way more elegant in its treatment of the medium than my rough regular expressions / text processing) will have to wait for another post.

Thanks for stopping by.

Toy Web Scraping Script Run Diagram Web Scraping Functions Highlighted Legend datafile str commodity_word_counts dict info_dict_merged dict colloquial_company_word_counts dict data_with_wikipedia dict data_with_company dict parsed_data dict wikipedia_report str info_output str input function

run.py code

""" Hamilton wrapper. """ # run.py import sys import pprint from hamilton import driver import dag_hamilton_to_blogger as dhtb dr = driver.Builder().with_modules(dhtb).build() dr.display_all_functions('dhtb.svg', deduplicate_inputs=True, keep_dot=True, orient='BR') results = dr.execute(['defluffed_lines', 'scale_and_translation', 'logo_positions', 'captured_values', 'scaled_elements', 'translated_elements', 'hamilton_logo_data', 'scale_and_translation_hamilton_logo', 'fauxcompany_logo_data', 'scale_and_translation_fauxcompany_logo', 'svg_ready_doc', 'written_svg'], inputs={'svg_file':'web_scraping_functions_highlighted.svg', 'outputfile':'test_output.svg', 'hamiltonlogofile':'hamilton_official_stripped.svg', 'hamiltonlogo_coords':{'min_x':-0.001, 'max_x':4353.846, 'min_y':-0.0006, 'max_y':4177.257}, 'fauxcompanylogofile':'fauxcompanylogo_stripped_down.svg', 'fauxcompanylogo_coords':{'min_x':11.542786063261742, 'max_x':705.10684, 'min_y':4.9643821, 'max_y':74.47416391682819}})

Main DAG Hamilton functions (dag_hamilton_to_blogger.py)

# python 3.12 """ Make DAG Hamilton graph show up in Blogger. """ import re import sys import pprint import math import copy import reusedfunctions as rf VIEWBOX_PAT = (r'[ ]viewBox[=]["][-]?[0-9]+[.]?[0-9]*[ ][-]?[0-9]+[.]?[0-9]*[ ]' r'([0-9]+[.]?[0-9]*)[ ]([0-9]+[.]?[0-9]*)') # 5 coordinates. POLYGON_PAT = (r'[]polygon' r'.*([ ]points[=]["])([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)["]') # 4 coordinates instead of 5. POLYGON_PAT_4 = (r'[]polygon' r'.*([ ]points[=]["])([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)[ ]' r'([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)["]') # x, y TEXTPAT = (r']text[ ].*' r'([ ]font[-]size[=])' r'["]([0-9]+[.]?[0-9]*)["]') # initial bezier curve notation PATHPAT = (r'[]path[ ].*' r'([ ]d[=]["]M)([-]?[0-9]+[.]?[0-9]*)[,]' r'([-]?[0-9]+[.]?[0-9]*)C') X_SIZE = 600 NEW_FIRST_LINE = '') IMAGE_FLAG = '') # 4 coords (arrow head). POLYGON_STR_4 = (r' points="{0:.3f},{1:.3f} {2:.3f},{3:.3f} ' r'{4:.3f},{5:.3f} {6:.3f},{7:.3f}"/>') PATH_START_STR = r' d="M{0:.3f},{1:.3f}C' PATH_STR_SEGMENT = ' {0:.3f},{1:.3f}' PATH_STR = r' {0:s}"/>' TEXT_STR = r' x="{0:.3f}" y="{1:.3f}"' TEXT_STR_FONT = r' font-size="{0:.3f}"' HAMILTON_LOGO_DIMENSIONS_PAT = (r'.*width[=]["]([0-9]+[.]?[0-9]*)px["][ ]' r'height[=]["]([0-9]+[.]?[0-9]*)px["][>]') FAUXCOMPANY_LOGO_DIMENSIONS_PAT = (r'[ ]width[=]["]([0-9]+[.]?[0-9]*)["][ ]' r'height[=]["]([0-9]+[.]?[0-9]*)["][ ][>]') # The official Hamilton logo splits the path into multiple # lines with the last one having the absolute location # ("C") of a bezier curve. HAMILTON_CHANGE_LINE_PAT = r'.*C[-]?[0-9]+[.]?[0-9]*' HAMILTON_TRANSFORM_FMT = (' transform="scale({scale:f}) ' 'translate({translate_x:f},{translate_y:f})" />') # One line of paths in Inkscape generated file. FAUXCOMPANY_CHANGE_LINE_PAT = r'.*d[=]["]m[ ]' # Inkscape put the closing tag /> on the following line. FAUXCOMPANY_TRANSFORM_FMT = (' transform="scale({scale:f}) ' 'translate({translate_x:f},{translate_y:f})"') # * - get rid of first 6 lines. # * - get rid of any line starting with: # # "

By Volker Hilsheimer

The Qt 6.8 should be around the corner by the time of Akademy, and we'll just have had our Qt Contributors Summit. This is a great time to look at some of the things we are working on for the upcoming releases, and where we intend to put our focus in the Qt Project and Qt Company R&D teams.

By Ulf Hermann

The talk will give an overview on how the QML language has developed since Qt5. It will point out the opportunities for better performance and maintainability arising from new tooling and a more extensive type system. It will also point out some sore spots to look out for and show the direction in which we hope to develop the language going forward.

This is a good place to also discuss KDE's feature wishes for the QML language. Since a lot of those have come up lately, I will prepare some structured notes of what I've heard of so far and reserve some time for it.

By Cristián Maureira-Fredes

Since its release, Qt has been exposed to other languages in order to bring the amazing features to other communities, and to combine our beloved framework with other language-specific features.

After the success of many language bindings like Python, particularly for the PyQt and PySide projects, one can ask: "Once the language bindings are complete, is the project done?"

This talk presents the many implemented and planned features that PySide (a.k.a. Qt for Python) has, which go beyond to the known Qt API, and the motivation behind those decisions.

The goal of the presentation is to highlight a success story of bringing Qt to a completely different language, and also the lessons learned that could be used in order to improve the main Qt implementation in C++.

Attendees also will be exposed to the current project plan for future Qt releases, and new prototypes that have been discussed.

By Andy Betts

Soon after the launch of Plasma 6, many contributors requested updates for the visuals in the new Plasma Desktop. Touted as a stability release, Plasma 6 evolved to be more consistent, more bug-free than predecessors. The only thing missing from the release was a refreshed style.

Taking this feedback into consideration, a small team of designers from the team took to review and create a few exciting changes for the current visual style.

The team worked on creating:

1. A graphical design system
2. New color selection
3. New font selection with specific sizes
4. New grids and spacing system
5. New editing workflow using Figma and Penpot
6. Updating all 22px icons to a 24px size
7. New shadows and blurs
8. In addition to these, new components such as buttons, dropdowns, toggles, checkboxes, tooltips, progress indicators, sliders, badges, inputs.

We would like to provide a preview to the community of all of these changes and gauge interest. We would like to request the developer community for their help.

We also would like to follow up with a couple of BoFs to see how we could execute some of these elements given our constraints.

Above all, we believe in moving our visual style forward. We want to give our users a consistent look that helps them achieve the most they can using our systems.

Many thanks to Helden Hoierman, @manueljilin, @PhilipB, @depman, @Akseli, @Natalie Clarius, @nathanu, @DSLitvinov who have contributed so much to this project.