Feeds

pre { white-space: pre; overflow-x: auto; font-size: 80%; }

Let’s speed up some software! Our motivation: we have an image, a photo of some text from a book. We want to turn it into a 1-bit image, with just black and white, extracting the text so we can easily read it.

We’ll use an example image from scikit-image, an excellent image processing library:

from skimage.data import page import numpy as np IMAGE = page() assert IMAGE.dtype == np.uint8

Here’s what it looks like (it’s licensed under this license):

Median-based local thresholding

The task we’re trying to do—turning darker areas into black, and lighter areas into white—is called thresholding. Since the image is different in different regions, with some darker and some lighter, we’ll get the best results if we use local thresholding, where the threshold is calculated from the pixel’s neighborhood.

Simplifying somewhat, for each pixel in the image we will:

1. Calculate the median of the surrounding neighborhood.
2. Subtract a magic constant from the calculated median to calculate our local threshold.
3. If the pixel’s value is bigger than the threshold, the result is white, otherwise it’s black.

scikit-image includes an implementation of this algorithm. Here’s how we use it:

from skimage.filters import threshold_local def skimage_median_local_threshold(img, neighborhood_size, offset): threshold = threshold_local( img, block_size=neighborhood_size, method="median", offset=offset ) result = (img > threshold).astype(np.uint8) result *= 255 return result # The neighborhood size and offset value were determined "empirically", i.e. # they're manually tuning the algorithm to work well with our specific # example image. SKIMAGE_RESULT = skimage_median_local_threshold(IMAGE, 11, 10)

And here’s what the results look like:

Let’s see if we can make this faster!

Step 1. Reimplement our own version

We’re going to be using the Numba compiler, which lets us compile Python code to machine code at runtime. Here’s an initial implementation of the algorithm; it’s not quite identical to the original, for example the way edge pixels are handled, but it’s close enough for our purposes:

from numba import jit @jit def median_local_threshold1(img, neighborhood_size, offset): # Neighborhood size must be an odd number: assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) # For every pixel: for i in range(img.shape[0]): # Calculate the Y borders of the neighborhood: min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) for j in range(img.shape[1]): # Calculate the X borders of the neighborhood: min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # Calculate the median: median = np.median(img[min_y:max_y, min_x:max_x]) # Set the image to black or white, depending how it relates to # the threshold: if img[i, j] > median - offset: # White: result[i, j] = 255 else: # Black: result[i, j] = 0 return result NUMBA_RESULT1 = median_local_threshold1(IMAGE, 11, 10)

Here’s the resulting image; it looks similar enough that for our purposes:

Now we can compare the performance of the two implementations:

Code Elapsed milliseconds skimage_median_local_threshold(IMAGE, 11, 10) 76 median_local_threshold1(IMAGE, 11, 10) 87

It’s slower. But that’s OK, we’re just getting started.

Step 2: A faster implementation of the median algorithm

Calculating a median is pretty expensive, and we’re doing it for every single pixel, so let’s see if we can speed it up.

The generic median implementation Numba provides is likely to be fairly generic, since it needs to work in a wide variety of circumstances. We can hypothesize that it’s not optimized for our particular case. And even if it is, having our own implementation will allow for a second round of optimization, as we’ll see in the next step.

We’re going to implement a histogram-based median, based on the fact we’re using 8-bit images that only have a limited range of potential values. The median is the value where 50% of the pixels’ values are smaller, and 50% are bigger.

Here’s the basic algorithm for a histogram-based median:

• Each pixel’s value will go into a different bucket in the histogram; since we know our image is 8-bit, we only need 256 buckets.
• Then, we add up the size of each bucket in the histogram, from smallest to largest, until we hit 50% of the pixels we inspected.

@jit def median_local_threshold2(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) # 😎 A histogram with a bucket for each of the 8-bit values possible in # the image. We allocate this once and reuse it. histogram = np.empty((256,), dtype=np.uint32) for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # Reset the histogram to zero: histogram[:] = 0 # Populate the histogram, counting how many of each value are in # the neighborhood we're inspecting: neighborhood = img[min_y:max_y, min_x:max_x].ravel() for k in range(len(neighborhood)): histogram[neighborhood[k]] += 1 # Use the histogram to find the median; keep adding buckets until # we've hit 50% of the pixels. The corresponding bucket is the # median. half_neighborhood_size = len(neighborhood) // 2 for l in range(256): half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: break median = l if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT2 = median_local_threshold2(IMAGE, 11, 10)

Here’s the resulting image:

And here’s the performance of our new implementation:

Code Elapsed milliseconds median_local_threshold1(IMAGE, 11, 10) 86 median_local_threshold2(IMAGE, 11, 10) 18

That’s better!

Step 3: Stop recalculating the histogram from scratch

Our algorithm uses a rolling neighborhood or window over the image, calculating the median for a window around each pixel. And the neighborhood for one pixel has a significant overlap for the neighborhood of the next pixel. For example, let’s say we’re looking at a neighborhood size of 3. We might calculate the median of this area:

...... .\\\.. .\\\.. .\\\.. ...... ......

And then when process the next pixel we’ll calculate the median of this area:

...... ..///. ..///. ..///. ...... ......

If we superimpose them, we can see there’s an overlap, the X:

...... .\XX/. .\XX/. .\XX/. ...... ......

Given the histogram for the first pixel, if we remove the values marked with \ and add the ones marked with /, we’ve calculated the exact histogram for the second pixel. So for a 3×3 neighborhood, instead of processing 3 columns we process 2, a minor improvement. For a 11×11 neighborhood, we will go from processing 11 columns to 2 columns, a much more significant improvement.

Here’s what the code looks like:

@jit def median_local_threshold3(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) histogram = np.empty((256,), dtype=np.uint32) for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) # Populate histogram as if we started one pixel to the left: histogram[:] = 0 initial_neighborhood = img[min_y:max_y, 0:radius].ravel() for k in range(len(initial_neighborhood)): histogram[initial_neighborhood[k]] += 1 for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # 😎 Instead of recalculating histogram from scratch, re-use the # previous pixel's histogram. # Substract left-most column we don't want anymore: if min_x > 0: for y in range(min_y, max_y): histogram[img[y, min_x - 1]] -= 1 # Add new right-most column: if max_x < img.shape[1]: for y in range(min_y, max_y): histogram[img[y, max_x - 1]] += 1 # Find the the median from the updated histogram: half_neighborhood_size = ((max_y - min_y) * (max_x - min_x)) // 2 for l in range(256): half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: break median = l if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT3 = median_local_threshold3(IMAGE, 11, 10)

Here’s the resulting image:

And here’s the performance of our latest code:

Code Elapsed microseconds median_local_threshold2(IMAGE, 11, 10) 17,066 median_local_threshold3(IMAGE, 11, 10) 6,386 Step #4: Adapative heuristics

Notice that a median’s definition is symmetrical:

1. The first value that is smaller than the highest 50% values.
2. Or, the first value that is larger than the lowest 50% values. We used this definition in our code above, adding up buckets from the smallest to the largest.

Depending on the distribution of values, one approach to adding up buckets to find the median may be faster than the other. For example, given a 0-255 range, if the median is going to be 10 we want to start from the smallest bucket to minimize additions. But if the median is going to be 200, we want to start from the largest bucket.

So which side we should start from? One reasonable heuristic is to look at the previous median we calculated, which most of the time will be quite similar to the new median. If the previous median was small, start from the smallest buckets; if it was large, start from the largest buckets.

@jit def median_local_threshold4(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) histogram = np.empty((256,), dtype=np.uint32) median = 0 for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) histogram[:] = 0 initial_neighborhood = img[min_y:max_y, 0:radius].ravel() for k in range(len(initial_neighborhood)): histogram[initial_neighborhood[k]] += 1 for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) if min_x > 0: for y in range(min_y, max_y): histogram[img[y, min_x - 1]] -= 1 if max_x < img.shape[1]: for y in range(min_y, max_y): histogram[img[y, max_x - 1]] += 1 half_neighborhood_size = ((max_y - min_y) * (max_x - min_x)) // 2 # 😎 Find the the median from the updated histogram, choosing # the starting side based on the previous median; we can go from # the leftmost bucket to the rightmost bucket, or in reverse: the_range = range(256) if median < 127 else range(255, -1, -1) for l in the_range: half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: median = l break if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT4 = median_local_threshold4(IMAGE, 11, 10)

The end result is 25% faster. Since the heuristic is tied to the image contents, the performance impact will depend on the image.

Code Elapsed microseconds median_local_threshold3(IMAGE, 11, 10) 6,381 median_local_threshold4(IMAGE, 11, 10) 4,920 The big picture

Here’s a performance comparison of all the versions of the code:

Code Elapsed microseconds skimage_median_local_threshold(IMAGE, 11, 10) 76,213 median_local_threshold1(IMAGE, 11, 10) 86,494 median_local_threshold2(IMAGE, 11, 10) 17,145 median_local_threshold3(IMAGE, 11, 10) 6,398 median_local_threshold4(IMAGE, 11, 10) 4,925

Let’s go over the steps we went through:

1. Switch to a compiled language: this gives us more control.
2. Reimplement the algorithm taking advantage of constrained requirements: our median only needed to handle uint8, so a histogram was a reasonable solution.
3. Reuse previous calculations to prevent repetition: our histogram for the neighborhood of a pixel is quite similar to that of the previous pixel. This means we can reuse some of the calculations.
4. Adaptively tweak the algorithm at runtime: as we run on an actual image, we use what we’ve learned up to this point to hopefully run faster later on. The decision from which side of the histogram to start is arbirary in general. But in this specific algorithm, the overlapping pixel neighborhoods mean we can make a reasonable guess.

This process demonstrates part of why generic libraries may be slower than custom code you write for your particular use case and your particular data.

Next steps

What else can you do to speed up this algorithm? Here are some ideas:

• There may be a faster alternative to histogram-based medians.
• We’re not fully taking advantage of histogram overlap; there’s also overlap between rows.
• The cumulative sum in the histogram doesn’t benefit from instruction-level parallelism or SIMD. It’s possible that using one of those would result in faster results even if it uses more instructions.
• So far the code has only used a single CPU. Given each row is calculated independently, parallelism would probably work well if done in horizontal stripes, probably taller than one pixel so as to maximize utilization of memory caches.

Want to learn more about optimizing compiled code for Python data processing? This article is an extract from a book I’m working on; test readers are currently going through initial drafts. Aimed at Python developers, data scientists, and scientists, the book covers topics like instruction-level parallelism, memory caches, and other performance optimization techniques. Learn more and sign up to get updates here.

Read more...

Visit any GitHub project page, and the first thing you see is something that looks like this:

“Code”, that’s fairly innocuous, and it’s what we came here for. The “Issues” and “Pull Requests” tabs, with their count of open issues, might give us some sense of “how active” the project is, or perhaps “how maintained”. Useful information for the casual visitor, undoubtedly.

However, there’s another user community that visits this page on the regular, and these same tabs mean something very different to them.

I’m talking about the maintainers (or, more commonly, maintainer, singular). When they see those tabs, all they see is work. The “Code” tab is irrelevant to them – they already have the code, and know it possibly better than they know their significant other(s) (if any). “Issues” and “Pull Requests” are just things that have to be done.

I know for myself, at least, that it is demoralising to look at a repository page and see nothing but work. I’d be surprised if it didn’t contribute in some small way to maintainers just noping the fudge out.

A Modest Proposal

So, here’s my thought. What if instead of the repo tabs looking like the above, they instead looked like this:

My conception of this is that it would, essentially, be a kind of “yearbook”, that people who used and liked the software could scribble their thoughts on. With some fairly straightforward affordances elsewhere to encourage its use, it could be a powerful way to show maintainers that they are, in fact, valued and appreciated.

There are a number of software packages I’ve used recently, that I’d really like to say a general “thanks, this is awesome!” to. However, I’m not about to make the Issues tab look even scarier by creating an “issue” to say thanks, and digging up an email address is often surprisingly difficult, and wouldn’t be a public show of my gratitude, which I believe is a valuable part of the interaction.

You Can’t Pay Your Rent With Kudos

Absolutely you cannot. A means of expressing appreciation in no way replaces the pressing need to figure out a way to allow open source developers to pay their rent. Conversely, however, the need to pay open source developers doesn’t remove the need to also show those people that their work is appreciated and valued by many people around the world.

Anyway, who knows a senior exec at GitHub? I’ve got an idea I’d like to run past them…

I am getting increasingly frustrated by Fira Mono's lack of italic support so I am looking at alternative fonts again.

Commit Mono

This time I seem to be settling on either Commit Mono or Space Mono. For now I'm using Commit Mono because it's a little more compressed than Fira and does have a italic version. I don't like how Space Mono's parenthesis (()) is "squarish", it feels visually ambiguous with the square brackets ([]), a big no-no for my primary use case (code).

So here I am using a new font, again. It required changing a bunch of configuration files in my home directory (which is in a private repository, sorry) and Emacs configuration (thankfully that's public!).

One gotcha is I realized I didn't actually have a global font configuration in Emacs, as some Faces define their own font family, which overrides the frame defaults.

This is what it looks like, before:

Fira Mono

After:

Commit Mono

(Notice how those screenshots are not sharp? I'm surprised too. The originals look sharp on my display, I suspect this is something to do with the Wayland transition. I've tried with both grim and flameshot, for what its worth.)

They are pretty similar! Commit Mono feels a bit more vertically compressed maybe too much so, actually -- the line height feels too low. But it's heavily customizable so that's something that's relatively easy to fix, if it's really a problem. Its weight is also a little heavier and wider than Fira which I find a little distracting right now, but maybe I'll get used to it.

All characters seem properly distinguishable, although, if I'd really want to nitpick I'd say the © and ® are too different, with the latter (REGISTERED SIGN) being way too small, basically unreadable here. Since I see this sign approximately never, it probably doesn't matter at all.

I like how the ampersand (&) is more traditional, although I'll miss the exotic one Fira produced... I like how the back quotes (`, GRAVE ACCENT) drop down low, nicely aligned with the apostrophe. As I mentioned before, I like how the bar on the "f" aligns with the other top of letters, something in Fira mono that really annoys me now that I've noticed it (it's not aligned!).

A UTF-8 test file

Here's the test sheet I've made up to test various characters. I could have sworn I had a good one like this lying around somewhere but couldn't find it so here it is, I guess.

US keyboard coverage: abcdefghijklmnopqrstuvwxyz`1234567890-=[]\;',./ ABCDEFGHIJKLMNOPQRSTUVWXYZ~!@#$%^&*()_+{}|:"<>? latin1 coverage: ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ EURO SIGN, TRADE MARK SIGN: €™ ambiguity test: e¢coC0ODQ iI71lL!|¦ b6G&0B83 [](){}/\.…·• zs$S52Z% ´`'"‘’“”«» all characters in a sentence, uppercase: the quick fox jumps over the lazy dog THE QUICK FOX JUMPS OVER THE LAZY DOG same, in french: voix ambiguë d'un cœur qui, au zéphyr, préfère les jattes de kiwis. VOIX AMBIGUË D'UN CŒUR QUI, AU ZÉPHYR, PRÉFÈRE LES JATTES DE KIWIS. Ligatures test: -<< -< -<- <-- <--- <<- <- -> ->> --> ---> ->- >- >>- =<< =< =<= <== <=== <<= <= => =>> ==> ===> =>= >= >>= <-> <--> <---> <----> <=> <==> <===> <====> :: ::: __ <~~ </ </> /> ~~> == != /= ~= <> === !== !=== =/= =!= <: := *= *+ <* <*> *> <| <|> |> <. <.> .> +* =* =: :> (* *) /* */ [| |] {| |} ++ +++ \/ /\ |- -| <!-- <!--- Box drawing alignment tests: █ ╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▉ ╱╲╱╲╳╳╳ ║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▊ ╲╱╲╱╳╳╳ ║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▋ ╱╲╱╲╳╳╳ ╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▌ ╲╱╲╱╳╳╳ ║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▍ ║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎ ╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ └╌╌┘ ╎ ┗╍╍┛ ┋ ▏▁▂▃▄▅▆▇█ Dashes alignment test: HYPHEN-MINUS, MINUS SIGN, EN, EM DASH, HORIZONTAL BAR, LOW LINE -------------------------------------------------- −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− –––––––––––––––––––––––––––––––––––––––––––––––––– —————————————————————————————————————————————————— ―――――――――――――――――――――――――――――――――――――――――――――――――― __________________________________________________

So there you have it, got completely nerd swiped by typography again. Now I can go back to writing a too-long proposal again.

Sources and inspiration for the above:

• the unicode(1) command, to lookup individual characters to disambiguate, for example, - (U+002D HYPHEN-MINUS, the minus sign next to zero on US keyboards) and − (U+2212 MINUS SIGN, a math symbol)

• searchable list of characters and their names - roughly equivalent to the unicode(1) command, but in one page, amazingly the /usr/share/unicode database doesn't have any one file like this

• bits/UTF-8-Unicode-Test-Documents - full list of UTF-8 characters

• UTF-8 encoded plain text file - nice examples of edge cases, curly quotes example and box drawing alignment test which, incidentally, showed me I needed specific faces customisation in Emacs to get the Markdown code areas to display properly, also the idea of comparing various dashes

• sample sentences in many languages - unused, "Sentences that contain all letters commonly used in a language"

• UTF-8 sampler - unused, similar

Other fonts

In my previous blog post about fonts, I had a list of alternative fonts, but it seems people are not digging through this, so I figured I would redo the list here to preempt "but have you tried Jetbrains mono" kind of comments.

My requirements are:

• no ligatures: yes, in the previous post, I wanted ligatures but I have changed my mind. after testing this, I find them distracting, confusing, and they often break the monospace nature of the display
• monospace: this is to display code
• italics: often used when writing Markdown, where I do make use of italics... Emacs falls back to underlining text when lacking italics which is hard to read
• free-ish, ultimately should be packaged in Debian

Here is the list of alternatives I have considered in the past and why I'm not using them:

• agave: recommended by tarzeau, not sure I like the lowercase a, a bit too exotic, packaged as fonts-agave

• Cascadia code: optional ligatures, multilingual, not liking the alignment, ambiguous parenthesis (look too much like square brackets), new default for Windows Terminal and Visual Studio, packaged as fonts-cascadia-code

• Fira Code: ligatures, was using Fira Mono from which it is derived, lacking italics except for forks, interestingly, Fira Code succeeds the alignment test but Fira Mono fails to show the X signs properly! packaged as fonts-firacode

• Hack: no ligatures, very similar to Fira, italics, good alternative, fails the X test in box alignment, packaged as fonts-hack

• Hermit: no ligatures, smaller, alignment issues in box drawing and dashes, packaged as fonts-hermit somehow part of cool-retro-term

• IBM Plex: irritating website, replaces Helvetica as the IBM corporate font, no ligatures by default, italics, proportional alternatives, serifs and sans, multiple languages, partial failure in box alignment test (X signs), fancy curly braces contrast perhaps too much with the rest of the font, packaged in Debian as fonts-ibm-plex

• Intel One Mono: nice legibility, no ligatures, alignment issues in box drawing, not packaged in Debian

• Iosevka: optional ligatures, italics, multilingual, good legibility, has a proportional option, serifs and sans, line height issue in box drawing, fails dash test, not in Debian

• Jetbrains Mono: (mandatory?) ligatures, good coverage, originally rumored to be not DFSG-free (Debian Free Software Guidelines) but ultimately packaged in Debian as fonts-jetbrains-mono

• Monoid: optional ligatures, feels much "thinner" than Jetbrains, not liking alignment or spacing on that one, ambiguous 2Z, problems rendering box drawing, packaged as fonts-monoid

• Mononoki: no ligatures, looks good, good alternative, suggested by the Debian fonts team as part of fonts-recommended, problems rendering box drawing, em dash bigger than en dash, packaged as fonts-mononoki

• Source Code Pro: italics, looks good, but dash metrics look whacky, not in Debian

• spleen: bitmap font, old school, spacing issue in box drawing test, packaged as fonts-spleen

• sudo: personal project, no ligatures, zero originally not dotted, relied on metrics for legibility, spacing issue in box drawing, not in Debian

So, if I get tired of Commit Mono, I might probably try, in order:

1. Hack
2. Jetbrains Mono
3. IBM Plex Mono

Iosevka, Monoki and Intel One Mono are also good options, but have alignment problems. Iosevka is particularly disappointing as the EM DASH metrics are just completely wrong (much too wide).

This was tested using the Programming fonts site which has all the above fonts, which cannot be said of Font Squirrel or Google Fonts, amazingly. Other such tools:

• Coding Font (broken in Firefox as of 2024-05-30)
• dev fonts comparator
• Font Squirrel
• Google Fonts
• Programming fonts

meta title="Playing with fonts again"

I am getting increasingly frustrated by Fira Mono's lack of italic support so I am looking at alternative fonts again.

This time I seem to be settling on either Commit Mono or Space Mono. For now I'm using Commit Mono because it's a little more compressed than Fira and does have a italic version. I don't like how Space Mono's parenthesis (()) is "squarish", it feels visually ambiguous with the square brackets ([]), a big no-no for my primary use case (code).

So here I am using a new font, again. It required changing a bunch of configuration files in my home directory (which is in a private repository, sorry) and Emacs configuration (thankfully that's public!).

One gotcha is I realized I didn't actually have a global font configuration in Emacs, as some Faces define their own font family, which overrides the frame defaults.

This is what it looks like, before:

Fira Mono

After:

Commit Mono

(Notice how those screenshots are not sharp? I'm surprised too. The originals look sharp on my display, I suspect this is something to do with the Wayland transition. I've tried with both grim and flameshot, for what its worth.)

They are pretty similar! Commit Mono feels a bit more vertically compressed maybe too much so, actually -- the line height feels too low. But it's heavily customizable so that's something that's relatively easy to fix, if it's really a problem. Its weight is also a little heavier and wider than Fira which I find a little distracting right now, but maybe I'll get used to it.

All characters seem properly distinguishable, although, if I'd really want to nitpick I'd say the © and ® are too different, with the latter (REGISTERED SIGN) being way too small, basically unreadable here. Since I see this sign approximately never, it probably doesn't matter at all.

I like how the ampersand (&) is more traditional, although I'll miss the exotic one Fira produced... I like how the back quotes (`, GRAVE ACCENT) drop down low, nicely aligned with the apostrophe. As I mentioned before, I like how the bar on the "f" aligns with the other top of letters, something in Fira mono that really annoys me now that I've noticed it (it's not aligned!).

Here's the test sheet I've made up to test various characters. I could have sworn I had a good one like this lying around somewhere but couldn't find it so here it is, I guess.

ASCII test abcdefghijklmnopqrstuvwxyz1234567890-= ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*()_+ ambiguous characters &iIL7l1!|[](){}/\oO0DQ8B;:,./?~`'"$ all characters in a sentence, uppercase the quick fox jumps over the lazy dog THE QUICK FOX JUMPS OVER THE LAZY DOG same, in french voix ambiguë d'un cœur qui, au zéphyr, préfère les jattes de kiwis. VOIX AMBIGUË D'UN CŒUR QUI, AU ZÉPHYR, PRÉFÈRE LES JATTES DE KIWIS. Box drawing alignment tests: █ ▉ ╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▊ ╱╲╱╲╳╳╳ ║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▋ ╲╱╲╱╳╳╳ ║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▌ ╱╲╱╲╳╳╳ ╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▍ ╲╱╲╱╳╳╳ ║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎ ║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▏ ╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ └╌╌┘ ╎ ┗╍╍┛ ┋ ▁▂▃▄▅▆▇█ MIDDLE DOT, BULLET, HORIZONTAL ELLIPSIS: ·•… curly ‘single’ and “double” quotes ACUTE ACCENT, GRAVE ACCENT: ´` EURO SIGN: € unicode A1-BF: ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ HYPHEN-MINUS, MINUS SIGN, EN, EM DASH, HORIZONTAL BAR, LOW LINE -------------------------------------------------- −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− –––––––––––––––––––––––––––––––––––––––––––––––––– —————————————————————————————————————————————————— ―――――――――――――――――――――――――――――――――――――――――――――――――― __________________________________________________

So there you have it, got completely nerd swiped by typography again. Now I can go back to writing a too-long proposal again.

Sources and inspiration for the above:

• the unicode(1) command, to lookup individual characters to disambiguate, for example, - (U+002D HYPHEN-MINUS, the minus sign next to zero on US keyboards) and − (U+2212 MINUS SIGN, a math symbol)

• searchable list of characters and their names - roughly equivalent to the unicode(1) command, but in one page, amazingly the /usr/share/unicode database doesn't have any one file like this

• bits/UTF-8-Unicode-Test-Documents - full list of UTF-8 characters

• UTF-8 encoded plain text file - nice examples of edge cases, curly quotes example and box drawing alignment test which, incidentally, showed me I needed specific faces customisation in Emacs to get the Markdown code areas to display properly, also the idea of comparing various dashes

• sample sentences in many languages - unused, "Sentences that contain all letters commonly used in a language"

• UTF-8 sampler - unused, similar

This episode focuses on the Harlequin application, a Python SQL IDE for your terminal written using the amazing Textual package.

I was honored to have Ted Conbeer, the creator of Harlequin, on the show to discuss his creation and the other things he does with Python.

Specifically, we focused on the following topics:

• Favorite Python packages
• Origins of Harlequin
• Why program for the terminal versus a GUI
• Lessons learned in creating the tool
• Asyncio
• and more!

Links

• Harlequin
• Textual
• BBC article on asyncio

The post Episode 42 – Harlequin – The SQL IDE for Your Terminal appeared first on Mouse Vs Python.

As of today, DEP-14 has been approved 🛫

The DEP was written and stewarded by Jake Howard. A very enthusiastic community has been active with feedback and encouragement, while the Django Steering Council gave the final inputs before its formal acceptance. The implementation of DEP-14 is expected to be a major leap forward for the “batteries included” philosophy of Django.

Whilst Django is a web framework, there's more to web applications than just the request-response lifecycle. Sending emails, communicating with external services or running complex actions should all be done outside the request-response cycle.

Django doesn't have a first-party solution for long-running tasks, however the ecosystem is filled with incredibly popular frameworks, all of which interact with Django in slightly different ways. Other frameworks such as Laravel have background workers built-in, allowing them to push tasks into the background to be processed at a later date, without requiring the end user to wait for them to occur.

Library maintainers must implement support for any possible task backend separately, should they wish to offload functionality to the background. This includes smaller libraries, but also larger meta-frameworks with their own package ecosystem such as Wagtail.

This proposal sets out to provide an interface and base implementation for long-running background tasks in Django.

• Read the full DEP-14 text
• Pull Request review

Future work

The DEP will now move on to the Implementation phase before being merged into Django itself.

If you would like to help or try it out, go have a look at django-tasks, a separate reference implementation by Jake Howard, the author of the DEP.

Jake will also be speaking about the DEP in his talk at DjangoCon Europe at DjangoCon Europe 2024 in Vigo next week.

Back in March I started to look at sponsors for the time I’m spending working on the Drupal core issue queue. It’s been a few months and I wanted to go back on all the sponsored commits I made as a Frontend Framework Manager, to show how the sponsorships helped Drupal for the past few months.

The sponsorship offer is simple: you send me a fixed monthly fee of 2500€, and I share the issue credit of every Drupal core commit that I make. I’m very thankful to Palantir.net and OPTASY who are sponsoring me. Thanks to them I was able to increase the amount of commits I can make to Drupal core. In the last 3 months I committed 61 issues (worth 610 weighted issue credits) and the more sponsors I have, the more time I can spend reviewing and committing issues.

1. Differentiate visually dragging with and without hierarchy A nice improvement for editors working a lot with lists and trees
2. Sticky table header is not sticky if --drupal-displace-offset-top is not defined
3. [jQuery 4] ajax.js and jquery.form.js use deprecated function $.parseJSON() Preparing for the next release of jQuery 4 with some cleanup.
4. cspell check is broken in commit-code-check.sh Sometimes we break the CI and it needs to be fixed 🤷
5. CKEditor admin toolbar config buttons using ::before to add content: have invalid screen reader text It takes dedication to land those accessibility fixes, kudos to our accessibility contributors.
6. Linking in CKEditor 5: URLs with top-level domain but without protocol should get a protocol added automatically
7. #states disable property has stopped working for submit button and other elements Sometimes when we clean-up code, we clean too much and break some other parts of the code
8. Setting width for sticky-header is broken
9. Negotiate max width/height of oEmbed assets more intelligently
10. States API doesn't work with multiple select fields This was a a 13 years old issue! It _always_ feels good to close an issue that old.
11. Add deprecation/bc support for library-overrides when files are moved Making sure backwards compatibility is working and useful
12. Remove default event from collpased nav-tabs button
13. [DrupalHtmlEngine] HTML-reserved characters (>, <, &) in <script> and <style> tag are converted to HTML entities It happens that we fix things for uses cases that stretch the reasonable (like having whole script tags in a WYSIWYG field)
14. Olivero: Show content preview checkbox is not center aligned with the layout builder buttons. Even a minor issue of a misaligned text by a few pixels is worth fixing
15. Drupal.theme.progressBar() does not escape output correctly
16. filter_autop should ignore twig.config debug html comments Making sure Developer experience doesn't impact regular users
17. tablePositionSticky should not be called on a non-array variable
18. CKEditor 5 table cell vertical align "middle" doesn't work
19. Move system/base component CSS to respective libraries where they exist A surprising performance improvement. There are still some low hanging fruits to improve the default frontend performance of Drupal
20. Remove country setting from the installer When you don't need a piece of data, just don't collect it
21. Media Library widget display doesn't return to first page on applying filters
22. Deprecate and remove the AJAX replace method That was a leftover D7 era deprecation
23. Claro should use libraries-extend for views_ui.css Even in core it happens that we don't use the right way to do something
24. Removal :tabbable usage in dialog.js Some more jQuery 4 preparation
25. Close icon is ovrlapping the title text in modal in claro Yes, typos can make it in the commit log
26. Convert Olivero's teaser into a single directory component Slowly but surely we're adding Single directory components to Drupal core
27. Refactor (if feasible) uses of the jQuery animate function to use Vanilla/native More CSS awesomeness making JavaScript code disapear
28. [11.x] Update to jQuery 4.0.x beta Drupal staying on the bleading edge of frontend developement :)
29. Refactor some uses of the jQuery parents function to use vanillaJS Did a small post earlier about this, CSS is really very good
30. [regression] Uncaught TypeError: Cannot read properties of null (reading 'style') (toolbar.js)
31. JSDoc for ajax command "changed" is incorrect There was a bunch of documentation fixes around this time
32. menu_heading_id variable is not set in menu-region--footer.html.twig
33. Add @file documentation to navigation.html.twig layout template
34. Add @file documentation to menu-region--footer.html.twig template
35. Views UI action buttons create janky layout shift on page load Polishing the loading of pages with heavy JS usage is important to show we care about UX
36. Remove bottom radius on hover state of expanded sub menu item
37. Setting empty URL when making embedded media a link in CKEditor5 causes JS errors
38. Update color of submenu title text
39. Collapsed nav-tabs status not exposed to screen reader There is a good number of accessibility fixes after this one, always nice to commit
40. Incorrect padding on child menu items
41. Claro: Form labels that are disabled have too low color contrast
42. Claro should not hardcode decimal list style type for <ol>
43. Some of string comparisons should use String.prototype.startsWith() or String.prototype.endsWith() Removing regular expressions as much as we can is a noble goal
44. Location of "Skip to Main" link below admin toolbar in Claro is problematic for screen magnifier users
45. Focus states on mobile second level navigation items can get cut off in Olivero
46. Regression: Shortcuts menu flickers when the page is loaded Those toolbar flickering issues are tricky. Thankfully the new navigation module code is simpler than the existing Toolbar code, so it's much much easier to deal with
47. escapeAdmin.js functionality should be removed(it not used anymore) Removed the feature that removed the overlay… for now, escapeAdmin will be back one way or another)
48. Navigation module offsets the Olivero skip link element
49. Umami page.tpl.php breaks block placeholders Sometime themes can break really nice Drupal features (like bigpipe)
50. Claro CSS for dropbutton items adds large gap of white space
51. Replace dialogContentResize jQuery event with CustomEvent Those events issues are really exciting, we're moving away slowly from jQuery for event management
52. Umami views should use responsive grid Another case of core not using the awesome features we provide, not anymore :)
53. Claro highlighted row not communicated to keyboard users
54. Fix overflow visibility for wrapper content in navigation CSS
55. Claro details component does not have the right class
56. Make drupal.tableheader only use CSS for sticky table headers I will always welcome CSS-removing-JS patches
57. Mismatch between implementation and description for Drupal.Message.prototype.remove().
58. "Skip to main content" link skips over content that is essential to the page, banner role should be for global content
59. Add pdureau as a co-maintainer for the Theme API with a focus on SDC Adding new maintainers is too rare. In this case the community is better for having him around
60. Choose an icon for the Announcements link
61. Remove deprecated moved_files entries in core

Many of these issues are maintenance focused, it’s not shiny, it’s not exciting, and it needs to be done. Sponsoring big initiatives like Starshot is exciting, let’s not forget the unexciting day to day that keeps things running. If you’re interested in supporting my work on Drupal core and keep the frontend fixes coming, consider sponsoring me.

PyCharm 2024.1.2 is here with features designed to enhance your productivity and streamline your development workflow. This update includes support for DRF viewsets and routers in the Endpoints tool window, code assistance for TypedDict and Unpack, and improved debugger performance when handling large collections.

You can download the latest version from our download page or update your current version through our free Toolbox App. 

For more details, please visit our What’s New page.

Download PyCharm 2024.1.2

Key features Support for DRF viewsets and routers in the Endpoints tool window

When working with the Django REST Framework in PyCharm, not only can you specify function-based or class-based views in the path, but you can now also specify viewsets and see the results in the Endpoints tool window. Additionally, you can map HTTP methods to viewset methods, and PyCharm will display the HTTP methods next to the relevant route, including for custom methods. Routes without @actions decorators are now displayed with the related viewset methods.

Learn more Code assistance for TypedDict and Unpack

PEP 692 made it possible to add type information for keyword arguments of different types by using TypedDict and Unpack. PyCharm allows you to use this feature confidently by providing parameter info, type checking, and code completion.

Improved debugger performance for large collections

PyCharm’s debugger now offers a smoother experience, even when very large collections are involved. You can now work on your data science projects without having to put up with high CPU loads and UI freezes.

Download PyCharm 2024.1.2

Be sure to check out our release notes to learn all of the details and ensure you don’t miss out on any new features.

We appreciate your support as we work to improve your PyCharm experience. Please report any bugs via our issue tracker so we can resolve them promptly. Connect with us on X (formerly Twitter) to share your feedback on PyCharm 2024.1.2!

Today, we will take a step back from reviewing the Migrate API. Instead, we will have an overview of content and configuration entities in Drupal 10. This is important for two reasons. Read more mauricio Wed, 05/29/2024 - 07:40

This episode focuses on the Harlequin application, a Python SQL IDE for your terminal written using the amazing Textual package.

I was honored to have Ted Conbeer, the creator of Harlequin, on the show to discuss his creation and the other things he does with Python.

Specifically, we focused on the following topics:

• Favorite Python packages

• Origins of Harlequin

• Why program for the terminal versus a GUI

• Lessons learned in creating the tool

• Asyncio

• and more!

Links

• Harlequin

• Textual

• BBC article on asyncio

We are thrilled to announce the launch of our new initiative led by Programs Manager, Joi Garrett. This program is designed to support the success of Drupal Local Associations by engaging directly with community leaders who work to promote the Drupal project in their global regions.

Connecting Communities

The heart of the Local Associations Initiative lies in fostering meaningful connections. We recognize the efforts of local leaders and the unique challenges they face. By hosting a series of virtual meetings, we aim to create a platform for leaders to share their experiences, successes and challenges. These sessions will not only provide valuable insight to the state of various local associations but help to strengthen our global community. 

Identifying and Addressing Common Needs

Understanding the diverse needs of our local associations is crucial. Through open dialogue in our virtual meetings, we will identify common needs and prioritize them. We hope by facilitating a collaborative environment the Drupal Association can support efforts for the most pressing issues faced by community leaders. The Drupal Association is committed to finding solutions that drive success. 

Join Us on This Journey

We invite local association leaders to participate in this initiative and attend the virtual meetings. Your insights and contributions are invaluable as we work together to strengthen our global Drupal Community. Stay tuned for announcements about the upcoming virtual meetings. Through this initiative, we aim to foster a collaborative environment where our global community feels more connected and supported. Once we have concluded the meetings, we will discuss the findings and future plans during DrupalCon Barcelona 2024. 

We have been collecting contact information of Local Association leaders for the past few months, if you would like to be included please fill out the following form.

Thank you to our local leaders for being an integral part of our community. We look forward to collaborating with you to make this initiative a success!

Continent Expected Start Europe April (working with Network of European Drupal Associations)  Asia June Australia July North America July South America August Africa August

CRUD operations are at the heart of nearly every application you interact with. As a developer, you usually want to create data, read or retrieve data, update data, and delete data. Whether you access a database or interact with a REST API, only when all four operations are present are you able to make a complete data roundtrip in your app.

Creating, reading, updating, and deleting are so vital in software development that these methods are widely referred to as CRUD. Understanding CRUD will give you an actionable blueprint when you build applications and help you understand how the applications you use work behind the scenes. So, what exactly does CRUD mean?

Get Your Code: Click here to download the free sample code that you’ll use to learn about CRUD operations in Python.

Take the Quiz: Test your knowledge with our interactive “What Are CRUD Operations?” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

What Are CRUD Operations?

In this quiz, you'll revisit the key concepts and techniques related to CRUD operations. These operations are fundamental to any system that interacts with a database, and understanding them is crucial for effective data management.

In Short: CRUD Stands for Create, Read, Update, and Delete

CRUD operations are the cornerstone of application functionality, touching every aspect of how apps store, retrieve, and manage data. Here’s a brief overview of the four CRUD operations:

• Create: This is about adding new entries to your database. But it’s also applicable to other types of persistent storage, such as files or networked services. When you perform a create operation, you’re initiating a journey for a new piece of data within your system.
• Read: Through reading, you retrieve or view existing database entries. This operation is as basic as checking your email or reloading a website. Every piece of information you get has been received from a database, thanks to the read operation.
• Update: Updating allows you to modify the details of data already in the database. For example, when you update a profile picture or edit a chat message. Each time, there’s an update operation at work, ensuring your new data is stored in the database.
• Delete: Deleting removes existing entries from the database. Whether you’re closing an account or removing a post, delete operations ensure that unwanted or unnecessary data can be properly discarded.

CRUD operations describe the steps that data takes from creation to deletion, regardless of what programming language you use. Every time you interact with an application, you’re likely engaging in one of the four CRUD operations.

Why Are CRUD Operations Essential?

Whether you’re working on a basic task list app or a complex e-commerce platform, CRUD operations offer a universal language for designing and manipulating data models. Knowing about CRUD as a user helps you understand what’s happening behind the curtains. As a developer, understanding CRUD provides you with a structured framework for storing data in your application with persistence:

In computer science, persistence refers to the characteristic of state of a system that outlives (persists more than) the process that created it. This is achieved in practice by storing the state as data in computer data storage. (Source)

So even when a program crashes or a user disconnects, the data is safe and can be retrieved later. This also means that the order of the operations is important. You can only read, update, or delete items that were previously created.

It’s good practice to implement each CRUD operation separately in your applications. For example, when you retrieve items, then you shouldn’t update them at the same time.

Note: An exception to this rule may be when you update a “last time retrieved” value after a read operation. Although the user performs a read CRUD operation to retrieve data, you may want to trigger an update operation in the back end to keep track of a user’s retrievals. This can be handy if you want to show the last visited posts to the user.

While CRUD describes a concept that’s independent of specific programming languages, one could argue that CRUD operations are strongly connected to SQL commands and HTTP methods.

What Are CRUD Operations in SQL?

The idea of CRUD is strongly connected with databases. That’s why it’s no surprise that CRUD operations correspond almost one-to-one with SQL commands:

CRUD Operation SQL Command Create INSERT Read SELECT Update UPDATE Delete DELETE

When you create data, you’re using the INSERT command to add new records to a table. After creation, you may read data using SELECT. With a SELECT query, you’re asking the database to retrieve the specific pieces of information you need, whether it’s a single value, a set of records, or complex relationships between data points.

The update operation corresponds to the UPDATE command in SQL, which allows you to modify data. It lets you edit or change an existing item.

Lastly, the delete operation relates to the DELETE command. This is the digital equivalent of shredding a confidential document. With DELETE, you permanently remove an item from the database.

Writing CRUD Operations in Raw SQL

CRUD operations describe actions. That’s why it’s a good idea to pull up your sleeves and write some code to explore how CRUD operations translate into raw SQL commands.

In the examples below, you’ll use Python’s built-in sqlite3 package. SQLite is a convenient SQL library to try things out, as you’ll work with a single SQLite database file.

You’ll name the database birds.db. As the name suggests, you’ll use the database to store the names of birds you like. To keep the example small, you’ll only keep track of the bird names and give them an ID as a unique identifier.

Read the full article at https://realpython.com/crud-operations/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

When I first got into web development after graduating from university, I got excited about how easy it was to build a website. A bit of knowledge goes a long way. I learned HTML, CSS, Ruby on Rails, and PHP, and tried all kinds of platforms and content management systems. Sometimes it felt hard and sometimes easy. But it was those “aha” moments when I could prototype a site in an afternoon that really got me excited. Which is why I think Drupal Starshot is such a promising initiative.

The goal of Starshot is to create an out-of-the-box version of Drupal that fast-tracks people through building and launching a website. It’ll attract new users, on-board them quickly, and give them an instant feeling of empowerment.

Starshot will show new users that, yes, you can build enterprise websites with Drupal — but you can also use it to launch a campaign, create an event website, or evolve a digital presence for your startup business. 

The Initiative was announced at DrupalCon Portland 2024. Already, more than 386 individuals have pledged support, and dozens of companies have expressed interest too. Starshot is shaping up to be one of the most exciting large-scale initiatives in Drupal’s 23 year history.

“Without a doubt, Starshot will be the largest change to Drupal since the foundational rewrite to modern object-oriented PHP that occurred with Drupal 8.” 

– Mike Herschel, Senior Front-End Developer at Agileana, former elected member of the Drupal Association Board of Directors

Is Starshot a Rewrite of Drupal?

No, Starshot isn’t a rewrite. It is built on Drupal core and include many features from recent Drupal initiatives, including:

• Recipes, packaged features that give you content types and content listings
• Project Browser, for adding more functionality
• Experience Builder, the improved version of Layout Builder for drag-and-drop page building 
• Automatic Updates

Drupal Starshot will be available alongside traditional Drupal Core on the Drupal.org download page later this year. Selecting Starshot will install the necessary features for various website use cases, simplifying the process for new users. Traditional Drupal Core will continue to be available for more custom sites. 

Watch the Driesnote from DrupalCon Portland 2024 where Starshot was revealed:

How Will Starshot Enhance the Open Web?

Starshot will increase Drupal usage and bring the power of open source to even more people by lowering the barrier to entry. It’s a great opportunity to package a version of Drupal that’s more attractive, user-friendly, and engaging for newcomers. This initiative will bring long-overdue improvements to usability, while maintaining Drupal’s strengths: content architecture, security, and multilingual support.

A key component of Starshot's development is its focus on the "Builder" persona—users who aspire to create amazing websites with Drupal but may not have extensive web development experience. By leveraging user research with Builders, Starshot will tailor its features and the editing interface around their needs. This means that folks with limited technical expertise can harness the power of Drupal to bring their creative ideas to life.

Starshot can pre-package Drupal with default configurations and pre-configured modules, leveraging the flexible Recipes initiative, and using the new Experience Builder to enhance page creation. Starshot should significantly reduce development and maintenance costs for simpler web builds. Plus, it will set good UX standards for Drupal websites across the board.

“Let’s reach for the stars and bring the open web to all.”

– Dries Buytaert, Creator and Project Lead of Drupal

How Can I Contribute?

The first releases of Starshot should be available before the end of 2024. The initiative needs lots of support to make this a reality!

Make your pledge on the Starshot landing page or reach out to us with the details of your expertise and availability. We’ll be happy to connect you with a meaningful way to contribute.

Want to help with marketing efforts related to Startshot? Go to the Drupal Marketing page or reach out to me directly.

Take Your Drupal Skills to New Heights

I’ve trained countless people in Drupal and web development over the years — always with the aim of empowering them and inspiring the same passion I have for building websites. If you’re looking to enhance your knowledge and gain valuable tools, you’ve come to the right place. My team and I offer course packages and custom training for site builders, content editors, UX/UI designers, front-end and back-end developers, and entire teams.

Learn more about Drupal training with Evolving Web.  

+ more awesome articles by Evolving Web

The person with the most trouble with duplicate material is the content writer. The hardest task is having to start over and make the necessary corrections after spending hours writing something and discovering that your content is plagiarized. 

This not only gives you a lot of work to do, but it may also interfere with the SEO of your website. Further, we’ll discuss the main problems brought on by duplicate content and discuss how the Drupal SEO module might assist in resolving this issue. 

As part of the Open Source AI Definition roadshow and as we approach the first release candidate of the draft, the Open Source Initiative (OSI) participated at PyCon US 2024, the annual gathering of the Python community. This opportunity was important because PyCon US brings together AI practitioners and developers alike, and having their input regarding what constitutes Open Source AI is of most value. The OSI organized a workshop and had a community booth there.

OSAID Workshop: compiling a FAQ to make the definition clear and easy to use

The OSI has embarked on a co-design process with multiple stakeholders to arrive at the Open Source AI Definition (OSAID). This process has been led by Mer Joyce, the co-design expert and facilitator, and Stefano Maffulli, the executive director of the OSI.

At the workshop organized at PyCon US, Mer provided an overview of the co-design process so far, summarized below.

The first step of the co-design process was to identify the freedoms needed for Open Source AI. After various online and in-person activities and discussions, including five workshops across the world, the community identified four freedoms:

1. To Use the system for any purpose and without having to ask for permission.
2. To Study how the system works and inspect its components.
3. To Modify the system for any purpose, including to change its output.
4. To Share the system for others to use with or without modifications, for any purpose.

The next step was to form four working groups to initially analyze four AI systems. To achieve better representation, special attention was given to diversity, equity and inclusion. Over 50% of the working group participants are people of color, 30% are black, 75% were born outside the US and 25% are women, trans and nonbinary.

These working groups discussed and voted on which AI system components should be required to satisfy the four freedoms for AI. The components we adopted are described in the Model Openness Framework developed by the Linux Foundation.

The vote compilation was performed based on the mean total votes per component (μ). Components which received over 2μ votes were marked as required and between 1.5μ and 2μ were marked likely required. Components that received between 0.5μ and μ were marked likely not required and less than 0.5μ as not required.

The working groups evaluated legal frameworks and legal documents for each component. Finally, each working group published a recommendation report. The end result is the OSAID with a comprehensive definition checklist encompassing a total of 17 components. More working groups are being formed to evaluate how well other AI systems align with the definition.

OSAID multi-stakeholder process: from component list to a definition checklist

After providing an overview of the co-design process, Mer went on to organize an exercise with the participants to compile a FAQ.

The questions raised at the workshop revolved around the following topics:

• End user comprehension: how and why are AI systems different from Open Source software? As an end-user, why should they care if an AI system is open?
• Datasets: Why is data itself not required? Should Open Source AI datasets be required to prove copyright compliance? How can one audit these systems for bias without the data? What does data provenance and data labeling entail?
• Models: How can proper attribution of model parameters be enforced? What is the ownership/attribution of model parameters which were trained by one author and then “fine-tuned” by another?
• Code: Can projects that include only source code (no data info or model weights) still use a regular Open Source license (MIT, Apache, etc.)?
• Governance: For a specific AI, who determines whether the information provided about the training, dataset, process, etc. is “sufficient” and how?
• Adoption of the OSAID: What are incentives for people/companies to adopt this standard?
• Legal weight: Is the OSAID supposed to have legal weight?

These questions and answers raised at the workshop will be important for enhancing the existing FAQ, which will be made available along with the OSAID.

OSAID workshop: a collection of post-its with questions raised by participants. Community Booth: gathering feedback on the “Unlock the OSAID” visualization

At the community booth, the OSI held two activities to draw in participants interested in Open Source AI. The first activity was a quiz developed by Ariel Jolo, program coordinator at the OSI, to assess participants’ knowledge of  Python and AI/ML. Once we had an understanding of their skills, we went on to the second and main activity, which was to gather feedback on the OSAID using a novel way to visualize how different AI systems match the current draft definition as described below.

Making it easy for different stakeholders to visualize whether or not an AI system matches the OSAID is a challenge, especially because there are so many components involved. This is where the visualization concept we named “Unlock the OSAID” came in. 

The OSI keyhole is a well recognized logo that represents the source code that unlocks the freedoms to use, study, modify, and share software. With the Unlock the OSAID, we played on that same idea, but now for AI systems. We displayed three keyholes representing the three domains these 17 components fall within: code, model and data information.

Here is the image representing the “code keyhole” with the required components to unlock the OSAID:

On the inner ring we have the required components to unlock the OSAID, while on the outer ring we have optional components. The required code components are: libraries and tools; inference; training, validation and testing; data pre-processing. The optional components are: inference for benchmark and evaluation code. 

To fully unlock the OSAID, an AI system must have all the required components for code, model and data information. To better understand how the “Unlock the OSAID” visualization works, let’s look at two hypothetical AI systems: example 1 and example 2.

Let’s start looking at example 1 (in red) and see if this system unlocks the OSAID for code:

Example 1 only provides inference code, so the key (in red) doesn’t “fit” the code keyhole (in green).

Now let’s look at example 2 (in blue):

Example 2 provides all required components (and more), so the key (in blue) fits the code keyhole (in green). Therefore, example 2 unlocks the OSAID for code. For example 2 to be considered Open Source AI, it would also have to unlock the OSAID for model and data information: 

We received good feedback from participants about the “Unlock the OSAID” visualization. Once participants grasped the concept of the keyholes and which components were required or optional, it was easy to identify if an AI system unlocks the OSAID or not. They could visually see if the keys fit the keyholes or not. If all keys fit, then that AI system adheres to the OSAID.

Final thoughts: engaging with the community and promoting Open Source principles

For me, the highlight of PyCon US was the opportunity to finally meet members of the OSI and the Python community in person, both new and old acquaintances. I had good conversations with Deb Nicholson (Python Software Foundation), Hannah Aubry (Fastly), Ana Hevesi (Uploop), Tom “spot” Callaway (AWS), Julia Ferraioli (AWS), Tony Kipkemboi (Streamlit), Michael Winser (Alpha-Omega), Jason C. MacDonald (OWASP), Cheuk Ting Ho (CMD Limes), Kamile Demir (Adobe), Mariatta Wijaya (PSF), Loren Clary (PSF) and Miaolai Zhou (AWS). I also interacted with many folks from the following communities: Python Brazil, Python en Español, PyLadies and Black Python Devs. It was great to bump into great legends like Seth Larson (PSF), Peter Wang (Anaconda) and Guido van Rossum.

I loved all the keynotes, in particular from Sumana Harihareswara about how she has improved Python Software Foundation’s infrastructure, and from Simon Willison about how we can all benefit from Open Source AI.

We also had a special dinner hosted by Stefano to celebrate this special milestone of the OSAID, with Stefano, Mer and I overlooking Pittsburgh.

Overall, our participation at PyCon US was a success. We shared the work OSI has been doing toward the first release candidate of the Open Source AI Definition, and we did it in an entertaining and engaging way, with plenty of connection throughout.

Photo credits: Ana Hevesi, Mer Joyce, and Nick Vidal

Do you want to look inside your Django request? How about all of your requests in development and see where they overlap? If that sounds useful, you should check out Kolo. It's a pretty incredible extension for your editor (VS Code at the moment, more editors to come most likely). We have Wilhelm Klopp on to tell us all about it.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Wil on Twitter</b>: <a href="https://twitter.com/wilhelmklopp" target="_blank" rel="noopener">@wilhelmklopp</a><br/> <b>Kolo</b>: <a href="https://kolo.app" target="_blank" rel="noopener">kolo.app</a><br/> <b>Kolo's info repo</b>: <a href="https://github.com/kolofordjango/kolo" target="_blank" rel="noopener">github.com</a><br/> <b>Kolo Playground</b>: <a href="https://play.kolo.app/" target="_blank" rel="noopener">play.kolo.app</a><br/> <b>Generating tests with Kolo</b>: <a href="https://blog.kolo.app/tests-no-joy.html" target="_blank" rel="noopener">kolo.app</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=NV6IfmrDY44" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/464/seeing-code-flows-and-generating-tests-with-kolo" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

A few weeks ago at DrupalCon Portland, I announced Drupal Starshot, a project to create the new default download of Drupal. Built on Drupal Core, Drupal Starshot will include popular features from the contributed project ecosystem. It focuses on delivering a great user experience right out of the box. Drupal Starshot builds on recent initiatives like Recipes, Project Browser, and Automatic Updates to elevate Drupal to new heights.

The response has been incredible! Hundreds of people have pledged their support on the Drupal Starshot page, and many more have asked how to get involved. Over the past few weeks, we have been planning and preparing, so I'm excited to share some next steps!

We're launching a series of sessions to get everyone up to speed and involved. These will be held as interactive Zoom calls, and the recordings will be shared publicly for everyone to watch at their convenience.

The main goal of these Zoom sessions is to help you get involved in each area. We'll cover details not included in my keynote, update you on our progress, and give you practical advice on where and how you can contribute.

We've scheduled six sessions, and we invite everyone to attend. The first one will be on this Friday on participation, funding, and governance! You can find the latest schedule online at https://www.drupal.org/starshot#sessions and the core calendar in the sidebar of the Drupal core news page.

We look forward to seeing you there and working together to make Drupal Starshot a success!
 

A few weeks ago at DrupalCon Portland, I announced Drupal Starshot, a project to create the new default download of Drupal. Built on Drupal Core, Drupal Starshot will include popular features from the contributed project ecosystem. It focuses on delivering a great user experience right out of the box. Drupal Starshot builds on recent initiatives like Recipes, Project Browser, and Automatic Updates to elevate Drupal to new heights.

The response has been incredible! Hundreds of people have pledged their support on the Drupal Starshot page, and many more have asked how to get involved. Over the past few weeks, we have been planning and preparing, so I'm excited to share some next steps!

We're launching a series of sessions to get everyone up to speed and involved. These will be held as interactive Zoom calls, and the recordings will be shared publicly for everyone to watch at their convenience.

The main goal of these Zoom sessions is to help you get involved in each area. We'll cover details not included in my keynote, update you on our progress, and give you practical advice on where and how you can contribute.

We've scheduled six sessions, and we invite everyone to attend. The first one will be on this Friday on participation, funding, and governance! You can find the latest schedule online at https://www.drupal.org/starshot#sessions and the core calendar in the sidebar of the Drupal core news page.

We look forward to seeing you there and working together to make Drupal Starshot a success!
 

Recently, we built a queue worker to send emails asynchronously, meeting a client's unique requirements to ensure email delivery. There is a lot of movement in the Drupal contribution space to innovate on the mailer. Traditionally, Drupal uses a plain PHP mailer to deliver transactional emails such as sign-up confirmation or password reset instructions. Nowadays, many websites rely on the contribution module Drupal Symfony Mailer to use the framework mailer by Symfony and leverage the flexible setup of mailer policies, transport, and HTML theming with templates. What needs to be added to the mix are easy-to-implement ways to send emails asynchronously. Today, we announce the first stable release of the new contribution module Symfony Mailer Queue.

Inspired by the successes of the "Blauer Engel Für FOSS" (BE4FOSS) project and KDE's ongoing Sustainable Software goal, KDE Eco has begun a new initiative: "Opt Green: Sustainable Software For Sustainable Hardware" (German: Nachhaltige Software Für Nachhaltige Hardware, or NS4NH).

Opt Green: Sustainable Software For Sustainable Hardware

By design, Free Software guarantees transparency and user autonomy. This gives you, the user, control of your hardware by removing unnecessary vendor dependencies. With Free Software, you're able to use your devices how you want, for as long as you want. There's no bloatware and you can block unwanted data use and ads from driving up energy demands and slowing down your device—while shutting the door to uninvited snooping in your private life as well. With software made for your needs and not the vendors', you can choose applications designed for the hardware you already own. Say goodbye to premature hardware obsolescence: lean, efficient Free Software runs on devices which are decades old!

Independent and sustainable Free Software is good for the users, and good for the environment.

Figure : The "Think Global, Act Local" campaign urged people to consider global health while taking action in their local communities. This new project urges people to do the same, but with computing. (Image from Karanjot Singh published under a CC-BY-4.0 license.)

Over the next two years, the "Opt Green" initiative will bring what KDE Eco has been doing for sustainable software directly to end users. A particular target group for the project is those whose consumer behavior is driven by principles related to the environment, and not just price or convenience: the "eco-consumers".

Through online and offline campaigns as well as installation workshops, we will demonstrate the power of Free Software to drive down resource and energy consumption, and keep devices in use for the lifespan of the hardware, not the software.

Our motto: The most environmentally-friendly device is the one you already own.

The topic of software-driven sustainability is relevant for all Free Software applications and developers. We'd love to have you join us and become partners in combatting the issue of software-driven environmental harm. Check out the project's Invent repository or the contact page to get involved today!

Software's Environmental Harm

On 14 October 2025, the end of support for Windows 10 is estimated to make e-waste out of 240 million computers ineligible for the upgrade to Windows 11. Moreover, macOS support for Intel-based Apple computers—the last of which were sold in 2020—is predicted to end (at the earliest) one year later in 2026, rendering even more millions upon millions of functioning devices obsolete. When users have no control over the software they rely on, they are left at a security risk when software support ends ... unless, of course, they purchase a new computer. (By comparison, consider that only in 2022 did Linus Torvalds first suggest ending Linux kernel support for Intel 486 processors from 1989. That's 33 years of support!)

Vendors frequently require buying a new device to support software updates. All too often, this is driven by economic imperatives rather than technological requirements. Moreover, while new hardware has become more and more powerful, new software offering similar or identical functionality has frequently become less efficient and more energy-intensive, which has rendered older, less powerful devices useless.

Already in 2015 Achim Steiner, former Executive Director of the UN Environment Programme (UNEP), warned of the "tsunami of electronic waste rolling out over the world".

In 2016, 44.7 million tonnes of e-waste were generated, estimated to be equivalent to 4500 Eiffel Towers. If you were to stack those Eiffel Towers on top of each other, the result would be 17 times higher than Mount Everest.

By 2017, United Nations University determined e-waste to be the fastest growing waste stream in the world.

In 2022, the amount of e-waste reached 59.4 million tonnes, a 33% increase since 2016.

The flow of e-waste continues to rise today.

Figure : In 2016, 44.7 million metric tons of e-waste was generated. This is estimated to be equivalent to 4,500 Eiffel Towers, which, when stacked, is 17 times higher than Mount Everest. (Image from KDE published under a CC-BY-SA-4.0 license. Design by Anita Sengupta.)

Software is a frequently unacknowledged yet significant factor for sustainability. Software determines a hardware's energy consumption and minimum system requirements. It determines how long a device can remain safely in use. With software running on everyday devices, from coffee machines to smartphones, from trains to drones, the role of software in keeping functioning hardware in use and out of the landfill grows more critical every day.

For consumers, the environmental harm may be out-of-sight and out-of-mind. Yet the environment is registering its effects, from the CO2 pumped into the atmosphere to the landfills that receive our discarded devices at their end of life, and the air, soil, and waters around them—not to mention the people and animals.

Figure : A young man is pictured burning electrical wires to recover copper at Agbogbloshie, Ghana, as another metal scrap worker arrives with more wires to be burned. A 2018 article in the "International Journal of Cancer" reports a correlation between proximity to e-waste burn sites and childhood lymphoma. (Image by Muntaka Chasant, published under a CC-BY-SA-4.0 license.)

It is particularly devastating when you consider the environmental and social harm caused by e-waste, especially when e-waste is generated earlier than necessary because of premature obsolescence. The production and transportation of a device accounts for 50–80+% of its carbon footprint over its lifecycle. A German Environment Report estimates you’d need to use a computer for over 30 years before efficiency gains in newly-produced devices justify their purchase.

Furthermore, the extraction of rare earth metals in production consumes copious amounts of energy and takes place under miserable social conditions, often in the Global South. For disposal, devices are typically returned to the Global South for end-of-life treatment, where they pollute the environment as toxic waste and cause enormous damage to workers' health or even death.

Figure : Apple's carbon footprint. From Apple (2019), "Environmental Responsibility Report: 2019 Progress Report, covering fiscal year 2018". (Image from KDE published under a CC-BY-SA-4.0 license. Design by Anita Sengupta.) Giving Consumers What They Want

Globally, interest in environmental harm and sustainable goods has been rising steadily from 2015 to 2021. In Europe, a 2020 Eurobarometer poll found that 50% of consumers indicate that two reasons they purchase new devices are performance issues and non-functioning software, and 8 in 10 consumers believe manufacturers should be required to make it easier to repair digital devices.

Free Software already gives consumers what they want, but most don't know it yet. Transparency in code makes lightweight, highly performative software possible, even on much older devices, while user autonomy ensures the right to repair when applications stop functioning.

Figure : KDE’s popular multi-platform PDF reader and universal document viewer Okular was awarded the Blue Angel ecolabel in 2022. (Image from KDE published under a CC-BY-4.0 license.)

In fact, the Blue Angel criteria for desktop software are at the forefront in recognizing the critical role of transparency and user autonomy in sustainable software design. From 2021-2023, the KDE Eco project "Blauer Engel Für FOSS" (BE4FOSS) had the goal of collecting and spreading information about the Blue Angel ecolabel among developer communities. In 2022, KDE’s popular PDF and universal document reader Okular became the first ever Blue Angel eco-certified software! The BE4FOSS project culminated with the KDE Eco handbook "Applying The Blue Angel Criteria To Free Software", which you can read here. KDE's Sustainable Software goal has continued this work by developing emulation tools like KdeEcoTest and Selenium AT-SPI to measure software's energy consumption in KDE's KEcoLab.

Now we want to take what we have achieved and bring it directly to eco-consumers.

Through educational campaigns and workshops, the "Opt Green" project aims to combat e-waste by keeping hardware in use with Free Software. Although the problem of software-driven e-waste is relevant for an increasing number of digital devices, the focus will be on desktop PCs, laptops, and, when possible, smartphones and tablets. We are planning to set up info-stands at fair-trade, organic, and artisanal markets as well as sustainability festivals such as the Umweltfestival in Berlin. We will distribute leaflets to consumers, and demo vendor-abandoned devices which are not only usable, but also a joy to use thanks to the tireless work of inspiring Free Software communities. Installation workshops will give users the know-how to keep their devices in use for as long as they want.

Consumers don’t need a new computer to get secure, cutting-edge software; they just need the right software. Free Software already gives consumers what they want today, and we will be working hard to make sure they know that.

Ready To Join Us?

Consumers want sustainable and repairable digital devices. We believe that providing users the software to keep devices in use and out of the landfill will drive demand for Free Software products and enable long-term hardware use.

Do you want to join us in this movement to combat e-waste with Free Software? See our contact info to get involved.

We need volunteers like you to bring the "Opt Green" campaign to towns and cities around the world. We need volunteers like you to design engaging guides and beautiful materials for global distribution. We need volunteers like you to report on the project in magazines and newspapers. Let's work together to bring sustainable software to your community!

Maybe you are interested in contributing to the development of measurement tools like KdeEcoTest and Selenium AT-SPI or improving KEcoLab automation? Or using such tools to measure your software application's energy consumption? Let's collaborate to make energy transparency a part of Free Software development today!

Or maybe you actively contribute to a Free Software project that will keep hardware in use for longer. Please be in touch! We want to promote the amazing work you do directly with consumers.

Additional ideas are more than welcome. Part of the project will be figuring out what works and engagement by people like you will make this project a success. We would love to have you join us. Learn more: https://eco.kde.org/get-involved/

Funding Notice

The NS4NH project is funded by the Federal Environment Agency and the Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV1). The funds are made available by resolution of the German Bundestag.

The publisher is responsible for the content of this publication.