FLOSS Project Planets
Python⇒Speed: Let’s optimize! Running 15× faster with a situation-specific algorithm
Let’s speed up some software! Our motivation: we have an image, a photo of some text from a book. We want to turn it into a 1-bit image, with just black and white, extracting the text so we can easily read it.
We’ll use an example image from scikit-image, an excellent image processing library:
from skimage.data import page import numpy as np IMAGE = page() assert IMAGE.dtype == np.uint8Here’s what it looks like (it’s licensed under this license):
Median-based local thresholdingThe task we’re trying to do—turning darker areas into black, and lighter areas into white—is called thresholding. Since the image is different in different regions, with some darker and some lighter, we’ll get the best results if we use local thresholding, where the threshold is calculated from the pixel’s neighborhood.
Simplifying somewhat, for each pixel in the image we will:
- Calculate the median of the surrounding neighborhood.
- Subtract a magic constant from the calculated median to calculate our local threshold.
- If the pixel’s value is bigger than the threshold, the result is white, otherwise it’s black.
scikit-image includes an implementation of this algorithm. Here’s how we use it:
from skimage.filters import threshold_local def skimage_median_local_threshold(img, neighborhood_size, offset): threshold = threshold_local( img, block_size=neighborhood_size, method="median", offset=offset ) result = (img > threshold).astype(np.uint8) result *= 255 return result # The neighborhood size and offset value were determined "empirically", i.e. # they're manually tuning the algorithm to work well with our specific # example image. SKIMAGE_RESULT = skimage_median_local_threshold(IMAGE, 11, 10)And here’s what the results look like:
Let’s see if we can make this faster!
Step 1. Reimplement our own versionWe’re going to be using the Numba compiler, which lets us compile Python code to machine code at runtime. Here’s an initial implementation of the algorithm; it’s not quite identical to the original, for example the way edge pixels are handled, but it’s close enough for our purposes:
from numba import jit @jit def median_local_threshold1(img, neighborhood_size, offset): # Neighborhood size must be an odd number: assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) # For every pixel: for i in range(img.shape[0]): # Calculate the Y borders of the neighborhood: min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) for j in range(img.shape[1]): # Calculate the X borders of the neighborhood: min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # Calculate the median: median = np.median(img[min_y:max_y, min_x:max_x]) # Set the image to black or white, depending how it relates to # the threshold: if img[i, j] > median - offset: # White: result[i, j] = 255 else: # Black: result[i, j] = 0 return result NUMBA_RESULT1 = median_local_threshold1(IMAGE, 11, 10)Here’s the resulting image; it looks similar enough that for our purposes:
Now we can compare the performance of the two implementations:
Code Elapsed milliseconds skimage_median_local_threshold(IMAGE, 11, 10) 76 median_local_threshold1(IMAGE, 11, 10) 87It’s slower. But that’s OK, we’re just getting started.
Step 2: A faster implementation of the median algorithmCalculating a median is pretty expensive, and we’re doing it for every single pixel, so let’s see if we can speed it up.
The generic median implementation Numba provides is likely to be fairly generic, since it needs to work in a wide variety of circumstances. We can hypothesize that it’s not optimized for our particular case. And even if it is, having our own implementation will allow for a second round of optimization, as we’ll see in the next step.
We’re going to implement a histogram-based median, based on the fact we’re using 8-bit images that only have a limited range of potential values. The median is the value where 50% of the pixels’ values are smaller, and 50% are bigger.
Here’s the basic algorithm for a histogram-based median:
- Each pixel’s value will go into a different bucket in the histogram; since we know our image is 8-bit, we only need 256 buckets.
- Then, we add up the size of each bucket in the histogram, from smallest to largest, until we hit 50% of the pixels we inspected.
Here’s the resulting image:
And here’s the performance of our new implementation:
Code Elapsed milliseconds median_local_threshold1(IMAGE, 11, 10) 86 median_local_threshold2(IMAGE, 11, 10) 18That’s better!
Step 3: Stop recalculating the histogram from scratchOur algorithm uses a rolling neighborhood or window over the image, calculating the median for a window around each pixel. And the neighborhood for one pixel has a significant overlap for the neighborhood of the next pixel. For example, let’s say we’re looking at a neighborhood size of 3. We might calculate the median of this area:
...... .\\\.. .\\\.. .\\\.. ...... ......And then when process the next pixel we’ll calculate the median of this area:
...... ..///. ..///. ..///. ...... ......If we superimpose them, we can see there’s an overlap, the X:
...... .\XX/. .\XX/. .\XX/. ...... ......Given the histogram for the first pixel, if we remove the values marked with \ and add the ones marked with /, we’ve calculated the exact histogram for the second pixel. So for a 3×3 neighborhood, instead of processing 3 columns we process 2, a minor improvement. For a 11×11 neighborhood, we will go from processing 11 columns to 2 columns, a much more significant improvement.
Here’s what the code looks like:
@jit def median_local_threshold3(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) histogram = np.empty((256,), dtype=np.uint32) for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) # Populate histogram as if we started one pixel to the left: histogram[:] = 0 initial_neighborhood = img[min_y:max_y, 0:radius].ravel() for k in range(len(initial_neighborhood)): histogram[initial_neighborhood[k]] += 1 for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # 😎 Instead of recalculating histogram from scratch, re-use the # previous pixel's histogram. # Substract left-most column we don't want anymore: if min_x > 0: for y in range(min_y, max_y): histogram[img[y, min_x - 1]] -= 1 # Add new right-most column: if max_x < img.shape[1]: for y in range(min_y, max_y): histogram[img[y, max_x - 1]] += 1 # Find the the median from the updated histogram: half_neighborhood_size = ((max_y - min_y) * (max_x - min_x)) // 2 for l in range(256): half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: break median = l if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT3 = median_local_threshold3(IMAGE, 11, 10)Here’s the resulting image:
And here’s the performance of our latest code:
Code Elapsed microseconds median_local_threshold2(IMAGE, 11, 10) 17,066 median_local_threshold3(IMAGE, 11, 10) 6,386 Step #4: Adapative heuristicsNotice that a median’s definition is symmetrical:
- The first value that is smaller than the highest 50% values.
- Or, the first value that is larger than the lowest 50% values. We used this definition in our code above, adding up buckets from the smallest to the largest.
Depending on the distribution of values, one approach to adding up buckets to find the median may be faster than the other. For example, given a 0-255 range, if the median is going to be 10 we want to start from the smallest bucket to minimize additions. But if the median is going to be 200, we want to start from the largest bucket.
So which side we should start from? One reasonable heuristic is to look at the previous median we calculated, which most of the time will be quite similar to the new median. If the previous median was small, start from the smallest buckets; if it was large, start from the largest buckets.
@jit def median_local_threshold4(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) histogram = np.empty((256,), dtype=np.uint32) median = 0 for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) histogram[:] = 0 initial_neighborhood = img[min_y:max_y, 0:radius].ravel() for k in range(len(initial_neighborhood)): histogram[initial_neighborhood[k]] += 1 for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) if min_x > 0: for y in range(min_y, max_y): histogram[img[y, min_x - 1]] -= 1 if max_x < img.shape[1]: for y in range(min_y, max_y): histogram[img[y, max_x - 1]] += 1 half_neighborhood_size = ((max_y - min_y) * (max_x - min_x)) // 2 # 😎 Find the the median from the updated histogram, choosing # the starting side based on the previous median; we can go from # the leftmost bucket to the rightmost bucket, or in reverse: the_range = range(256) if median < 127 else range(255, -1, -1) for l in the_range: half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: median = l break if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT4 = median_local_threshold4(IMAGE, 11, 10)The end result is 25% faster. Since the heuristic is tied to the image contents, the performance impact will depend on the image.
Code Elapsed microseconds median_local_threshold3(IMAGE, 11, 10) 6,381 median_local_threshold4(IMAGE, 11, 10) 4,920 The big pictureHere’s a performance comparison of all the versions of the code:
Code Elapsed microseconds skimage_median_local_threshold(IMAGE, 11, 10) 76,213 median_local_threshold1(IMAGE, 11, 10) 86,494 median_local_threshold2(IMAGE, 11, 10) 17,145 median_local_threshold3(IMAGE, 11, 10) 6,398 median_local_threshold4(IMAGE, 11, 10) 4,925Let’s go over the steps we went through:
- Switch to a compiled language: this gives us more control.
- Reimplement the algorithm taking advantage of constrained requirements: our median only needed to handle uint8, so a histogram was a reasonable solution.
- Reuse previous calculations to prevent repetition: our histogram for the neighborhood of a pixel is quite similar to that of the previous pixel. This means we can reuse some of the calculations.
- Adaptively tweak the algorithm at runtime: as we run on an actual image, we use what we’ve learned up to this point to hopefully run faster later on. The decision from which side of the histogram to start is arbirary in general. But in this specific algorithm, the overlapping pixel neighborhoods mean we can make a reasonable guess.
This process demonstrates part of why generic libraries may be slower than custom code you write for your particular use case and your particular data.
Next stepsWhat else can you do to speed up this algorithm? Here are some ideas:
- There may be a faster alternative to histogram-based medians.
- We’re not fully taking advantage of histogram overlap; there’s also overlap between rows.
- The cumulative sum in the histogram doesn’t benefit from instruction-level parallelism or SIMD. It’s possible that using one of those would result in faster results even if it uses more instructions.
- So far the code has only used a single CPU. Given each row is calculated independently, parallelism would probably work well if done in horizontal stripes, probably taller than one pixel so as to maximize utilization of memory caches.
Want to learn more about optimizing compiled code for Python data processing? This article is an extract from a book I’m working on; test readers are currently going through initial drafts. Aimed at Python developers, data scientists, and scientists, the book covers topics like instruction-level parallelism, memory caches, and other performance optimization techniques. Learn more and sign up to get updates here.
Read more...Matthew Palmer: GitHub's Missing Tab
Visit any GitHub project page, and the first thing you see is something that looks like this:
“Code”, that’s fairly innocuous, and it’s what we came here for. The “Issues” and “Pull Requests” tabs, with their count of open issues, might give us some sense of “how active” the project is, or perhaps “how maintained”. Useful information for the casual visitor, undoubtedly.
However, there’s another user community that visits this page on the regular, and these same tabs mean something very different to them.
I’m talking about the maintainers (or, more commonly, maintainer, singular). When they see those tabs, all they see is work. The “Code” tab is irrelevant to them – they already have the code, and know it possibly better than they know their significant other(s) (if any). “Issues” and “Pull Requests” are just things that have to be done.
I know for myself, at least, that it is demoralising to look at a repository page and see nothing but work. I’d be surprised if it didn’t contribute in some small way to maintainers just noping the fudge out.
A Modest ProposalSo, here’s my thought. What if instead of the repo tabs looking like the above, they instead looked like this:
My conception of this is that it would, essentially, be a kind of “yearbook”, that people who used and liked the software could scribble their thoughts on. With some fairly straightforward affordances elsewhere to encourage its use, it could be a powerful way to show maintainers that they are, in fact, valued and appreciated.
There are a number of software packages I’ve used recently, that I’d really like to say a general “thanks, this is awesome!” to. However, I’m not about to make the Issues tab look even scarier by creating an “issue” to say thanks, and digging up an email address is often surprisingly difficult, and wouldn’t be a public show of my gratitude, which I believe is a valuable part of the interaction.
You Can’t Pay Your Rent With KudosAbsolutely you cannot. A means of expressing appreciation in no way replaces the pressing need to figure out a way to allow open source developers to pay their rent. Conversely, however, the need to pay open source developers doesn’t remove the need to also show those people that their work is appreciated and valued by many people around the world.
Anyway, who knows a senior exec at GitHub? I’ve got an idea I’d like to run past them…
Anarcat: Playing with fonts again
I am getting increasingly frustrated by Fira Mono's lack of italic support so I am looking at alternative fonts again.
Commit MonoThis time I seem to be settling on either Commit Mono or Space Mono. For now I'm using Commit Mono because it's a little more compressed than Fira and does have a italic version. I don't like how Space Mono's parenthesis (()) is "squarish", it feels visually ambiguous with the square brackets ([]), a big no-no for my primary use case (code).
So here I am using a new font, again. It required changing a bunch of configuration files in my home directory (which is in a private repository, sorry) and Emacs configuration (thankfully that's public!).
One gotcha is I realized I didn't actually have a global font configuration in Emacs, as some Faces define their own font family, which overrides the frame defaults.
This is what it looks like, before:
Fira Mono
After:
Commit Mono
(Notice how those screenshots are not sharp? I'm surprised too. The originals look sharp on my display, I suspect this is something to do with the Wayland transition. I've tried with both grim and flameshot, for what its worth.)
They are pretty similar! Commit Mono feels a bit more vertically compressed maybe too much so, actually -- the line height feels too low. But it's heavily customizable so that's something that's relatively easy to fix, if it's really a problem. Its weight is also a little heavier and wider than Fira which I find a little distracting right now, but maybe I'll get used to it.
All characters seem properly distinguishable, although, if I'd really want to nitpick I'd say the © and ® are too different, with the latter (REGISTERED SIGN) being way too small, basically unreadable here. Since I see this sign approximately never, it probably doesn't matter at all.
I like how the ampersand (&) is more traditional, although I'll miss the exotic one Fira produced... I like how the back quotes (`, GRAVE ACCENT) drop down low, nicely aligned with the apostrophe. As I mentioned before, I like how the bar on the "f" aligns with the other top of letters, something in Fira mono that really annoys me now that I've noticed it (it's not aligned!).
A UTF-8 test fileHere's the test sheet I've made up to test various characters. I could have sworn I had a good one like this lying around somewhere but couldn't find it so here it is, I guess.
US keyboard coverage: abcdefghijklmnopqrstuvwxyz`1234567890-=[]\;',./ ABCDEFGHIJKLMNOPQRSTUVWXYZ~!@#$%^&*()_+{}|:"<>? latin1 coverage: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ EURO SIGN, TRADE MARK SIGN: €™ ambiguity test: e¢coC0ODQ iI71lL!|¦ b6G&0B83 [](){}/\.…·• zs$S52Z% ´`'"‘’“”«» all characters in a sentence, uppercase: the quick fox jumps over the lazy dog THE QUICK FOX JUMPS OVER THE LAZY DOG same, in french: voix ambiguë d'un cœur qui, au zéphyr, préfère les jattes de kiwis. VOIX AMBIGUË D'UN CŒUR QUI, AU ZÉPHYR, PRÉFÈRE LES JATTES DE KIWIS. Ligatures test: -<< -< -<- <-- <--- <<- <- -> ->> --> ---> ->- >- >>- =<< =< =<= <== <=== <<= <= => =>> ==> ===> =>= >= >>= <-> <--> <---> <----> <=> <==> <===> <====> :: ::: __ <~~ </ </> /> ~~> == != /= ~= <> === !== !=== =/= =!= <: := *= *+ <* <*> *> <| <|> |> <. <.> .> +* =* =: :> (* *) /* */ [| |] {| |} ++ +++ \/ /\ |- -| <!-- <!--- Box drawing alignment tests: █ ╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▉ ╱╲╱╲╳╳╳ ║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▊ ╲╱╲╱╳╳╳ ║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▋ ╱╲╱╲╳╳╳ ╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▌ ╲╱╲╱╳╳╳ ║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▍ ║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎ ╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ └╌╌┘ ╎ ┗╍╍┛ ┋ ▏▁▂▃▄▅▆▇█ Dashes alignment test: HYPHEN-MINUS, MINUS SIGN, EN, EM DASH, HORIZONTAL BAR, LOW LINE -------------------------------------------------- −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− –––––––––––––––––––––––––––––––––––––––––––––––––– —————————————————————————————————————————————————— ―――――――――――――――――――――――――――――――――――――――――――――――――― __________________________________________________So there you have it, got completely nerd swiped by typography again. Now I can go back to writing a too-long proposal again.
Sources and inspiration for the above:
the unicode(1) command, to lookup individual characters to disambiguate, for example, - (U+002D HYPHEN-MINUS, the minus sign next to zero on US keyboards) and − (U+2212 MINUS SIGN, a math symbol)
searchable list of characters and their names - roughly equivalent to the unicode(1) command, but in one page, amazingly the /usr/share/unicode database doesn't have any one file like this
bits/UTF-8-Unicode-Test-Documents - full list of UTF-8 characters
UTF-8 encoded plain text file - nice examples of edge cases, curly quotes example and box drawing alignment test which, incidentally, showed me I needed specific faces customisation in Emacs to get the Markdown code areas to display properly, also the idea of comparing various dashes
sample sentences in many languages - unused, "Sentences that contain all letters commonly used in a language"
UTF-8 sampler - unused, similar
In my previous blog post about fonts, I had a list of alternative fonts, but it seems people are not digging through this, so I figured I would redo the list here to preempt "but have you tried Jetbrains mono" kind of comments.
My requirements are:
- no ligatures: yes, in the previous post, I wanted ligatures but I have changed my mind. after testing this, I find them distracting, confusing, and they often break the monospace nature of the display
- monospace: this is to display code
- italics: often used when writing Markdown, where I do make use of italics... Emacs falls back to underlining text when lacking italics which is hard to read
- free-ish, ultimately should be packaged in Debian
Here is the list of alternatives I have considered in the past and why I'm not using them:
agave: recommended by tarzeau, not sure I like the lowercase a, a bit too exotic, packaged as fonts-agave
Cascadia code: optional ligatures, multilingual, not liking the alignment, ambiguous parenthesis (look too much like square brackets), new default for Windows Terminal and Visual Studio, packaged as fonts-cascadia-code
Fira Code: ligatures, was using Fira Mono from which it is derived, lacking italics except for forks, interestingly, Fira Code succeeds the alignment test but Fira Mono fails to show the X signs properly! packaged as fonts-firacode
Hack: no ligatures, very similar to Fira, italics, good alternative, fails the X test in box alignment, packaged as fonts-hack
Hermit: no ligatures, smaller, alignment issues in box drawing and dashes, packaged as fonts-hermit somehow part of cool-retro-term
IBM Plex: irritating website, replaces Helvetica as the IBM corporate font, no ligatures by default, italics, proportional alternatives, serifs and sans, multiple languages, partial failure in box alignment test (X signs), fancy curly braces contrast perhaps too much with the rest of the font, packaged in Debian as fonts-ibm-plex
Intel One Mono: nice legibility, no ligatures, alignment issues in box drawing, not packaged in Debian
Iosevka: optional ligatures, italics, multilingual, good legibility, has a proportional option, serifs and sans, line height issue in box drawing, fails dash test, not in Debian
Jetbrains Mono: (mandatory?) ligatures, good coverage, originally rumored to be not DFSG-free (Debian Free Software Guidelines) but ultimately packaged in Debian as fonts-jetbrains-mono
Monoid: optional ligatures, feels much "thinner" than Jetbrains, not liking alignment or spacing on that one, ambiguous 2Z, problems rendering box drawing, packaged as fonts-monoid
Mononoki: no ligatures, looks good, good alternative, suggested by the Debian fonts team as part of fonts-recommended, problems rendering box drawing, em dash bigger than en dash, packaged as fonts-mononoki
Source Code Pro: italics, looks good, but dash metrics look whacky, not in Debian
spleen: bitmap font, old school, spacing issue in box drawing test, packaged as fonts-spleen
sudo: personal project, no ligatures, zero originally not dotted, relied on metrics for legibility, spacing issue in box drawing, not in Debian
So, if I get tired of Commit Mono, I might probably try, in order:
- Hack
- Jetbrains Mono
- IBM Plex Mono
Iosevka, Monoki and Intel One Mono are also good options, but have alignment problems. Iosevka is particularly disappointing as the EM DASH metrics are just completely wrong (much too wide).
This was tested using the Programming fonts site which has all the above fonts, which cannot be said of Font Squirrel or Google Fonts, amazingly. Other such tools:
- Coding Font (broken in Firefox as of 2024-05-30)
- dev fonts comparator
- Font Squirrel
- Google Fonts
- Programming fonts
Antoine Beaupré: 2024-05-29-playing-with-fonts-again
meta title="Playing with fonts again"
I am getting increasingly frustrated by Fira Mono's lack of italic support so I am looking at alternative fonts again.
This time I seem to be settling on either Commit Mono or Space Mono. For now I'm using Commit Mono because it's a little more compressed than Fira and does have a italic version. I don't like how Space Mono's parenthesis (()) is "squarish", it feels visually ambiguous with the square brackets ([]), a big no-no for my primary use case (code).
So here I am using a new font, again. It required changing a bunch of configuration files in my home directory (which is in a private repository, sorry) and Emacs configuration (thankfully that's public!).
One gotcha is I realized I didn't actually have a global font configuration in Emacs, as some Faces define their own font family, which overrides the frame defaults.
This is what it looks like, before:
Fira MonoAfter:
Commit Mono(Notice how those screenshots are not sharp? I'm surprised too. The originals look sharp on my display, I suspect this is something to do with the Wayland transition. I've tried with both grim and flameshot, for what its worth.)
They are pretty similar! Commit Mono feels a bit more vertically compressed maybe too much so, actually -- the line height feels too low. But it's heavily customizable so that's something that's relatively easy to fix, if it's really a problem. Its weight is also a little heavier and wider than Fira which I find a little distracting right now, but maybe I'll get used to it.
All characters seem properly distinguishable, although, if I'd really want to nitpick I'd say the © and ® are too different, with the latter (REGISTERED SIGN) being way too small, basically unreadable here. Since I see this sign approximately never, it probably doesn't matter at all.
I like how the ampersand (&) is more traditional, although I'll miss the exotic one Fira produced... I like how the back quotes (`, GRAVE ACCENT) drop down low, nicely aligned with the apostrophe. As I mentioned before, I like how the bar on the "f" aligns with the other top of letters, something in Fira mono that really annoys me now that I've noticed it (it's not aligned!).
Here's the test sheet I've made up to test various characters. I could have sworn I had a good one like this lying around somewhere but couldn't find it so here it is, I guess.
ASCII test abcdefghijklmnopqrstuvwxyz1234567890-= ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%^&*()_+ ambiguous characters &iIL7l1!|[](){}/\oO0DQ8B;:,./?~`'"$ all characters in a sentence, uppercase the quick fox jumps over the lazy dog THE QUICK FOX JUMPS OVER THE LAZY DOG same, in french voix ambiguë d'un cœur qui, au zéphyr, préfère les jattes de kiwis. VOIX AMBIGUË D'UN CŒUR QUI, AU ZÉPHYR, PRÉFÈRE LES JATTES DE KIWIS. Box drawing alignment tests: █ ▉ ╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▊ ╱╲╱╲╳╳╳ ║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▋ ╲╱╲╱╳╳╳ ║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▌ ╱╲╱╲╳╳╳ ╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▍ ╲╱╲╱╳╳╳ ║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎ ║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▏ ╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ └╌╌┘ ╎ ┗╍╍┛ ┋ ▁▂▃▄▅▆▇█ MIDDLE DOT, BULLET, HORIZONTAL ELLIPSIS: ·•… curly ‘single’ and “double” quotes ACUTE ACCENT, GRAVE ACCENT: ´` EURO SIGN: € unicode A1-BF: ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿ HYPHEN-MINUS, MINUS SIGN, EN, EM DASH, HORIZONTAL BAR, LOW LINE -------------------------------------------------- −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− –––––––––––––––––––––––––––––––––––––––––––––––––– —————————————————————————————————————————————————— ―――――――――――――――――――――――――――――――――――――――――――――――――― __________________________________________________So there you have it, got completely nerd swiped by typography again. Now I can go back to writing a too-long proposal again.
Sources and inspiration for the above:
the unicode(1) command, to lookup individual characters to disambiguate, for example, - (U+002D HYPHEN-MINUS, the minus sign next to zero on US keyboards) and − (U+2212 MINUS SIGN, a math symbol)
searchable list of characters and their names - roughly equivalent to the unicode(1) command, but in one page, amazingly the /usr/share/unicode database doesn't have any one file like this
bits/UTF-8-Unicode-Test-Documents - full list of UTF-8 characters
UTF-8 encoded plain text file - nice examples of edge cases, curly quotes example and box drawing alignment test which, incidentally, showed me I needed specific faces customisation in Emacs to get the Markdown code areas to display properly, also the idea of comparing various dashes
sample sentences in many languages - unused, "Sentences that contain all letters commonly used in a language"
UTF-8 sampler - unused, similar
Mike Driscoll: Episode 42 – Harlequin – The SQL IDE for Your Terminal
This episode focuses on the Harlequin application, a Python SQL IDE for your terminal written using the amazing Textual package.
I was honored to have Ted Conbeer, the creator of Harlequin, on the show to discuss his creation and the other things he does with Python.
Specifically, we focused on the following topics:
- Favorite Python packages
- Origins of Harlequin
- Why program for the terminal versus a GUI
- Lessons learned in creating the tool
- Asyncio
- and more!
The post Episode 42 – Harlequin – The SQL IDE for Your Terminal appeared first on Mouse Vs Python.
Django Weblog: Django Enhancement Proposal 14: Background Workers
As of today, DEP-14 has been approved 🛫
The DEP was written and stewarded by Jake Howard. A very enthusiastic community has been active with feedback and encouragement, while the Django Steering Council gave the final inputs before its formal acceptance. The implementation of DEP-14 is expected to be a major leap forward for the “batteries included” philosophy of Django.
Whilst Django is a web framework, there's more to web applications than just the request-response lifecycle. Sending emails, communicating with external services or running complex actions should all be done outside the request-response cycle.
Django doesn't have a first-party solution for long-running tasks, however the ecosystem is filled with incredibly popular frameworks, all of which interact with Django in slightly different ways. Other frameworks such as Laravel have background workers built-in, allowing them to push tasks into the background to be processed at a later date, without requiring the end user to wait for them to occur.
Library maintainers must implement support for any possible task backend separately, should they wish to offload functionality to the background. This includes smaller libraries, but also larger meta-frameworks with their own package ecosystem such as Wagtail.
This proposal sets out to provide an interface and base implementation for long-running background tasks in Django.
Future workThe DEP will now move on to the Implementation phase before being merged into Django itself.
If you would like to help or try it out, go have a look at django-tasks, a separate reference implementation by Jake Howard, the author of the DEP.
Jake will also be speaking about the DEP in his talk at DjangoCon Europe at DjangoCon Europe 2024 in Vigo next week.
Théodore 'nod_' Biadala: Sponsored Drupal Contribution
Back in March I started to look at sponsors for the time I’m spending working on the Drupal core issue queue. It’s been a few months and I wanted to go back on all the sponsored commits I made as a Frontend Framework Manager, to show how the sponsorships helped Drupal for the past few months.
The sponsorship offer is simple: you send me a fixed monthly fee of 2500€, and I share the issue credit of every Drupal core commit that I make. I’m very thankful to Palantir.net and OPTASY who are sponsoring me. Thanks to them I was able to increase the amount of commits I can make to Drupal core. In the last 3 months I committed 61 issues (worth 610 weighted issue credits) and the more sponsors I have, the more time I can spend reviewing and committing issues.
- Differentiate visually dragging with and without hierarchy A nice improvement for editors working a lot with lists and trees
- Sticky table header is not sticky if --drupal-displace-offset-top is not defined
- [jQuery 4] ajax.js and jquery.form.js use deprecated function $.parseJSON() Preparing for the next release of jQuery 4 with some cleanup.
- cspell check is broken in commit-code-check.sh Sometimes we break the CI and it needs to be fixed 🤷
- CKEditor admin toolbar config buttons using ::before to add content: have invalid screen reader text It takes dedication to land those accessibility fixes, kudos to our accessibility contributors.
- Linking in CKEditor 5: URLs with top-level domain but without protocol should get a protocol added automatically
- #states disable property has stopped working for submit button and other elements Sometimes when we clean-up code, we clean too much and break some other parts of the code
- Setting width for sticky-header is broken
- Negotiate max width/height of oEmbed assets more intelligently
- States API doesn't work with multiple select fields This was a a 13 years old issue! It _always_ feels good to close an issue that old.
- Add deprecation/bc support for library-overrides when files are moved Making sure backwards compatibility is working and useful
- Remove default event from collpased nav-tabs button
- [DrupalHtmlEngine] HTML-reserved characters (>, <, &) in <script> and <style> tag are converted to HTML entities It happens that we fix things for uses cases that stretch the reasonable (like having whole script tags in a WYSIWYG field)
- Olivero: Show content preview checkbox is not center aligned with the layout builder buttons. Even a minor issue of a misaligned text by a few pixels is worth fixing
- Drupal.theme.progressBar() does not escape output correctly
- filter_autop should ignore twig.config debug html comments Making sure Developer experience doesn't impact regular users
- tablePositionSticky should not be called on a non-array variable
- CKEditor 5 table cell vertical align "middle" doesn't work
- Move system/base component CSS to respective libraries where they exist A surprising performance improvement. There are still some low hanging fruits to improve the default frontend performance of Drupal
- Remove country setting from the installer When you don't need a piece of data, just don't collect it
- Media Library widget display doesn't return to first page on applying filters
- Deprecate and remove the AJAX replace method That was a leftover D7 era deprecation
- Claro should use libraries-extend for views_ui.css Even in core it happens that we don't use the right way to do something
- Removal :tabbable usage in dialog.js Some more jQuery 4 preparation
- Close icon is ovrlapping the title text in modal in claro Yes, typos can make it in the commit log
- Convert Olivero's teaser into a single directory component Slowly but surely we're adding Single directory components to Drupal core
- Refactor (if feasible) uses of the jQuery animate function to use Vanilla/native More CSS awesomeness making JavaScript code disapear
- [11.x] Update to jQuery 4.0.x beta Drupal staying on the bleading edge of frontend developement :)
- Refactor some uses of the jQuery parents function to use vanillaJS Did a small post earlier about this, CSS is really very good
- [regression] Uncaught TypeError: Cannot read properties of null (reading 'style') (toolbar.js)
- JSDoc for ajax command "changed" is incorrect There was a bunch of documentation fixes around this time
- menu_heading_id variable is not set in menu-region--footer.html.twig
- Add @file documentation to navigation.html.twig layout template
- Add @file documentation to menu-region--footer.html.twig template
- Views UI action buttons create janky layout shift on page load Polishing the loading of pages with heavy JS usage is important to show we care about UX
- Remove bottom radius on hover state of expanded sub menu item
- Setting empty URL when making embedded media a link in CKEditor5 causes JS errors
- Update color of submenu title text
- Collapsed nav-tabs status not exposed to screen reader There is a good number of accessibility fixes after this one, always nice to commit
- Incorrect padding on child menu items
- Claro: Form labels that are disabled have too low color contrast
- Claro should not hardcode decimal list style type for <ol>
- Some of string comparisons should use String.prototype.startsWith() or String.prototype.endsWith() Removing regular expressions as much as we can is a noble goal
- Location of "Skip to Main" link below admin toolbar in Claro is problematic for screen magnifier users
- Focus states on mobile second level navigation items can get cut off in Olivero
- Regression: Shortcuts menu flickers when the page is loaded Those toolbar flickering issues are tricky. Thankfully the new navigation module code is simpler than the existing Toolbar code, so it's much much easier to deal with
- escapeAdmin.js functionality should be removed(it not used anymore) Removed the feature that removed the overlay… for now, escapeAdmin will be back one way or another)
- Navigation module offsets the Olivero skip link element
- Umami page.tpl.php breaks block placeholders Sometime themes can break really nice Drupal features (like bigpipe)
- Claro CSS for dropbutton items adds large gap of white space
- Replace dialogContentResize jQuery event with CustomEvent Those events issues are really exciting, we're moving away slowly from jQuery for event management
- Umami views should use responsive grid Another case of core not using the awesome features we provide, not anymore :)
- Claro highlighted row not communicated to keyboard users
- Fix overflow visibility for wrapper content in navigation CSS
- Claro details component does not have the right class
- Make drupal.tableheader only use CSS for sticky table headers I will always welcome CSS-removing-JS patches
- Mismatch between implementation and description for Drupal.Message.prototype.remove().
- "Skip to main content" link skips over content that is essential to the page, banner role should be for global content
- Add pdureau as a co-maintainer for the Theme API with a focus on SDC Adding new maintainers is too rare. In this case the community is better for having him around
- Choose an icon for the Announcements link
- Remove deprecated moved_files entries in core
Many of these issues are maintenance focused, it’s not shiny, it’s not exciting, and it needs to be done. Sponsoring big initiatives like Starshot is exciting, let’s not forget the unexciting day to day that keeps things running. If you’re interested in supporting my work on Drupal core and keep the frontend fixes coming, consider sponsoring me.
PyCharm: PyCharm 2024.1.2: What’s New!
PyCharm 2024.1.2 is here with features designed to enhance your productivity and streamline your development workflow. This update includes support for DRF viewsets and routers in the Endpoints tool window, code assistance for TypedDict and Unpack, and improved debugger performance when handling large collections.
You can download the latest version from our download page or update your current version through our free Toolbox App.
For more details, please visit our What’s New page.
Key features Support for DRF viewsets and routers in the Endpoints tool windowWhen working with the Django REST Framework in PyCharm, not only can you specify function-based or class-based views in the path, but you can now also specify viewsets and see the results in the Endpoints tool window. Additionally, you can map HTTP methods to viewset methods, and PyCharm will display the HTTP methods next to the relevant route, including for custom methods. Routes without @actions decorators are now displayed with the related viewset methods.
Learn more Code assistance for TypedDict and UnpackPEP 692 made it possible to add type information for keyword arguments of different types by using TypedDict and Unpack. PyCharm allows you to use this feature confidently by providing parameter info, type checking, and code completion.
Improved debugger performance for large collectionsPyCharm’s debugger now offers a smoother experience, even when very large collections are involved. You can now work on your data science projects without having to put up with high CPU loads and UI freezes.
Download PyCharm 2024.1.2Be sure to check out our release notes to learn all of the details and ensure you don’t miss out on any new features.
We appreciate your support as we work to improve your PyCharm experience. Please report any bugs via our issue tracker so we can resolve them promptly. Connect with us on X (formerly Twitter) to share your feedback on PyCharm 2024.1.2!
Tag1 Consulting: Migrating Your Data from Drupal 7 to Drupal 10: Drupal Entities Overview
The Python Show: 42 - Harlequin - The SQL IDE for Your Terminal
This episode focuses on the Harlequin application, a Python SQL IDE for your terminal written using the amazing Textual package.
I was honored to have Ted Conbeer, the creator of Harlequin, on the show to discuss his creation and the other things he does with Python.
Specifically, we focused on the following topics:
Favorite Python packages
Origins of Harlequin
Why program for the terminal versus a GUI
Lessons learned in creating the tool
Asyncio
and more!
Drupal Association blog: Introducing the Local Associations Initiative: Empowering Drupal Communities Worldwide
We are thrilled to announce the launch of our new initiative led by Programs Manager, Joi Garrett. This program is designed to support the success of Drupal Local Associations by engaging directly with community leaders who work to promote the Drupal project in their global regions.
Connecting CommunitiesThe heart of the Local Associations Initiative lies in fostering meaningful connections. We recognize the efforts of local leaders and the unique challenges they face. By hosting a series of virtual meetings, we aim to create a platform for leaders to share their experiences, successes and challenges. These sessions will not only provide valuable insight to the state of various local associations but help to strengthen our global community.
Identifying and Addressing Common NeedsUnderstanding the diverse needs of our local associations is crucial. Through open dialogue in our virtual meetings, we will identify common needs and prioritize them. We hope by facilitating a collaborative environment the Drupal Association can support efforts for the most pressing issues faced by community leaders. The Drupal Association is committed to finding solutions that drive success.
Join Us on This JourneyWe invite local association leaders to participate in this initiative and attend the virtual meetings. Your insights and contributions are invaluable as we work together to strengthen our global Drupal Community. Stay tuned for announcements about the upcoming virtual meetings. Through this initiative, we aim to foster a collaborative environment where our global community feels more connected and supported. Once we have concluded the meetings, we will discuss the findings and future plans during DrupalCon Barcelona 2024.
We have been collecting contact information of Local Association leaders for the past few months, if you would like to be included please fill out the following form.
Thank you to our local leaders for being an integral part of our community. We look forward to collaborating with you to make this initiative a success!
Continent Expected Start Europe April (working with Network of European Drupal Associations) Asia June Australia July North America July South America August Africa AugustReal Python: What Are CRUD Operations?
CRUD operations are at the heart of nearly every application you interact with. As a developer, you usually want to create data, read or retrieve data, update data, and delete data. Whether you access a database or interact with a REST API, only when all four operations are present are you able to make a complete data roundtrip in your app.
Creating, reading, updating, and deleting are so vital in software development that these methods are widely referred to as CRUD. Understanding CRUD will give you an actionable blueprint when you build applications and help you understand how the applications you use work behind the scenes. So, what exactly does CRUD mean?
Get Your Code: Click here to download the free sample code that you’ll use to learn about CRUD operations in Python.
Take the Quiz: Test your knowledge with our interactive “What Are CRUD Operations?” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
What Are CRUD Operations?In this quiz, you'll revisit the key concepts and techniques related to CRUD operations. These operations are fundamental to any system that interacts with a database, and understanding them is crucial for effective data management.
In Short: CRUD Stands for Create, Read, Update, and DeleteCRUD operations are the cornerstone of application functionality, touching every aspect of how apps store, retrieve, and manage data. Here’s a brief overview of the four CRUD operations:
- Create: This is about adding new entries to your database. But it’s also applicable to other types of persistent storage, such as files or networked services. When you perform a create operation, you’re initiating a journey for a new piece of data within your system.
- Read: Through reading, you retrieve or view existing database entries. This operation is as basic as checking your email or reloading a website. Every piece of information you get has been received from a database, thanks to the read operation.
- Update: Updating allows you to modify the details of data already in the database. For example, when you update a profile picture or edit a chat message. Each time, there’s an update operation at work, ensuring your new data is stored in the database.
- Delete: Deleting removes existing entries from the database. Whether you’re closing an account or removing a post, delete operations ensure that unwanted or unnecessary data can be properly discarded.
CRUD operations describe the steps that data takes from creation to deletion, regardless of what programming language you use. Every time you interact with an application, you’re likely engaging in one of the four CRUD operations.
Why Are CRUD Operations Essential?Whether you’re working on a basic task list app or a complex e-commerce platform, CRUD operations offer a universal language for designing and manipulating data models. Knowing about CRUD as a user helps you understand what’s happening behind the curtains. As a developer, understanding CRUD provides you with a structured framework for storing data in your application with persistence:
In computer science, persistence refers to the characteristic of state of a system that outlives (persists more than) the process that created it. This is achieved in practice by storing the state as data in computer data storage. (Source)
So even when a program crashes or a user disconnects, the data is safe and can be retrieved later. This also means that the order of the operations is important. You can only read, update, or delete items that were previously created.
It’s good practice to implement each CRUD operation separately in your applications. For example, when you retrieve items, then you shouldn’t update them at the same time.
Note: An exception to this rule may be when you update a “last time retrieved” value after a read operation. Although the user performs a read CRUD operation to retrieve data, you may want to trigger an update operation in the back end to keep track of a user’s retrievals. This can be handy if you want to show the last visited posts to the user.
While CRUD describes a concept that’s independent of specific programming languages, one could argue that CRUD operations are strongly connected to SQL commands and HTTP methods.
What Are CRUD Operations in SQL?The idea of CRUD is strongly connected with databases. That’s why it’s no surprise that CRUD operations correspond almost one-to-one with SQL commands:
CRUD Operation SQL Command Create INSERT Read SELECT Update UPDATE Delete DELETEWhen you create data, you’re using the INSERT command to add new records to a table. After creation, you may read data using SELECT. With a SELECT query, you’re asking the database to retrieve the specific pieces of information you need, whether it’s a single value, a set of records, or complex relationships between data points.
The update operation corresponds to the UPDATE command in SQL, which allows you to modify data. It lets you edit or change an existing item.
Lastly, the delete operation relates to the DELETE command. This is the digital equivalent of shredding a confidential document. With DELETE, you permanently remove an item from the database.
Writing CRUD Operations in Raw SQLCRUD operations describe actions. That’s why it’s a good idea to pull up your sleeves and write some code to explore how CRUD operations translate into raw SQL commands.
In the examples below, you’ll use Python’s built-in sqlite3 package. SQLite is a convenient SQL library to try things out, as you’ll work with a single SQLite database file.
You’ll name the database birds.db. As the name suggests, you’ll use the database to store the names of birds you like. To keep the example small, you’ll only keep track of the bird names and give them an ID as a unique identifier.
Read the full article at https://realpython.com/crud-operations/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Evolving Web: Starshot Initiative: Blast-Off for Drupal Beginners
When I first got into web development after graduating from university, I got excited about how easy it was to build a website. A bit of knowledge goes a long way. I learned HTML, CSS, Ruby on Rails, and PHP, and tried all kinds of platforms and content management systems. Sometimes it felt hard and sometimes easy. But it was those “aha” moments when I could prototype a site in an afternoon that really got me excited. Which is why I think Drupal Starshot is such a promising initiative.
The goal of Starshot is to create an out-of-the-box version of Drupal that fast-tracks people through building and launching a website. It’ll attract new users, on-board them quickly, and give them an instant feeling of empowerment.
Starshot will show new users that, yes, you can build enterprise websites with Drupal — but you can also use it to launch a campaign, create an event website, or evolve a digital presence for your startup business.
The Initiative was announced at DrupalCon Portland 2024. Already, more than 386 individuals have pledged support, and dozens of companies have expressed interest too. Starshot is shaping up to be one of the most exciting large-scale initiatives in Drupal’s 23 year history.
“Without a doubt, Starshot will be the largest change to Drupal since the foundational rewrite to modern object-oriented PHP that occurred with Drupal 8.”– Mike Herschel, Senior Front-End Developer at Agileana, former elected member of the Drupal Association Board of Directors
Is Starshot a Rewrite of Drupal?No, Starshot isn’t a rewrite. It is built on Drupal core and include many features from recent Drupal initiatives, including:
- Recipes, packaged features that give you content types and content listings
- Project Browser, for adding more functionality
- Experience Builder, the improved version of Layout Builder for drag-and-drop page building
- Automatic Updates
Drupal Starshot will be available alongside traditional Drupal Core on the Drupal.org download page later this year. Selecting Starshot will install the necessary features for various website use cases, simplifying the process for new users. Traditional Drupal Core will continue to be available for more custom sites.
Watch the Driesnote from DrupalCon Portland 2024 where Starshot was revealed:
How Will Starshot Enhance the Open Web?Starshot will increase Drupal usage and bring the power of open source to even more people by lowering the barrier to entry. It’s a great opportunity to package a version of Drupal that’s more attractive, user-friendly, and engaging for newcomers. This initiative will bring long-overdue improvements to usability, while maintaining Drupal’s strengths: content architecture, security, and multilingual support.
A key component of Starshot's development is its focus on the "Builder" persona—users who aspire to create amazing websites with Drupal but may not have extensive web development experience. By leveraging user research with Builders, Starshot will tailor its features and the editing interface around their needs. This means that folks with limited technical expertise can harness the power of Drupal to bring their creative ideas to life.
Starshot can pre-package Drupal with default configurations and pre-configured modules, leveraging the flexible Recipes initiative, and using the new Experience Builder to enhance page creation. Starshot should significantly reduce development and maintenance costs for simpler web builds. Plus, it will set good UX standards for Drupal websites across the board.
“Let’s reach for the stars and bring the open web to all.”– Dries Buytaert, Creator and Project Lead of Drupal
How Can I Contribute?The first releases of Starshot should be available before the end of 2024. The initiative needs lots of support to make this a reality!
Make your pledge on the Starshot landing page or reach out to us with the details of your expertise and availability. We’ll be happy to connect you with a meaningful way to contribute.
Want to help with marketing efforts related to Startshot? Go to the Drupal Marketing page or reach out to me directly.
Take Your Drupal Skills to New HeightsI’ve trained countless people in Drupal and web development over the years — always with the aim of empowering them and inspiring the same passion I have for building websites. If you’re looking to enhance your knowledge and gain valuable tools, you’ve come to the right place. My team and I offer course packages and custom training for site builders, content editors, UX/UI designers, front-end and back-end developers, and entire teams.
Learn more about Drupal training with Evolving Web.
+ more awesome articles by Evolving WebLN Webworks: How to Improve Drupal SEO with the Help of a Global Module
The person with the most trouble with duplicate material is the content writer. The hardest task is having to start over and make the necessary corrections after spending hours writing something and discovering that your content is plagiarized.
This not only gives you a lot of work to do, but it may also interfere with the SEO of your website. Further, we’ll discuss the main problems brought on by duplicate content and discuss how the Drupal SEO module might assist in resolving this issue.
Talk Python to Me: #464: Seeing code flows and generating tests with Kolo
Drupal Starshot blog: Announcing Drupal Starshot sessions
A few weeks ago at DrupalCon Portland, I announced Drupal Starshot, a project to create the new default download of Drupal. Built on Drupal Core, Drupal Starshot will include popular features from the contributed project ecosystem. It focuses on delivering a great user experience right out of the box. Drupal Starshot builds on recent initiatives like Recipes, Project Browser, and Automatic Updates to elevate Drupal to new heights.
The response has been incredible! Hundreds of people have pledged their support on the Drupal Starshot page, and many more have asked how to get involved. Over the past few weeks, we have been planning and preparing, so I'm excited to share some next steps!
We're launching a series of sessions to get everyone up to speed and involved. These will be held as interactive Zoom calls, and the recordings will be shared publicly for everyone to watch at their convenience.
The main goal of these Zoom sessions is to help you get involved in each area. We'll cover details not included in my keynote, update you on our progress, and give you practical advice on where and how you can contribute.
We've scheduled six sessions, and we invite everyone to attend. The first one will be on this Friday on participation, funding, and governance! You can find the latest schedule online at https://www.drupal.org/starshot#sessions and the core calendar in the sidebar of the Drupal core news page.
We look forward to seeing you there and working together to make Drupal Starshot a success!
Drupal Core News: Announcing Drupal Starshot sessions
A few weeks ago at DrupalCon Portland, I announced Drupal Starshot, a project to create the new default download of Drupal. Built on Drupal Core, Drupal Starshot will include popular features from the contributed project ecosystem. It focuses on delivering a great user experience right out of the box. Drupal Starshot builds on recent initiatives like Recipes, Project Browser, and Automatic Updates to elevate Drupal to new heights.
The response has been incredible! Hundreds of people have pledged their support on the Drupal Starshot page, and many more have asked how to get involved. Over the past few weeks, we have been planning and preparing, so I'm excited to share some next steps!
We're launching a series of sessions to get everyone up to speed and involved. These will be held as interactive Zoom calls, and the recordings will be shared publicly for everyone to watch at their convenience.
The main goal of these Zoom sessions is to help you get involved in each area. We'll cover details not included in my keynote, update you on our progress, and give you practical advice on where and how you can contribute.
We've scheduled six sessions, and we invite everyone to attend. The first one will be on this Friday on participation, funding, and governance! You can find the latest schedule online at https://www.drupal.org/starshot#sessions and the core calendar in the sidebar of the Drupal core news page.
We look forward to seeing you there and working together to make Drupal Starshot a success!
simonbaese - blog: Drupal: Asynchronously send emails with Symfony Mailer Queue
Opt Green: KDE Eco's New Sustainable Software Project
Inspired by the successes of the "Blauer Engel Für FOSS" (BE4FOSS) project and KDE's ongoing Sustainable Software goal, KDE Eco has begun a new initiative: "Opt Green: Sustainable Software For Sustainable Hardware" (German: Nachhaltige Software Für Nachhaltige Hardware, or NS4NH).
Opt Green: Sustainable Software For Sustainable HardwareBy design, Free Software guarantees transparency and user autonomy. This gives you, the user, control of your hardware by removing unnecessary vendor dependencies. With Free Software, you're able to use your devices how you want, for as long as you want. There's no bloatware and you can block unwanted data use and ads from driving up energy demands and slowing down your device—while shutting the door to uninvited snooping in your private life as well. With software made for your needs and not the vendors', you can choose applications designed for the hardware you already own. Say goodbye to premature hardware obsolescence: lean, efficient Free Software runs on devices which are decades old!
Independent and sustainable Free Software is good for the users, and good for the environment.
Figure : The "Think Global, Act Local" campaign urged people to consider global health while taking action in their local communities. This new project urges people to do the same, but with computing. (Image from Karanjot Singh published under a CC-BY-4.0 license.)Over the next two years, the "Opt Green" initiative will bring what KDE Eco has been doing for sustainable software directly to end users. A particular target group for the project is those whose consumer behavior is driven by principles related to the environment, and not just price or convenience: the "eco-consumers".
Through online and offline campaigns as well as installation workshops, we will demonstrate the power of Free Software to drive down resource and energy consumption, and keep devices in use for the lifespan of the hardware, not the software.
Our motto: The most environmentally-friendly device is the one you already own.
The topic of software-driven sustainability is relevant for all Free Software applications and developers. We'd love to have you join us and become partners in combatting the issue of software-driven environmental harm. Check out the project's Invent repository or the contact page to get involved today!
Software's Environmental HarmOn 14 October 2025, the end of support for Windows 10 is estimated to make e-waste out of 240 million computers ineligible for the upgrade to Windows 11. Moreover, macOS support for Intel-based Apple computers—the last of which were sold in 2020—is predicted to end (at the earliest) one year later in 2026, rendering even more millions upon millions of functioning devices obsolete. When users have no control over the software they rely on, they are left at a security risk when software support ends ... unless, of course, they purchase a new computer. (By comparison, consider that only in 2022 did Linus Torvalds first suggest ending Linux kernel support for Intel 486 processors from 1989. That's 33 years of support!)
Vendors frequently require buying a new device to support software updates. All too often, this is driven by economic imperatives rather than technological requirements. Moreover, while new hardware has become more and more powerful, new software offering similar or identical functionality has frequently become less efficient and more energy-intensive, which has rendered older, less powerful devices useless.
Already in 2015 Achim Steiner, former Executive Director of the UN Environment Programme (UNEP), warned of the "tsunami of electronic waste rolling out over the world".
In 2016, 44.7 million tonnes of e-waste were generated, estimated to be equivalent to 4500 Eiffel Towers. If you were to stack those Eiffel Towers on top of each other, the result would be 17 times higher than Mount Everest.
By 2017, United Nations University determined e-waste to be the fastest growing waste stream in the world.
In 2022, the amount of e-waste reached 59.4 million tonnes, a 33% increase since 2016.
The flow of e-waste continues to rise today.
Figure : In 2016, 44.7 million metric tons of e-waste was generated. This is estimated to be equivalent to 4,500 Eiffel Towers, which, when stacked, is 17 times higher than Mount Everest. (Image from KDE published under a CC-BY-SA-4.0 license. Design by Anita Sengupta.)Software is a frequently unacknowledged yet significant factor for sustainability. Software determines a hardware's energy consumption and minimum system requirements. It determines how long a device can remain safely in use. With software running on everyday devices, from coffee machines to smartphones, from trains to drones, the role of software in keeping functioning hardware in use and out of the landfill grows more critical every day.
For consumers, the environmental harm may be out-of-sight and out-of-mind. Yet the environment is registering its effects, from the CO2 pumped into the atmosphere to the landfills that receive our discarded devices at their end of life, and the air, soil, and waters around them—not to mention the people and animals.
Figure : A young man is pictured burning electrical wires to recover copper at Agbogbloshie, Ghana, as another metal scrap worker arrives with more wires to be burned. A 2018 article in the "International Journal of Cancer" reports a correlation between proximity to e-waste burn sites and childhood lymphoma. (Image by Muntaka Chasant, published under a CC-BY-SA-4.0 license.)It is particularly devastating when you consider the environmental and social harm caused by e-waste, especially when e-waste is generated earlier than necessary because of premature obsolescence. The production and transportation of a device accounts for 50–80+% of its carbon footprint over its lifecycle. A German Environment Report estimates you’d need to use a computer for over 30 years before efficiency gains in newly-produced devices justify their purchase.
Furthermore, the extraction of rare earth metals in production consumes copious amounts of energy and takes place under miserable social conditions, often in the Global South. For disposal, devices are typically returned to the Global South for end-of-life treatment, where they pollute the environment as toxic waste and cause enormous damage to workers' health or even death.
Figure : Apple's carbon footprint. From Apple (2019), "Environmental Responsibility Report: 2019 Progress Report, covering fiscal year 2018". (Image from KDE published under a CC-BY-SA-4.0 license. Design by Anita Sengupta.) Giving Consumers What They WantGlobally, interest in environmental harm and sustainable goods has been rising steadily from 2015 to 2021. In Europe, a 2020 Eurobarometer poll found that 50% of consumers indicate that two reasons they purchase new devices are performance issues and non-functioning software, and 8 in 10 consumers believe manufacturers should be required to make it easier to repair digital devices.
Free Software already gives consumers what they want, but most don't know it yet. Transparency in code makes lightweight, highly performative software possible, even on much older devices, while user autonomy ensures the right to repair when applications stop functioning.
Figure : KDE’s popular multi-platform PDF reader and universal document viewer Okular was awarded the Blue Angel ecolabel in 2022. (Image from KDE published under a CC-BY-4.0 license.)In fact, the Blue Angel criteria for desktop software are at the forefront in recognizing the critical role of transparency and user autonomy in sustainable software design. From 2021-2023, the KDE Eco project "Blauer Engel Für FOSS" (BE4FOSS) had the goal of collecting and spreading information about the Blue Angel ecolabel among developer communities. In 2022, KDE’s popular PDF and universal document reader Okular became the first ever Blue Angel eco-certified software! The BE4FOSS project culminated with the KDE Eco handbook "Applying The Blue Angel Criteria To Free Software", which you can read here. KDE's Sustainable Software goal has continued this work by developing emulation tools like KdeEcoTest and Selenium AT-SPI to measure software's energy consumption in KDE's KEcoLab.
Now we want to take what we have achieved and bring it directly to eco-consumers.
Through educational campaigns and workshops, the "Opt Green" project aims to combat e-waste by keeping hardware in use with Free Software. Although the problem of software-driven e-waste is relevant for an increasing number of digital devices, the focus will be on desktop PCs, laptops, and, when possible, smartphones and tablets. We are planning to set up info-stands at fair-trade, organic, and artisanal markets as well as sustainability festivals such as the Umweltfestival in Berlin. We will distribute leaflets to consumers, and demo vendor-abandoned devices which are not only usable, but also a joy to use thanks to the tireless work of inspiring Free Software communities. Installation workshops will give users the know-how to keep their devices in use for as long as they want.
Consumers don’t need a new computer to get secure, cutting-edge software; they just need the right software. Free Software already gives consumers what they want today, and we will be working hard to make sure they know that.
Ready To Join Us?Consumers want sustainable and repairable digital devices. We believe that providing users the software to keep devices in use and out of the landfill will drive demand for Free Software products and enable long-term hardware use.
Do you want to join us in this movement to combat e-waste with Free Software? See our contact info to get involved.
We need volunteers like you to bring the "Opt Green" campaign to towns and cities around the world. We need volunteers like you to design engaging guides and beautiful materials for global distribution. We need volunteers like you to report on the project in magazines and newspapers. Let's work together to bring sustainable software to your community!
Maybe you are interested in contributing to the development of measurement tools like KdeEcoTest and Selenium AT-SPI or improving KEcoLab automation? Or using such tools to measure your software application's energy consumption? Let's collaborate to make energy transparency a part of Free Software development today!
Or maybe you actively contribute to a Free Software project that will keep hardware in use for longer. Please be in touch! We want to promote the amazing work you do directly with consumers.
Additional ideas are more than welcome. Part of the project will be figuring out what works and engagement by people like you will make this project a success. We would love to have you join us. Learn more: https://eco.kde.org/get-involved/
Funding NoticeThe NS4NH project is funded by the Federal Environment Agency and the Federal Ministry for the Environment, Nature Conservation, Nuclear Safety and Consumer Protection (BMUV1). The funds are made available by resolution of the German Bundestag.
The publisher is responsible for the content of this publication.
Python Morsels: Equality versus identity in Python
Equality checks whether two objects represent the same value. Identity checks whether two variables point to the same object.
Table of contents
- The equality operator in Python
- The is operator in Python
- How equality and identity work differently?
- Inequality and non-identity operators
- Where are identity checks used?
- Equality vs. Identity
Let's say we have two variables that point to two lists:
>>> a = [2, 1, 3] >>> b = [2, 1, 3, 4]When we use the == operator to check whether these lists are equal, we'll see that they are not equal:
>>> a == b FalseThese lists don't have the same values right now, so they're not equal.
Let's update the first list so that these two lists do have equivalent values:
>>> a.append(4) >>> a [2, 1, 3, 4]If we use == again, we'll see that these lists are equal now:
>>> a == b TruePython's == operator checks for equality. Two objects are equal if they represent the same data.
The is operator in PythonPython also has an is …
Read the full article: https://www.pythonmorsels.com/equality-vs-identity/