Feeds
Ned Batchelder: Coverage at a crossroads
This is an interesting time for coverage.py: I’m trying to make use of new facilities in Python to drastically reduce the execution-time overhead, but it’s raising tricky questions about how coverage should work.
The current situation is a bit involved. I’ll try to explain, but this will get long, with a few sections. Come talk in Discord about anything here, or anything about coverage at all.
- How coverage works today
- SlipCover
- Python 3.12: sys.monitoring
- Using lines for branches
- Problems
- Where are we?
Much of this is discussed in How coverage.py works, but I’ll give a quick overview here to set the stage for the rest of this post.
Trace functionCoverage knows what code is executed because it registers a trace function which gets called for each line of Python execution. This is the source of the overhead: calling a function is expensive. Worse, for statement coverage, we only need one bit of information for each line: was it ever executed? But the trace function will be invoked for every execution of the line, taking time but giving us no new information.
ArcsThe other thing to know about how coverage works today is arcs. Coverage measures branch coverage by tracking the previous line that was executed, then the trace function can record an arc: the previous line number and the current line number as a pair. Taken all together, these arcs show how execution has moved through the code.
Most arcs are uninteresting. Consider this simple program:
1print("Hello")2print("world")
3print("bye")
This will result in arcs (1, 2) and (2, 3). Those tell us that lines 1, 2, and 3 were all executed, but nothing more interesting. Lots of arcs are this kind of straight-line information.
But when there are choices in the execution path, arcs tell us about branches taken:
1a = 12if a == 1:
3 print("a is one!")
4else:
5 print("a isn't one!")
6print("Done")
Now we’ll collect these arcs during execution: (1, 2), (2, 3), (3, 6). When coverage.py analyzes the code, it will determine that there were two possible arcs that didn’t happen: (2, 5) and (5, 6).
The set of all possible arcs is used to determine where the branches are. A branch is a line that has more than one possible arc leaving it. In this case, the possible arcs include (2, 3) and (2, 5), so line 2 is a branch, and only one of the possible arcs happened, so line 2 is marked as a partial branch.
SlipCoverSlipCover is a completely different implementation of a coverage measurement tool, focused on minimal execution overhead. They’ve accomplished this in a few clever ways, primarily by instrumenting the code to directly announce what is happening rather than using a trace function. Synthetic bytecode or source lines are inserted into your code to record data during execution without using a trace function.
SlipCover’s author (Juan Altmayer Pizzorno) and I have been talking for years about how SlipCover and coverage.py each work, with the ultimate goal to incorporate SlipCover-like techniques into coverage.py. SlipCover is an academic project, so was never meant to be widely used and maintained long-term.
One of the ways that SlipCover reduces overhead is to remove instrumentation once it has served its purpose. After a line has been marked as executed, there is no need to keep that line’s inserted bytecode. The extra tracking code can be removed to avoid its execution overhead.
Instrumenting and un-instrumenting code is complex. With Python 3.12, we might be able to get the best aspects of instrumented code without having to jump through complicated hoops.
Python 3.12: sys.monitoringPython 3.12 introduced a new way for Python to track execution. The sys.monitoring feature lets you register for events when lines are executed. This is similar to a classic trace function, but the crucial difference is you can disable the event line-by-line. Once a line has reported its execution to coverage.py, that line’s event can be disabled, and the line will run at full speed in the future without the overhead of reporting to coverage. Other lines in the same file will still report their events, and they can each be disabled once they have fired once. This gives us low overhead because lines mostly run at full speed.
Coverage.py has supported line coverage in 3.12 since 7.4.0 with only 5% overhead or so.
Unfortunately, branch coverage is a different story. sys.monitoring has branch events, but they are disabled based only on the “from” line, not on the “from/to” pair. In our example above, line 2 could branch to 3 or to 5. When sys.monitoring tells us line 2 branched to 3, we have to keep the event enabled until we get an event announcing line 2 branching to line 5. This means we continue to suffer event overhead during execution of the 2-to-3 branch, plus we have to do our own bookkeeping to recognize when both branch destinations have been seen before we can disable the event.
As a result, branch coverage doesn’t get the same advantages from sys.monitoring as statement coverage.
I’ve been talking to the core Python devs about changing how sys.monitoring works for branches. But even if we come to a workable design, because of Python’s yearly release cycle it wouldn’t be available until 3.14 in October 2025 at the earliest.
Using lines for branchesIn SlipCover, Juan came up with an interesting approach to use sys.monitoring line events for measuring branch coverage. SlipCover already had a few ways of instrumenting code, rewriting the code during import to add statements that report execution data. The new idea was to add lines at the destinations of branches. The lines don’t have to do anything. If sys.monitoring reports that the line was executed, then it must mean that the branch to that line happened.
As an example, our code from above would be rewritten to look something like this:
a = 1if a == 1: # line 2
NO-OP # line A, marked as 2->3
print("a is one!") # line 3
else:
NO-OP # line B, marked as 2->5
print("a isn't one!") # line 5
print("Done")
If sys.monitoring reports that line A was executed, we can record a branch from 2 to 3 and disable the event for line A. This reduces the overhead and still lets us later get an event for line B when line 2 branches to line 5.
This seems to give us the best of all the approaches: events can be disabled for each choice from a branch independently, and the inserted lines can be as minimal as possible.
ProblemsThere are a few problems with adapting this clever approach to coverage.py.
Moving away from arcsThis technique changes the data we collect during execution. We no longer get a (1, 2) arc, for example. Overall, that’s fine because that arc isn’t involved in computing branches anyway. But arcs are used as intermediate data throughout coverage.py, including its extensive test suite. How can we move to arc-less measurement on 3.12+ without refactoring many tests and while still supporting Pythons older than 3.12?
I’ve gotten a start on adapting some test helpers to avoid having to change the tests, so this might not be a big blocker.
Is every multi-arc a branch?Another problem is how coverage.py determines branches. As I mentioned above, coverage.py statically analyzes code to determine all possible arcs. Any line that could arc to more than one next line is considered a branch. This works great for classic branches like if statements. But what about this code?
1def func(x):2 try:
3 if x == 10:
4 print("early return")
5 return
6 finally:
7 print("finally")
8 print("finished")
If you look at line 7, there are two places it could go next. If x is 10, line 7 will return from the function because of the return on line 5. If x is not 10, the line 7 will be followed by line 8. Coverage.py’s static analysis understands these possibilities and includes both (7, return) and (7, 8) in the possible arcs, so it considers line 7 a branch. But is it? The conditional here is really on line 3, which is already considered a branch.
I mention this as a problem here because the clever NO-OP rewriting scheme depends on being able to insert a line at a destination that clearly indicates where the branch started from. In this finally clause, where would we put the NO-OP line for the return? The rewriting scheme breaks down if the same destination can be reached from different starting points.
But maybe those cases are exactly the ones that shouldn’t be considered branches in the first place?
Where are we?I’ve been hacking on this in a branch in the coveragepy repo. It’s a complete mess with print-debugging throughout just to see where this idea could lead.
For measuring performance, we have a cobbled-together benchmark tool in the benchmark directory of the repo. I could use help making it more repeatable and useful if you are interested.
I’m looking for thoughts about how to resolve some of these issues, and how to push forward on a faster coverage.py. There’s now a #coverage-py channel in the Python Discord where I’d welcome feedback or ideas about any of this, or anything to do with coverage.py.
Real Python: The Real Python Podcast – Episode #209: Python's Command-Line Utilities & Music Information Retrieval Tools
What are the built-in Python modules that can work as useful command-line tools? How can these tools add more functionality to Windows machines? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Web Review, Week 2024-25
Let’s go for my web review for the week 2024-25.
Proton is transitioning towards a non-profit structureTags: tech, internet, ethics, privacy
Very interesting move. I wish them well!
https://proton.me/blog/proton-non-profit-foundation
Tags: tech, java
Oracle doing Oracle things I guess… The surprising bit to me is the fact that so many people still seem to use Java SE while there are other excellent alternatives.
https://www.theregister.com/2024/06/20/oracle_java_licence_teams/
Tags: tech, microsoft, security
A deep dive into the events which led to the SolarWinds breaches. The responsibility from Microsoft as an organization is staggering. Their handling of security matters massively failed once more. I don’t get how governmental agencies or other companies can still turn to Microsoft with sensitive data.
https://www.propublica.org/article/microsoft-solarwinds-golden-saml-data-breach-russian-hackers
Tags: tech, microsoft, security
Very unsurprising, the harm is probably done though. They’ll have to work hard for their reputation to recover (even though it was probably low already).
Tags: tech, gpt, surveillance
It was already hard to trust this company, but now… that clearly gives an idea of the kind of monetization channels they’re contemplating.
https://futurism.com/the-byte/snowden-openai-calculated-betrayal
Tags: tech, ai, machine-learning, gpt, copilot, security, privacy
The creative ways to exfiltrate data from chat systems built with LLMs…
https://embracethered.com/blog/posts/2024/github-copilot-chat-prompt-injection-data-exfiltration/
Tags: tech, ai, machine-learning, gpt, data-science, criticism, funny
OK, this is a rant about the state of the market and people drinking kool-aid. A bit long but I found it funny and well deserved at times.
https://ludic.mataroa.blog/blog/i-will-fucking-piledrive-you-if-you-mention-ai-again/
Tags: tech, ai, machine-learning, gpt, self-hosting, criticism
Since there are ways to offset the plagiarism a bit, let’s do it. Obviously it’s not perfect but that’s a start.
https://blog.zgp.org/block-ai-training-on-a-web-site/
Tags: tech, foss, maintenance, life, history
Very interesting piece… shows how someone can end up maintaining something essential for decades. This is a lesson for us all.
https://lwn.net/SubscriberLink/978463/be23210c163a2107/
Tags: tech, networking, security, self-hosting, internet
This is indeed a real concern… with no propre solution in sight.
https://www.macchaffee.com/blog/2024/ddos-attacks/
Tags: tech, networking, security
On the peculiarities of running a network for a university… this is an interesting way to frame it as basically being an ISP with benefits.
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/OurNetworkTrafficIsUnknown
Tags: tech, shell, scripting
This is indeed an easy mistake to do. It’s better be avoided.
https://mywiki.wooledge.org/ParsingLs
Tags: tech, tools, git, cad
Interesting trick for a zip based format containing mostly text.
https://blog.lambda.cx/posts/freecad-and-git/
Tags: tech, python, memory, performance
Interesting dive into how join() and generator behave in CPython.
https://berglyd.net/blog/2024/06/joining-strings-in-python/
Tags: tech, programming, python
That’s what happens where references are half hidden in a language. You think each closure get a different copy but in fact they all refer to the same object.
https://utcc.utoronto.ca/~cks/space/blog/python/UnderstandingClosureOddity
Tags: tech, json, security
JSON, its grammar and the security implications. The approach of looking at a restricted subset is interesting.
https://neilmadden.blog/2023/05/31/regular-json/
Tags: tech, programming, rust
Ever wondered how this operator is implemented in Rust? It’s not that complicated.
https://blog.sulami.xyz/posts/demystifying-rusts-questionmark-operator/
Tags: tech, data-visualization
Why box plots are hard to grasp and probably badly designed. There are good alternatives out there though.
https://nightingaledvs.com/ive-stopped-using-box-plots-should-you/
Tags: tech, complexity, probability, simulation
Some problems are indeed tackled faster by having a simulation allowing to explore potential solutions. It’s tempting to go very formal and theoretical but it’d require more effort and be more error prone.
https://sirupsen.com/napkin/problem-16-simulation
Tags: tech, library, api, maintenance
Good musing about major version numbers and backward compatibility. It is indeed important to communicate breaking changes properly and to not have those too often.
https://blog.cessen.com/post/2022_07_09_major_version_numbers_may_not_be_sacred
Tags: tech, software, programming, work, complexity
It might not look like a lot from the outside, but “just implementation details” in fact hides quite some work and complexity.
https://ntietz.com/blog/whats-behind-just-implementation/
Tags: tech, software, organization, complexity
Very nice piece about the various types of complexities we encounter in our trade, and what we can or should do about it.
https://olano.dev/blog/a-note-on-essential-complexity
Tags: tech, software, management
This is a funny pretense, and yet… If any of this remind you of a real context, this would be paper cuts. Have enough of those and indeed the organization might grind to a halt.
https://erikbern.com/2023/12/13/simple-sabotage-for-software.html
Tags: tech, requirements, software, product-management
This is indeed a good way to classify events probability in requirements. It definitely impacts how you handle them in software.
https://lukeplant.me.uk/blog/posts/never-sometimes-always/
Tags: tech, communication, talk
Nice trick, definitely should use it more often.
https://tidyfirst.substack.com/p/start-presentations-on-the-second
Tags: tech, information, social-media, criticism
Indeed the analogy from “ultra-processed food” is an interesting one in the information context.
https://calnewport.com/on-ultra-processed-content/
Bye for now!
mark.ie: My Drupal Core Contributions for week-ending June 21th, 2024
Here's what I've been working on for my Drupal contributions this week. Thanks to Code Enigma for sponsoring the time to work on these.
Bits from Debian: Looking for the artwork for Trixie the next Debian release
Each release of Debian has a shiny new theme, which is visible on the boot screen, the login screen and, most prominently, on the desktop wallpaper. Debian plans to release Trixie, the next release, next year. As ever, we need your help in creating its theme! You have the opportunity to design a theme that will inspire thousands of people while working in their Debian systems.
For the most up to date details, please refer to the wiki.
We would also like to take this opportunity to thank Juliette Taka Belin for doing the Emerald theme for bookworm.
The deadlines for submissions is: 2024-09-19
The artwork is usually picked based on which themes look the most:
- ''Debian'': admittedly not the most defined concept, since everyone has their own take on what Debian means to them.
- ''plausible to integrate without patching core software'': as much as we love some of the insanely hot looking themes, some would require heavy GTK+ theming and patching GDM/GNOME.
- ''clean / well designed'': without becoming something that gets annoying to look at a year down the road. Examples of good themes include Joy, Lines, Softwaves and futurePrototype.
If you'd like more information or details, please post to the Debian Desktop mailing list.
Bits from Debian: Recherche d'un thème pour Trixie, la prochaine version de Debian
Chaque édition de Debian bénéficie d'un nouveau thème brillant visible sur l'écran d'amorçage, l'écran de connexion et, de la façon la plus évidente, sur le fond d'écran. Debian prévoit de publier Trixie, sa prochaine version, durant l'année à venir. Et, comme toujours, nous avons besoin de vous pour créer son thème ! Vous avez l'occasion de concevoir un thème qui inspirera des milliers de personnes quand ils travaillent sur leur machine Debian.
Pour disposer des détails les plus récents, veuillez vous référer à la page du wiki.
Nous voudrions profiter de cette occasion pour remercier Juliette Taka Belin pour avoir créé le thème Emerald pour Bookworm.
La date limite pour les propositions est le 19 septembre 2024.
Le thème est habituellement choisi en se basant sur ce qui paraît le plus :
- « Debian » : il est vrai que ce n'est pas le concept le mieux défini, dans la mesure où chacun a son sentiment sur ce que représente Debian pour eux ;
- « possible à intégrer sans corriger le logiciel de base » : autant nous aimons les thèmes follement excitants, certains nécessiteraient un gros travail d'adaptation des thèmes GTK+ et la correction de GDM/GNOME ;
- « clair et bien conçu » : sans être quelque chose qui devient ennuyeux à regarder au bout d'un an. Parmi les bons thèmes, on peut citer Joy, Lines, Softwaves et futurePrototype.
Si vous souhaitez disposer plus d'informations, veuillez utiliser la liste de diffusion Debian Desktop.
Bits from Debian: Looking for the artwork for Trixie the next Debian release
Each release of Debian has a shiny new theme, which is visible on the boot screen, the login screen and, most prominently, on the desktop wallpaper. Debian plans to next release, Trixie, next year. As ever, we need your help in creating its theme! You have the opportunity to design a theme that will inspire thousands of people while working in their Debian systems.
For the most up to date details, please refer to the wiki.
We would also like to take this opportunity to thank Juliette Taka Belin for doing the Emerald theme for bookworm.
The deadlines for submissions is: 2024-09-19
The artwork is usually picked based on which themes look the most:
- ''Debian'': admittedly not the most defined concept, since everyone has their own take on what Debian means to them.
- ''plausible to integrate without patching core software'': as much as we love some of the insanely hot looking themes, some would require heavy GTK+ theming and patching GDM/GNOME.
- ''clean / well designed'': without becoming something that gets annoying to look at a year down the road. Examples of good themes include Joy, Lines, Softwaves and futurePrototype.
If you'd like more information or details, please post to the Debian Desktop mailing list.
mark.ie: A bash script to set up Drupal for local development using DDEV
Last week I wrote about how to set up Drupal for core contributing using DDEV. This week I decided to write a bash script so I wouldn't have to remember what I did, it would "just work".
health @ Savannah: MyGNUHealth 2.2 series released!
Dear all
I am happy to announce the release of MyGNUHealth 2.2.0!
The new series of the GNU Health Personal Health record comes with many improvements and bug fixes. Some highlights of this new version:
- Support for Kivy 2.3.0
- Localization. MyGNUHealth now has support for different languages. English, Spanish and Chinese are available to use, and French, German, Italian are ready to be translated. There will be a translation component for MyGNUHealth at Codeberg's Weblate instance.
- Bluetooth functionality: Starting with MyGH series 2.2 we provide bluetooth integration for open compatible devices and health trackers. We include the link with the Pinetime Smartwatch (experimental) and the possibility to link to any open hardware device (glucometer, scales, blood pressure monitors, .. ). We need to get a list of available medical devices that respect our privacy and freedom, so let us know of any!
- Charts now allow to select date ranges with calendar widgets
- The Book of Life have a revised format for the pages.
- The charts have been improved in the format and include x axis labels.
Thanks to Kivy, Mygnuhealth codebase can be ported to other architectures and operating systems such as Android AOSP (Pierre Michel is working on this) and GNU/Linux phones.
In addition to Savannah, we have incorporated Codeberg to the GNU Health development environment. Mailing lists, news and file downloads are at GNU, while the development repositories are at Codeberg (https://codeberg.org/gnuhealth)
You can download the latest MyGNUhealth sourcecode from GNU ftp site, pypi (using pip) or from your operating system package (like openSUSE).
Upgrading should be straightforward, and all the health history will remain in the MyGH database. In any case, please make sure you make a backup before upgrading (and daily ;) ).
Thank you to all the contributors that have possible this milestone!
Happy hacking
Luis
death and gravity: reader 3.13 released – scheduled updates
Hi there!
I'm happy to announce version 3.13 of reader, a Python feed reader library.
What's new? #Here are the highlights since reader 3.12.
Scheduled updates #reader now allows updating feeds at different rates via scheduled updates.
The way it works is quite simple: each feed has an update interval that determines when the feed should be updated next; calling update_feeds(scheduled=True) updates only feeds that should be updated at or before the current time.
The interval can be configured by the user globally or per-feed through the .reader.update tag. In addition, you can specify a jitter; for an interval of 24 hours, a jitter of 0.25 means the update will occur any time in the first 6 hours of the interval.
In the future, the same mechanism will be used to handle 429 Too Many Requests.
Improved documentation #As part of rewriting the Updating feeds user guide section to talk about scheduled updates, I've added a new section about being polite to servers.
Also, we have a new recipe for adding custom headers when retrieving feeds.
mark_as_read reruns #You can now re-run the mark_as_read plugin for existing entries by adding the .reader.mark-as-read.once tag to a feed. Thanks to Michael Han for the pull request!
That's it for now. For more details, see the full changelog.
Want to contribute? Check out the docs and the roadmap.
Learned something new today? Share this with others, it really helps! PyCoder's Weekly HN Reddit linkedin Twitter
What is reader? #reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.
reader allows you to:
- retrieve, store, and manage Atom, RSS, and JSON feeds
- mark articles as read or important
- add arbitrary tags/metadata to feeds and articles
- filter feeds and articles
- full-text search articles
- get statistics on feed and user activity
- write plugins to extend its functionality
...all these with:
- a stable, clearly documented API
- excellent test coverage
- fully typed Python
To find out more, check out the GitHub repo and the docs, or give the tutorial a try.
Why use a feed reader library? #Have you been unhappy with existing feed readers and wanted to make your own, but:
- never knew where to start?
- it seemed like too much work?
- you don't like writing backend code?
Are you already working with feedparser, but:
- want an easier way to store, filter, sort and search feeds and entries?
- want to get back type-annotated objects instead of dicts?
- want to restrict or deny file-system access?
- want to change the way feeds are retrieved by using Requests?
- want to also support JSON Feed?
- want to support custom information sources?
... while still supporting all the feed types feedparser does?
If you answered yes to any of the above, reader can help.
The reader philosophy #- reader is a library
- reader is for the long term
- reader is extensible
- reader is stable (within reason)
- reader is simple to use; API matters
- reader features work well together
- reader is tested
- reader is documented
- reader has minimal dependencies
So you can:
- have full control over your data
- control what features it has or doesn't have
- decide how much you pay for it
- make sure it doesn't get closed while you're still using it
- really, it's easier than you think
Obviously, this may not be your cup of tea, but if it is, reader can help.
Gunnar Wolf: A new RISC-V toy... requiring almost no tinkering
Shortly before coming back from Argentina, I got news of a very interesting set of little machines, the MilkV Duo. The specs looked really interesting and fun to play with, particularly those of the “bigger” model, Milk-V DUO S Some of the highlights:
- The SG2000
SoC
is a Dual-architecture beast. A hardware switch controls whether the CPU is an
ARM or a RISC-V.
- Not only that: It has a second (albeit lesser) RISC-V core that can run independently. They mention this computer can run simultaneously Linux and FreeRTOS!
- 512MB RAM
- Sweet form factor (4.2×4.2cm)
- Peeking around their Web site, it is one of the most open and well documented
computers in their hardware range.
- Schematics at different levels of detail
- Datasheet. It’s preliminary, and it’s written in Mandarin, but the information it contains is quite descriptive and clear.
- Full specifications (all 710 pages of it!) Again, it has a lot of Mandarin in it, but it conveys a lot of useful information.
Naturally, for close to only US$12 (plus shipping) for the configuration I wanted… I bought one, and got it delivered in early May. The little box sat on my desk for close to six weeks until I had time to start tinkering with it…
I must say I am surprised. Not only the little bugger delivers what it promises, but it is way more mature than what I expected: It can be used right away without much tinkering with! I mean, I have played with it for less than an hour by now, and I’ve even managed to get (almost) regular Debian working.
Milk-V distributes a simple, 58.9MB compressed Linux image, based on Buildroot, a simple Linux image generator mostly used for embedded applications, as well as its source tree. I thought that would be a good starting point to work on setting up a minimal Debian filesystem, as I did with the CuBox-i4Pro ten years ago, and maybe even to grow towards a more official solution, akin to what we currently have for the Raspberry Pi family…
…Until I discovered what looks like a friendly and very active online community of Milk-V users! I haven’t yet engaged in it, but I stumbled across a thread announcing the availability of Debian images for the Milk-V family.
And yes, it feels like a very normal Debian system. /etc/apt/sources.list does point to a third-party repository, but it’s for only four packages, all related to pinmux controlfor CVITEK chips. It does feel like a completely normal Debian system! It is not as snappy and fast to load as Buildroot, but given Debian’s generality, that’s completely as expected. Even the wireless network, one of the usual pain points, works just out of the box! The Debian images can be built or downloaded from this Git repository.
In case you wonder how is this system booting or what hardware it detects, I captured two boot logs:
Krita Monthly Update – Edition 16
Welcome to the latest development and community news curated for you by the Krita-promo team.
Development report Krita is 25 years old!Artwork by David Revoy (CC BY-SA)
May 31, 2024 marks Krita’s 25th birthday. As one would expect, there have been many changes over the years – even the name changed several times. You can get a look inside Krita’s history in this blog post written by @Halla, Krita’s Maintainer for more than 20 years.
In honour of this milestone, @RamonM prepared a special treat for all Krita users: a video interview with @Halla.
Your feedback is requested-
5.2.3-Beta1 was released June 5th. This release represents a complete rework of the build system and numerous fixes by the core Krita developer team as well as freyalupen, Grum999, NabilMaghfurUsman, Deif_Lou, Alvin Wong, Rasyuqa A. H. and Mathias Wein. There are a number of first-time contributors whose names appear next to their contribution in the release notes.
-
Testing packages for every platform are provided on the release notes page. Please report your findings and feedback in the Testers Wanted thread.
-
Text property editing: Merge request 2092 is almost finished, awaiting review. Testing builds are now available on the CI. @Wolthera is requesting user feedback on the UX. Please read this post and share your comments there.
-
Google Summer of Code (GSOC) @Ken_Lo is seeking input on the Pixel Perfect project https://krita-artists.org/t/pixel-perfect-line-setting-for-pixel-art-brushes/42629/16
- Free transform bounding box rotation by Stuffin has been merged. This completes the feature request Adjusting the transform box to match the object angle in the drawing and can be tested in the 5.3.0-prealpha nightly. (Note to testers: Adjusting the bounding box is activated with Ctrl+Alt.) Thanks stuffin!
-
Grum999 is improving the python API so that it is more robust for python developers and they can access more of krita’s internal features through python. There is a work-in-progress MR to add new scripting functions for accessing Grids, Guides, and Mirror Axes from the document, and signals for changes in the document and view. Check the MR
-
@Ralek has added lossless transformation conditions - Rotations in increments of 90 degrees, and perfect x and y mirrors should now be lossless. This should greatly help out pixel artists, who I believe previously could not use these functions at all. Check the MR
Community Report May 2024 Monthly Art Challenge
And the winner is… Cat Reflection by Elixiah.
For the June Art Challenge, Elixiah has chosen Magnificent Dragon with an interesting optional challenge for any who care to give themselves an added stretch.
Featured artworkTen images were submitted to the Best of Krita-Artists Nominations thread which was open from April 14th to May 11th. When voting closed on May 14th, these five had the most votes and were added to the Krita-Artists featured artwork banner.
Quiet Morning by @Gurkirat_Singh.
Pollinatrix Terrae by @jimplex.
The Lone Rider-2 by @rohithela.
005 (Spider in the web) by @HappyBuket.
Challenge Horn by @MangooSalade.
In addition to their place of honour on the banner, all five will be entered into the Best of Krita-Artists 2024 competition next January. The Best of Krita-Artists May/June Nominations thread will be open for submissions until June 11, 2024. You are invited to join in by nominating your favourite piece of Krita artwork!
Noteworthy PluginCreate a New View as Window and Topped by Cliscylla saves steps by opening a new view and setting it to always stay on top.
Tutorial of the monthHow to record video directly from Krita and post to social media by Deevad is a comprehensive tutorial for beginner and intermediate Krita users. It takes the viewer through the initial screen set up and recommended canvas dimensions right through to the export process.
Ways to help KritaKrita is a Free and Open Source application, mostly developed by an international team of enthusiastic volunteers. Donations from Krita users to support maintenance and development is appreciated.
Visit Krita’s funding page to see how donations are used and explore a one-time or monthly contribution.
Notable Changes in the codeThis section has been compiled by freyalupen. (May 6 - June 6, 2024)
Stable branch (5.2.3-beta1):Bugfixes:
- General Don't waste memory generating empty animation frames on images with no animation. This was a regression in 5.2.x. (commit, Dmitry Kazakov)
- Storyboard Docker Fix reordering storyboard scenes causing all frame data to be deleted while still appearing to be present. (BUG:476440) (merge request, Freya Lupen)
- Android: Animation Fix crash when attempting to load audio on Android, a regression present in 5.2.2.1. (merge request, Dmitry Kazakov)
Stable branch (5.2.3-beta1+):
Bugfixes:
- Animation Fix crash when adding a keyframe column with a locked layer selected. (BUG:486893) (commit, Dmitry Kazakov)
- Keyboard shortcuts While continuing making a Selection, ignore other modifier shortcuts to avoid conflicts. (merge request, Aqaao)
- File Formats: TIFF Ask to use PSD data in TIFF only if any was found. (BUG:488024) (commit, Freya Lupen)
- General, macOS Fix update of "read-only" state of the document when loading and saving. Fixes a crash on macOS when loading TIFF or JPEG-XL recent file icons (which load a temporary document). (BUG:487544) (commit, Dmitry Kazakov)
- Android, Recorder Docker Fix saving Recorder frames as JPEG on Android, a regression present in 5.2.2.1. (BUG:487667) (commit, Dmitry Kazakov)
- Android, General Improve Krita's apk icons to follow Android design guidelines. (BUG:463043) (merge request, Jesse 205)
- Scripting Generate a Python type stub file for Krita's API, which can be used to setup type auto-completion in IDEs, located inside the Krita package at /lib/krita-python-libs/PyKrita/krita.pyi. (merge request, Kate Corcoran)
Stable branch (5.2.3-beta1+) backports from Unstable: Bugfixes:
- Recorder Docker Reworked default recorder docker FFmpeg profiles. If canvas size changes during recording, the export profiles now keep aspect instead of stretching (BUG:429326). Issues with resize, result preview, and extend result are avoided (BUG:455006, BUG:450790, BUG:485515, BUG:485514). For MP4, detect whether openh264 or libx264 is present instead of using separate profiles. Also, prevent an error when using FFmpeg 7. (merge request, Ralek Kolemios)
- Selection Tools Fix issue making selections on color-labeled reference selections. (BUG:486419) (commit, Deif Lou)
Features:
- Transform Tool Allow rotating the free transform bounding box with Ctrl+Alt, in order to make transformations along an arbitrary axis. (WISHBUG:383587) (merge request, Stuffins)
Bugfixes:
- Transform Tool Make sure perfect mirrors and 90-degree rotations are transformed losslessly. (merge request, Ralek Kolemios)
- Shortcuts Fix Sample Screen Color getting stuck if activated multiple times without completing. (BUG:485739) (merge request, Deif Lou)
- Scripting Fix setting Color Adjustment (perchannel) and Cross-Channel filters from Python scripts. (merge request, Deif Lou)
These changes are made available for testing in the following Nightly builds:
- Stable "Krita Plus" (5.2.3-beta1+): Linux - Windows - macOS [unsigned currently] - Android (arm64-v8a / arm32-v7a / x86_64)
- Unstable "Krita Next" (5.3.0-prealpha): Linux - Windows - macOS [unsigned currently] - Android (arm64-v8a / arm32-v7a / x86_64)
Krita is a free and open source project. Please consider supporting the project with donations or by buying training videos or the artbook! With your support, we can keep the core team working on Krita full-time.
Donate Buy somethingSeth Michael Larson: CPython vulnerability data infrastructure (CVE and OSV)
Published 2024-06-21 by Seth Larson
Reading time: minutes
Let's talk about some vulnerability data infrastructure for the Python Software Foundation (PSF). In the recent past, most of the vulnerability data processes were manual. This worked okay because the PSF processes a low double-digit number of vulnerabilities each year (2023 saw 12 published vulnerabilities).
However, almost all of this manual processing was being done by me as full-time staff. Imagining this work being done by either someone else on staff or a volunteer isn't great, because it's a non-zero amount of extra work. Automation to the rescue!
How the vulnerability data flowsThe PSF uses the CVE database as its “source of truth” for vulnerability data which then gets imported into our Open Source Vulnerability (OSV) database by translating CVE records into OSV records.
We manually update CVE information in a program called Vulnogram which provides a helpful UI for CVE services.
So what is the minimum amount of information we need to manually create to automatically generate the rest? This is the current list of data the PSF CVE Numbering Authority team creates manually:
- Advisory text and description
- CVE reference to the advisory
- GitHub issue (as a CVE reference)
GitHub RepositoryGitHub Reposito...CVE
ServicesCVE...PSF OSV DatabasePSF OSV DatabaseGitHub
IssueGitHub...GitHub
Merged PRGitHub...git commitgit commitgit tag
(v3.12.4)git tag...CVE RecordCVE RecordCVE Affected VersionsCVE Affected...CVE ReferencesCVE ReferencesOSV RecordOSV RecordOSV Affected CommitsOSV Affected...OSV Affected TagsOSV Affected...OSV ReferencesOSV ReferencesText is not SVG - cannot display
Blue items are manually created, green items are automatically generated.
Advisories are sent to the security-announce@python.org mailing list and then the description is reused as the CVE record's description. The linkages between a GitHub pull request and a GitHub issue is maintained by Bedevere which automatically updates metadata as new pull requests are opened for an issue.
From this information we can use scripts to generate the rest in two stages:
- CVE affected versions and references are populated by finding git tags that contain git commits.
- All OSV record information is generated from CVE records. New OSV records are automatically assigned their IDs by OSV tooling. The central OSV database calculates affected git tags on our behalf.
The PSRT publishes advisories and patches once they're available in the latest CPython versions. For low-severity vulnerabilities there typically isn't an immediate release of all bugfix and security branches (ie 3.12, 3.11, 3.10, 3.9, etc). This means that many vulnerability records will be need to be updated over time as fixes are merged and released into other CPython versions so these scripts run periodically to avoid tracking these updates manually.
Other items- This week marks my one-year anniversary in the role of Security Developer-in-Residence. Woohoo! 🥳
- Advisories and records were published for CVE-2024-0397 and CVE-2024-4032.
- Wrote the draft for "Trusted Publishers for All Package Repositories" for the OpenSSF Securing Software Repositories WG. This document would be used by other package repositories looking to implement Trusted Publishers like PyPI has.
- Working on documenting more of PSRT processes like membership.
- Triaging reports to the PSRT.
- Reviewing PEP 740 and other work for generating publish provenance from William Woodruff.
- Google Summer of Code contributor, Nate Ohlson has been making excellent progress on the effort for CPython to adopt the "Hardened Compiler Options Guide" from the OpenSSF Best Practices WG. You can follow along with his progress on Mastodon and on GitHub.
That's all for this post! 👋 If you're interested in more you can read the last report.
Thanks for reading! ♡ Did you find this article helpful and want more content like it?
Get notified of new posts by subscribing to the RSS feed or the email newsletter.
This work is licensed under CC BY-SA 4.0
The Python Show: 45 - Computer Vision and Data Science with Python
In this episode, we welcome Kyle Stratis from Real Python to the show to chat with us about computer vision and Python.
We chatted about several different topics including:
Writing about Python on Real Python
Data science
Artificial intelligence
Python packaging
and much more!
Paolo Melchiorre: Django 5 by Example preface
The story of my experience in writing the preface of the book “Django By Example” by Antonio Melé.
The Drop is Always Moving: Drupal 10.3.0 is out! The third and final feature release of Drupal 10 ships with a new experimental Navigation UI, stable Workspaces and Single-Directory Components, simplified menu editing, taxonomy moderation, new recipe...
Drupal 10.3.0 is out! The third and final feature release of Drupal 10 ships with a new experimental Navigation UI, stable Workspaces and Single-Directory Components, simplified menu editing, taxonomy moderation, new recipe and access policy APIs and more. https://www.drupal.org/blog/drupal-10-3-0
Drupal blog: Drupal 10.3 is now available
The third and final feature release of Drupal 10 ships with a new experimental Navigation user interface, stable Workspaces functionality, stable Single-Directory Components support, simplified menu editing, taxonomy moderation support, new recipe and access policy APIs, performance improvements and more.
New experimental Navigation moduleThe new Navigation module provides a redesigned collapsible, vertical navigation sidebar for the administrative user interface. Sub-menus open on a full height drawer that can accommodate deeper navigation levels. On smaller viewports, the toolbar is placed on top of the content, and opens with an overlay.
The Navigation module allows multiple types of customization, like adding new custom menus or changing the default Drupal logo provided. It also uses the Layout Builder module, so that site builders can easily add or reorder these menu blocks.
The Navigation module includes a new content creation and management menu, which allows quick access to content-related tasks to increase usability for content users.
Stable Workspaces moduleThe Workspaces module allows Drupal sites to have multiple work environments, enabling site owners to stage multiple content changes to be deployed to the live site all at once. It has long been available in Drupal core as an experimental module. Following the module's use in live projects, the remaining stable blocking issues have been resolved, so now it is available to all!
Workspaces are sets of content changes that are prepared and reviewed separately from the live site. This is a differentiating feature for Drupal that is important for many large organizations' websites. An organization might use Workspaces to ensure all relevant content goes live simultaneously for a new product launch, or with the outcomes of sporting or election events.
Stable Single-Directory ComponentsSingle-Directory Components (SDCs) are Drupal core’s implementation of a user interface components system. Within SDC, all files necessary to render the user interface component are grouped together in a single directory. This includes Twig, YAML, and optional CSS and JavaScript. SDC support was added to Drupal core in 10.1 as an experimental module. The solution has been very well-received and is now part of the base system. No need to enable a module to use this feature.
Simplified content organizationMenu item editing is now simplified. Advanced options are displayed in a sidebar to help content editors focus on what is most important for the menu item. Taxonomy terms also now have both a dedicated user interface to edit earlier revisions and content moderation support.
New Recipes and Default Content APIsDrupal recipes allow the automation of Drupal module installation and configuration. Drupal recipes are easy to share, and can be composed from other Drupal recipes. For example, Drupal 10.3 includes a Standard recipe providing the same functionality as the Standard install profile. It is a combination of 16 component recipes that can be reused in other recipes.
Recipes provide similar functionality to install profiles but are more flexible. With install profiles only one can be installed on a site. With recipes, multiple recipes can be applied after each other.
Install profiles/distributions Recipes Lock-in Not possible to uninstall (until Drupal 10.3) No lock-in Inheritance Cannot extend other profiles or distributions Can be based on other recipes Composability Cannot install multiple profiles or distributions Multiple recipes can be applied on the site and be the basis of another recipeThe recently announced Starshot Initiative will rely heavily on recipes to provide composable features.
The added APIs include Configuration Actions, Configuration Checkpoints and Default Content.
Additionally, it is now possible to install Drupal without an install profile, or to uninstall an install profile after Drupal is already set up.
More flexible access management with the new Access Policy APIThe new Access Policy API supports the implementation of access management solutions that go beyond permissions and user roles. Other conditions and contexts may be taken into account, like whether the user used two-factor authentication, or whether they reached a rate limit of an activity. Drupal's existing permission- and role-based access control has been converted to the new API, and custom or contributed projects can add more access policies.
The future of Drupal 10Drupal 10.3 is the final feature release of Drupal 10. Drupal 11 is scheduled to be released the week of July 29th. With that, Drupal 10 goes into long-term support. While more minor releases will be made available of Drupal 10, they will not contain new features, only functionality backported to support security and a smoother upgrade to Drupal 11. Drupal 10's future minor releases will be supported until mid- to late 2026, when Drupal 12 is released and Drupal 11 enters long-term support.
Core maintainer team updatesCristina Chumillas (at Lullabot), Sally Young (also at Lullabot) and Théodore Biadala (at Très Bien Tech) were all promoted from provisional to full Drupal Core Frontend Framework Managers.
Alex Pott (at Acro Commerce and Thunder), Adam Globus-Hoenich (at Acquia) and Jim Birch (at Kanopi Studios) are the maintainers of the new Default Content and Recipes subsystems.
Andrei Mateescu (at Tag1 Consulting) is the maintainer of the newly stable Workspaces module.
Ivan Berdinsky (at Skilld) became a co-maintainer of the Umami demo.
Daniel Veza (at PreviousNext) is a new co-maintainer of Layout Builder.
Mateu Aguiló Bosch (at Lullabot) and Pierre Dureau are new co-maintainers of the Theme API, focusing on Single Directory Components.
Want to get involved?If you are looking to make the leap from Drupal user to Drupal contributor, or you want to share resources with your team as part of their professional development, there are many opportunities to deepen your Drupal skill set and give back to the community. Check out the Drupal contributor guide, or join us at DrupalCon Barcelona and attend sessions, network, and enjoy mentorship for your first contributions.
mark.ie: My LocalGov Drupal contributions for week-ending June 21th, 2024
Here's what I've been working on for my LocalGov Drupal contributions this week. Thanks to Big Blue Door for sponsoring the time to work on these.
PyBites: Introducing eXact-RAG: the ultimate local Multimodal Rag
Exact-RAG is a powerful multimodal model designed for Retrieval-Augmented Generation (RAG). It seamlessly integrates text, visual and audio information, allowing for enhanced content understanding and generation.
In the rapidly evolving landscape of the Large language model (LLM), the quest for more efficient and versatile models continues unabated.
One of the latest advancements in this realm is the emergence of eXact-RAG, a multimodal RAG (Retrieval Augmented Generation) system that leverages state-of-the-art technologies to deliver powerful results.
eXact-RAG stands out for its integration of LangChain and Ollama for backend and model serving, FastAPI for REST API service, and its adaptability through the utilization of ChromaDB or Elasticsearch.
Coupled with an intuitive user interface built on Streamlit, eXact-RAG represents a significant leap forward in LLMs capabilities.
A Step back: what is a RAG?Retrieval Augmented Generation, or RAG, is an architectural approach that can improve the efficacy of Large Language Model (LLM) applications by leveraging custom data. This is done by retrieving data/documents relevant to a question or task and providing them as context for the LLM. RAG has shown success in support chatbots and Q&A systems that need to maintain up-to-date information or access domain-specific knowledge.
RAG process schema. Source: neo4j.comAs the name suggests, RAG has two phases: retrieval and content generation. In the retrieval phase, algorithms search for and retrieve snippets of information relevant to the user’s prompt or question. In an open-domain, consumer setting, those facts can come from indexed documents on the internet; in a closed-domain, enterprise setting, a narrower set of sources are typically used for added security and reliability.
Understanding eXact-RAGAt its core, eXact-RAG combines the principles of RAG, which focuses on retrieval-based conversational agents, with multimodal capabilities, enabling it to process and generate responses from various modalities such as text, images, and audio.
This versatility makes eXact-RAG well-suited for a wide range of applications, from chatbots to content recommendation systems and beyond.
Technologies Powering eXact-RAG- LangChain and Ollama: LangChain and Ollama serve as the backbone of eXact-RAG, providing robust infrastructure for model development, training, and serving.
LangChain offers a comprehensive suite of tools for natural language understanding and processing, while Ollama specializes in multimodal learning, enabling eXact-RAG to seamlessly integrate and process diverse data types. - FastAPI for REST API Service: FastAPI, known for its high performance and simplicity, serves as the interface for eXact-RAG, facilitating seamless communication between the backend system and external applications.
Its asynchronous capabilities ensure rapid response times, crucial for real-time interactions. - ChromaDB or Elasticsearch: eXact-RAG offers flexibility in data storage and retrieval by supporting both ChromaDB and Elasticsearch.
ChromaDB provides a lightweight solution suitable for simpler tasks, while Elasticsearch caters to more complex operations involving vast amounts of data. This versatility enables users to tailor eXact-RAG to their specific needs, balancing performance and scalability accordingly.
The user interface of eXact-RAG is built on Streamlit, a popular framework for creating interactive web applications with Python. Streamlit’s intuitive design and seamless integration with Python libraries allow users to interact with eXact-RAG effortlessly.
Through the interface, users can input queries, explore results, and interact with generated content across various modalities, enhancing the overall user experience.
Related article: From concepts to MVPs: Validate Your Idea in few Lines of Code with Streamlit
Applications of eXact-RAGThe versatility of eXact-RAG opens up a myriad of applications across different domains:
- Conversational Agents: eXact-RAG can power chatbots and virtual assistants capable of engaging users in natural and meaningful conversations, leveraging both text and multimedia inputs.
- Content Recommendation: By analyzing user preferences and behavior, eXact-RAG can recommend personalized content, including articles, videos, and images, tailored to individual tastes and interests.
- Information Retrieval: eXact-RAG excels at retrieving relevant information from large datasets, making it invaluable for tasks such as question answering, document summarizing, and knowledge base retrieval.
The first step to use eXact-RAG is to run your preferred LLM model using Ollama or get an OpenAI token. Both are supported by the RAG and to be sure to configure them rightly, it’s necessary to fill the settings.toml with the preferred options. Here an example:
[embedding] type = "openai" api_key = "" chat.model_name = "gpt-3.5-turbo" ... [database] type = "chroma" persist_directory = "persist" ...In this example it’s possible to configure the LLM model and the vector database in which store the embeddings of your local data.
eXact-RAG is a multimodal RAG, for this reason it can ingest different kind of data like audio files or images. It is possible to choose, in the installation phase, which “backends” to install:
poetry install # -E audio -E image- audio extra will install openai-whisper for speech-to-text
- image extra will install transformers and pillow for image captioning*
* this feature gives the possibility to process images even if the user has not the (hardware) possibility to run a Vision model like llava locally but still wants to pass images as data.
Now the job is done! The following command
poetry run python exact_rag/main.pystarts the server and at http://localhost:8080/docs it is showed the OpenAPI document (swagger) with all the available endpoints.
DemoeXact-RAG was built as a server for multimodal RAGs but we provide also a user interface just for demo purposes to test all the features.
To run the UI just use the command:
Now, the page at http://localhost:8501 will show a chat interface like in the following example:
ConclusioneXact-RAG represents a significant advancement in the field of multimodal RAG systems, offering unparalleled versatility and performance through its integration of cutting-edge technologies.
With its robust backend powered by LangChain and Ollama, flexible data storage options, and user-friendly interface built on Streamlit, eXact-RAG is poised to revolutionize various applications of natural language processing and multimodal learning.
As the demand for sophisticated LLM solutions continues to grow, eXact-RAG stands ready to meet the challenges of tomorrow’s digital landscape.
C.J. Collier: Signed NVIDIA drivers on Google Cloud Dataproc 2.2
Hello folks,
I’ve been working this year on better integrating NVIDIA hardware with the Google Cloud Dataproc product (Hadoop on Google Cloud) running the default cluster node image. We have an open bug[1] in the initialization-actions repo regarding creation failures upon enabling secure boot. This is because with secure boot, kernel driver code has its signature verified before insmod places the symbols into kernel memory. The verification process involves reading trust root certificates from EFI variables, and validating that the signatures on the kernel driver either a) were made directly by one of the certificates in the boot sector or b) were made by certificates which chain up to one of them.
This means that Dataproc disk images must have a certificate installed into them. My work on the internals will likely start producing images which have certificates from Google in them. In the meantime, however, our users are left without a mechanism to have both secure boot enabled and install out-of-tree kernel modules such as the NVIDIA GPU drivers. To that end, I’ve got PR #83[2] open with the GoogleCloudDataproc/custom-images github repository. This PR introduces a new argument to the custom image creation script, `–trusted-cert`, the argument of which is the path to a DER-encoded certificate to be included in the certificate database in the EFI variables of the disk’s boot sector.
I’ve written up the instructions on creating a custom image with a trusted certificate here:
https://github.com/cjac/custom-images/blob/secure-boot-custom-image/examples/secure-boot/README.md
Here is a set of commands that can be used to create a general purpose GCE disk image with the certificate inserted into EFI.
wget 'https://github.com/cjac/custom-images/raw/secure-boot-custom-image/examples/secure-boot/create-key-pair.sh' bash create-key-pair.sh cacert_der=tls/db.der # The Microsoft Corporation UEFI CA 2011 ms_uefi_ca="tls/MicCorUEFCA2011_2011-06-27.crt" test -f "${ms_uefi_ca}" || \ curl -L -o ${ms_uefi_ca} 'https://go.microsoft.com/fwlink/p/?linkid=321194' JSON="$(gcloud compute images --project debian-cloud list --format json)" SRC_IMAGE_NAME="$(echo ${JSON} | jq -r '.[] | .name' | grep -i ^debian-12-bookworm-v)" NEW_IMAGE_NAME="debian12-with-db-key-list-${USER}-$(date +%F)" SRC_DISK_ZONE="${ZONE}" gcloud -q compute images create "${NEW_IMAGE_NAME}" \ --source-disk "${SRC_IMAGE_NAME}" \ --source-disk-zone "${SRC_DISK_ZONE}" \ --signature-database-file="${cacert_der},${ms_uefi_ca}" \I’d love to hear your feedback!
[1] https://github.com/GoogleCloudDataproc/initialization-actions/issues/1058
[2] https://github.com/GoogleCloudDataproc/custom-images/pull/83