FLOSS Project Planets

Lots of KDE folks are winding down for well-deserved end-of-year breaks, but that didn't stop a bunch of people from landing some awesome changes anyway! This will be a short one, and I may skip next week as many of us are going to be focusing on family time. But in the meantime, check out what we have here:

Notable UI Improvements

When applying screen settings fails due to a graphics driver issue, the relevant page in System Settings now tells you about it, instead of failing silently. (Xaver Hugl, 6.3.0. Link)

Added a new Breeze open-link icon with the typical "arrow pointing out of the corner of a square" appearance, which should start showing up in places where web URLs are opened from things that don't clearly look like blue clickable links. (Carl Schwan, Frameworks 6.10. Link)

Notable Bug Fixes

Fixed one of the most common recent Powerdevil crashes. (Jakob Petsovits, 6.2.5. Link)

Recording a specific window in Spectacle and OBS now produces a recording with the correct scale when using any screen scaling (Xaver Hugl, 6.2.5. Link)

When using a WireGuard VPN, the "Persistent keepalive" setting now works. (Adrian Thiele, 6.2.5. Link)

Implemented multiple fixes and improvements for screen brightness and dimming. (Jakob Petsovits, 6.3.0. Link 1, link 2, link 3, and link 4)

Auto-updates in Discover now work again! (Harald Sitter, 6.3.0. Link)

Vastly improved game controller joystick support in Plasma, fixing many weird and random-seeming bugs. (Arthur Kasimov, 6.3.0. Link)

For printers that report per-color ink levels, System Settings' Printers page now displays the ink level visualization in the actual ink colors again. (Kai Uwe Broulik, 6.3.0. Link)

Pager widgets on very thin floating panels are now clickable in all the places they're supposed to be clickable. (Niccolò Venerandi, 6.3.0. Link)

Wallpapers with very very special EXIF metadata can no longer generate text labels that escape from their intended boundaries on Plasma's various wallpaper chooser views. (Jonathan Riddell and Nate Graham, Frameworks 6.10. Link)

Fixed one of the most common Qt crashes affecting Plasma and KDE apps. (Fabian Kosmale, Qt 6.8.2. Link)

Other bug information of note:

• 2 Very high priority Plasma bugs (same as last week). Current list of bugs
• 34 15-minute Plasma bugs (up from 32 last week). Current list of bugs
• 125 KDE bugs of all kinds fixed over the last week. Full list of bugs

Notable in Performance & Technical

Significantly reduced the CPU usage of System Monitor during the time after you open the app but before you visit to the History page. More CPU usage fixes are in the pipeline, too! (Arjen Hiemstra, 6.2.5. Link)

Plasma Browser Integration now works for the Flatpak-packaged version of Firefox. (Harald Sitter, 6.3.0. Link)

How You Can Help

KDE has become important in the world, and your time and contributions have helped us get there. As we grow, we need your support to keep KDE sustainable.

You can help KDE by becoming an active community member and getting involved somehow. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine!

You don’t have to be a programmer, either. Many other opportunities exist:

• Triage and confirm bug reports, maybe even identify their root cause
• Contribute designs for wallpapers, icons, and app interfaces
• Design and maintain websites
• Translate user interface text items into your own language
• Promote KDE in your local community
• …And a ton more things!

You can also help us by donating to our yearly fundraiser! Any monetary contribution — however small — will help us cover operational costs, salaries, travel expenses for contributors, and in general just keep KDE bringing Free Software to the world.

To get a new Plasma feature or a bugfix mentioned here, feel free to push a commit to the relevant merge request on invent.kde.org.

We did it again™! Just in time, we’re excited to announce the release of Grml stable version 2024.12, code-named ‘Adventgrenze’! (If you’re not familiar with Grml, it’s a Debian-based live system tailored for system administrators.)

This new release is built on Debian trixie, and for the first time, we’re introducing support for 64-bit ARM CPUs (arm64 architecture)!

I’m incredibly proud of the hard work that went into this release. A significant amount of behind-the-scenes effort went into reworking our infrastructure and redesigning the build process. Special thanks to Chris and Darsha – our Grml developer days in November and December were a blast!

For a detailed overview of the changes between releases 2024.02 and 2024.12, check out our official release announcement. And, as always, after a release comes the next one – exciting improvements are already in the works!

BTW: recently we also celebrated 20(!) years of Grml Releases. If you’re a Grml and or grml-zsh user, please join us in celebrating and send us a postcard!

As explained in the previous article in the series I've started working on maintaining Views Data Export again.

I've decided to document my work in 2 week 'sprints'. And so this article is about what I did in Sprint 1.

Sprint progress

At the start of the sprint there in the Drupal.org issue queue there were:

• 204 open bugs
• 276 other open issues.

So that's a total of 480 open issues.

By the end it looked like this:

• 91 open bugs
• 17 fixed issues.
• 81 other open issues

So that's a total of 189 open issues, a 60% reduction from before!

Key goals

In this sprint I wanted to:

• Tame the issue queues on Drupal.org and get a handle on what the common frustrations and missing features were.
• Read and understand all of the code in the Drupal 8.x-1.x branch.

 

Taming the issue queue

As mentioned in a previous article I decided to close down pretty much all the tickets for the Drupal 7 version of the module. This is the codebase that I'm most familiar with, but it's causing a lot of noise in the issue queue, so getting rid of that is a great first step, and pretty easy.

https://www.drupal.org/project/views_data_export/issues/3492246 was my ticket where I detailed what I was going to do, and then I went about doing that.

This felt immensely good! I went through each Drupal 7 ticket and gave it a quick scan and then pasted-in my prepared closing statement. It took just over an hour, and was like taking a trip down memory lane: seeing all those old issues come up and remembering when I triaged some of them originally.

After this initial round of work, I've also been working in the 8.x-1.x queue to close out duplicate and solved issues. I've been focussing on support requests which are usually super quick to evaluate and close out. However, this means that I've not really had a chance to look through all the feature requests and bugs, so I still don't really have a handle on what's needed/broken with the module.

Understanding the code

I had a good old read of the code. There's some really great stuff in there, and there's some obvious room for improvement.

But, at least I know what the code does now, and can see some obvious problems/issues. But also, the codebase is small, and there some automated tests, so we've got a great platform to get going with.

Giving direction

There were a few tickets for 8.x-1.x where there were contributors making great contributions and I was able to provide some guidance of how to implement a feature or resolve a bug. I feel like the issue queue has been lacking any kind of technical leadership and so many tickets are collections of patches where developers are fixing the problem they have in quite a specific way. I'm really looking forward to giving some direction to these contributions and then at some point committing and releasing the great work!

Future roadmap/goals

I'm not committing myself to doing these exactly, or any particular order, but this is my high-level list of hopes/dreams/desires, I'll copy and paste this to the next sprint summary article as I go and adjust as required.

• Get the project page updated with information relevant to Drupal 8.x-1.x version of the module
• Update the documentation on Drupal.org
• Not have any duplicate issues on Drupal.org

Last Chance Proposal Help: MidCamp 2025 Session Proposal Workshop

Missed the last Session Proposal Workshop? Don't worry; we have another one in January right before the submission deadline!

🚀 Ready to take your session ideas to the next level? Whether you're a seasoned speaker or a first-time presenter, the MidCamp 2025 Session Proposal Workshop is here to help you craft standout submissions.

📅 Date: January 7, 2025
🕒 Time: 3:00 PM - 4:00 PM CST
🌐 Location: Virtual via MidCamp Slack (#speakers channel)

Facilitated by Aaron Feledy

This workshop will be led by Aaron Feledy, a seasoned Drupal contributor and experienced speaker. Aaron brings years of expertise in proposal crafting and conference speaking, offering practical advice to help you refine and elevate your session submissions.

Why Attend?

Submitting a session proposal can be daunting—but it doesn't have to be! This workshop is designed to guide you through the process, from brainstorming topics to refining your submission. Our expert facilitators will share insider tips on what makes a proposal stand out to reviewers and resonate with attendees.

What You’ll Learn:

• How to choose and frame a compelling topic
• Crafting clear, concise, and engaging abstracts
• Tips for tailoring your proposal to different audiences
• Insight into the MidCamp review process

Session Submissions Now Open

Ready to submit? Session submissions for MidCamp 2025 are now open! Visit the MidCamp 2025 session submission page for guidelines and start your journey to the stage.

How to Join:

Simply join the MidCamp Slack and head over to the #speakers channel on December 12th at 3:00 PM CST. No registration required—just jump in and start collaborating!

I started this blog back in 2010. Back then I used Wordpress and it worked reasonably well. In 2018 I decided to switch to a static generated site, mostly because the Wordpress blog felt slow to load and it was hassle to maintain. Back then the go-to static site generator was Jekyll, so I went with that. Lately I’ve been struggling with it though, because in order to keep all the plugins working, I needed to use older versions or Ruby, which meant I had to use Docker to build the blog locally. Overall, it felt like too much work and for the past few years I’ve been eyeing Hugo - more so since Carl and others migrated most of KDE websites to it. I mean, if it’s good enough for KDE, it’s good enough for me, right?

So this year I finally got around to do the switch. I migrated all the content from Jekyll. This time I actually went through every single post, converted it to proper Markdown, fixed formatting, images etc. It was a nice trip down the memory lane, reading all the old posts, remembering all the sprints and Akademies… I also took the opportunity to clean up the tags and categories, so that they are more consistent and useful.

Finally, I have rewritten the theme - I originally ported the template from Wordpress to Jekyll, but it was a bit of a mess, responsivity was “hacked” in via JavaScript. Web development (and my skills) has come a long way since then, so I was able to leverage more modern CSS and HTML features to make the site look the same, but be more responsive and accessible.

Comments

When I switched from Wordpress to Jekyll, I was looking for a way to preserve comments. I found Isso, which is basically a small CGI server backed with SQLite that you can run on the server and embed it into your static website through JavaScript. It could also natively import comments from Wordpress, so that’s the main reason why I went with it, I think. Isso was not perfect (although the development has picked up again in the past few years) and it kept breaking for me. I think it haven’t worked for the past few years on my blog and I just couldn’t be bothered to fix it. So, I decided to ditch it in favor of another solution…

I wanted to keep the comments for old posts by generating them as static HTML from the Isso’s SQLite database, alas the database file was empty. Looks like I lost all comments at some point in 2022. It sucks, but I guess it’s not the end of the world. Due to the nature of how Isso worked, not even the Wayback Machine was able to archive the comments, so I guess they are lost forever…

For this new blog, I decided to use Carl’s approach with embedding replies to a Mastodon. I think it’s a neat idea and it’s probably the most reliable solution for comments on a static blog (that I don’t have to pay for, host myself or deal with privacy concerns or advertising).

I have some more ideas regarding the comments system, but that’s for another post ;-) Hopefully I’ll get to blog more often now that I have a shiny new blog!

Happy Holidays 🎄

Enjoy the holidays and see you in 2025 🥳!

Automatically post to BlueSky Anonymous (not verified) Fri, 12/20/2024 - 07:00 Tags AI Web Development BlueSky Social Media Custom Development Automation Drupal Planet

• Read more about Automatically post to BlueSky
• Add new comment

Since the 2024 election, the BlueSky social network has exploded in popularity, and appears to be replacing the cesspool that used to be Twitter. I'm not much of a social media person -- I much prefer hanging out in smaller spaces with people with shared interests. If you're like me, I would highly recommend finding a Mastodon server that caters to your interests, where you're sure to find rewarding conversations.

A coworker asked recently about how people use VMs locally for dev work, so I figured I’d take a few minutes to write up a bit about what I do. There are many use cases for local virtual machines in software development and testing. They’re self-contained, meaning you can make a mess of them without impacting your day-to-day computing environment. They can run different distributions, kernels, and even entirely different operating systems from the one you use regularly. Etc. They’re also cheaper than cloud services and provide finer grained control over the resources.

I figured I’d share a little bit about how I manage different virtual machines in case anybody finds this useful. This is what works for me, but it won’t necessarily work for you, or maybe you’ve already got something better. I’ve found it to be easy to work with, light weight, and is easy to evolve my needs change.

Use short-lived VMs

Rather than keep a long-lived “development” VM around that you customize over time, I recommend automating the common customizations and provisioning new VMs regularly. If I’m working on reproducing a bug or testing a change prior to submitting it upstream, I’ll do this work in a VM and delete the VM when when I’m done. When provisioning VMs this frequently, though, walking through the installation process for every new VM is tedious and a waste of time. Since most of my work is done in Debian, so I start with images generated daily by the cloud team. These images are available for multiple releases and architectures. The ‘nocloud’ variant boots to a root prompt and can be useful directly, or the ‘generic’ images can be used for cloud-init based customization.

Automating image preparation

This makefile lets me do something like make image and get a new qcow2 image with the latest build of a given Debian release (sid by default, with others available by specifying DIST).

DATESTAMP=$(shell date +"%Y-%m-%d") FLAVOR?=generic ARCH?=$(shell dpkg --print-architecture) DIST?=sid RELEASE=$(DIST) URL_PATH=https://cloud.debian.org/images/cloud/$(DIST)/daily/latest/ ifeq ($(DIST),trixie) RELEASE=13 endif ifeq ($(DIST),bookworm) RELEASE=12 endif ifeq ($(DIST),bullseye) RELEASE=11 endif debian-$(DIST)-$(FLAVOR)-$(ARCH)-daily.tar.xz: curl --fail --connect-timeout 20 -LO \ $(URL_PATH)/debian-$(RELEASE)-$(FLAVOR)-$(ARCH)-daily.tar.xz $(DIST)-$(FLAVOR)-$(DATESTAMP).qcow2: debian-$(RELEASE)-$(FLAVOR)-$(ARCH)-daily.tar.xz tar xvf debian-$(RELEASE)-$(FLAVOR)-$(ARCH)-daily.tar.xz qemu-img convert -O qcow2 disk.raw $@ rm -f disk.raw qemu-img resize $@ 20g qemu-img snapshot -c untouched $@ image: $(DIST)-$(FLAVOR)-$(DATESTAMP).qcow2 .PHONY: image Customize the VM environment with cloud-init

While the ‘nocloud’ images can be useful, I typically find that I want to apply the same modifications to each new VM I launch, and they don’t provide facilities for automating this. The ‘generic’ images, on the other hand, run cloud-init by default. Using cloud-init, I can create my user account, point apt at local mirrors, install my preferred tools, ensure the root filesystem is resized to make full use of the backing storage, etc.

The cloud-init configuration on the generic images will read from a local config drive, which can contain an ISO9660 (cdrom) filesystem image. This image can be generated from a subdirectory containing the various cloud-init input files using the following make syntax:

IMDS_FILES=$(shell find seedconfig -path '*/.git/*' \ -prune -o -type f -name '*.in.json' -print) \ seedconfig/openstack/latest/user_data seed.iso: $(IMDS_FILES) genisoimage -V config-2 -o $@ -J -R -m '*~' -m '.git' seedconfig

With the image in place, the VM can be created with

qemu-system-x86_64 -machine q35,accel=kvm -cpu host -m 4g -drive file=${img},index=0,if=virtio,media=disk -drive file=seed.iso,media=cdrom,format=raw,index=2,if=virtio -nic user -nographic

This invokes qemu with the root volume and ISO image attached as disks, uses an emulated “q35” machine with the host’s CPU and KVM acceleration, the userspace network stack, and a serial console. The first time the VM boots, cloud-init will apply the configuration from the cloud-config available in the ISO9660 filesystem.

Alternatives to cloud-init

virt-customize is another tool accomplishing the same type of customization. I use cloud-init because it works directly with cloud providers in addition to local VM images. You could also use something like ansible.

Variations

I have a variant of this that uses a bridged network, which I’ll write more about later. The bridge is nice because it’s more featureful, with full support for IPv6, etc, but it needs a bit more infrastructure in place.

It also can be helpful to use 9p or virtfs to share filesystem state between the host the VM. I don’t tend to rely on these, and will instead use rsync or TRAMP for moving files around.

Containers are also useful, of course, and there are plenty of times when the full isolation of a VM is not worth the overhead.

Curious about how Drupal is evolving, shaping the future of web experiences, and connecting a global community of innovators? Step inside my journey at DrupalCon Singapore 2024, where sessions on cutting-edge technologies, hands-on mentoring, and insightful discussions collided to spark new ideas. From decoupled architectures and automated testing to dynamic SEO enhancements and multilingual site strategies, discover what happened on the conference floor—and how these fresh perspectives can fuel your own Drupal ambitions.

Let’s go for my web review for the week 2024-51.

Advice for First-Time Open Source Contributors

Tags: tech, foss, community

Definitely a good list of advices for first time contributors.

https://www.yegor256.com/2024/12/15/open-source-beginner-advice.html

Europe’s Starlink competitor is go - The Verge

Tags: tech, internet, geospatial

IRIS² is the friendly reminder that tens of thousand of low orbit satellites is not the only design… and likely not the smartest one.

https://www.theverge.com/2024/12/16/24322358/iris2-starlink-rival-europe-date-cost

Buying a TV in 2025? Expect lower prices, more ads, and an OS war. - Ars Technica

Tags: tech, tv, attention-economy, advertisement

The TV market is really turning into an anti-consumer one.

https://arstechnica.com/gadgets/2024/12/buying-a-tv-in-2025-expect-lower-prices-more-ads-and-an-os-war/

Re: Re: Bluesky and Decentralization

Tags: tech, social-media, bluesky, fediverse, architecture

Yet another long piece in this interesting and in depth conversation about Bluesky. The fact that it stays civil is called out explicitly and this is appreciated.

https://dustycloud.org/blog/re-re-bluesky-decentralization/

Bluesky at a crossroads as users petition to ban Jesse Singal over anti-trans views, harassment

Tags: tech, social-media, moderation, bluesky, politics

Bluesky is already hitting growth pains regarding moderation and its guidelines. By being centralized it is also more at risk within the current US political climate.

https://techcrunch.com/2024/12/13/bluesky-is-at-a-crossroads-as-users-petition-to-ban-jesse-singal-over-anti-trans-views-harassment/

Yes, That Viral LinkedIn Post You Read Was Probably AI-Generated

Tags: tech, social-media, linkedin, ai, machine-learning, gpt, fake

Kind of unsurprising right? I mean LinkedIn is clearly a deformed version of reality where people write like corporate drones most of the time. It was only a matter of time until robot generated content would be prevalent there, it’s just harder to spot since even humans aren’t behaving genuinely there.

https://www.wired.com/story/linkedin-ai-generated-influencers/

AI and Internet Hygiene - by kate wagner - the late review

Tags: tech, internet, web, ai, machine-learning, gpt, fake, knowledge

Indeed, we’ll have to relearn “internet hygiene”, it is changing quickly now that we prematurely unleashed LLM content on the open web.

https://www.late-review.com/p/ai-and-internet-hygiene

Is AI progress slowing down?

Tags: tech, ai, machine-learning, gpt, criticism

A good balanced post on the topic. Maybe we’ll finally see a resurgence of real research innovation and not just stupid scaling at all costs. Reliability will stay the important factor of course and this one is still hard to crack.

https://www.aisnakeoil.com/p/is-ai-progress-slowing-down

Analog AI Startup Aims to Lower the Power of Gen AI - IEEE Spectrum

Tags: tech, analogic, ai, machine-learning, neural-networks, hardware

It looks like analog chips for neural network workloads are on the verge of finally becoming reality. This would reduce consumption by an order of magnitude and hopefully more later on. Very early days for this new attempt, let’s see if it holds its promises.

https://spectrum.ieee.org/analog-ai-2669898661

When should we require that firmware be free?

Tags: tech, hardware, foss

A good question, it is somewhat of a grey area at times. We need to come up with better answers.

https://mjg59.dreamwidth.org/70895.html

In search of a faster SQLite

Tags: tech, databases, sqlite, asynchronous, rust, system, filesystem

Interesting explanation of a research paper exploring the possibility of a faster SQLite by focusing on async I/O.

https://avi.im/blag/2024/faster-sqlite/

Java in the Small

Tags: tech, java, tools

I wouldn’t use it as much as advocated in this article, still this is a good reminder that Java became way more approachable for smaller programs in recent years.

https://horstmann.com/unblog/2024-12-11/index.html

How do you do, fellow web developers? A growing disconnect.

Tags: tech, career, complexity, learning

It tries hard at not being a “get off my lawn” post. It clearly points some kind of disconnects though. They’re real. I guess it’s to be expected with the breadth of our industry. There are so many abstractions piled onto each other that it’s difficult to explore them all.

https://rakhim.exotext.com/web-developers-a-growing-disconnect

Visitor Pattern Considered Pointless - Use Pattern Switches Instead

Tags: tech, design, pattern, java, type-systems

One of my favorite of the traditional design patterns in object oriented languages. Now obviously when you get pattern matching in your language… you don’t need the visitor pattern anymore.

https://nipafx.dev/java-visitor-pattern-pointless/

Estimating projects sells them short (and that’s okay)

Tags: tech, project-management, estimates

I don’t exactly use this approach to factor in the uncertainty… but I guess there’s something to be made out of this proposal. I’ll keep it in mind for my next project.

https://ntietz.com/blog/estimating-projects-short-sale/

The One Way I Know a Team is in Trouble

Tags: leadership, management, communication

Interesting ideas about leadership lacking in impact. Indeed it should be seen as a communal function, it’s not about individuals leading each in their own directions. Think about it in a systemic way.

https://suzansfieldnotes.substack.com/p/the-one-way-i-know-a-team-is-in-trouble

Seven quiet breakthroughs for climate and nature in 2024 you might have missed

Tags: ecology, politics, law, energy

This is not all bad news, there are a few things to rejoice about.

https://www.bbc.com/future/article/20241216-seven-quiet-breakthroughs-for-climate-and-nature-in-2024-you-might-have-missed

Bye for now!

What are the current approaches for analyzing emotions within a piece of text? Which tools and Python packages should you use for sentiment analysis? This week, Jodie Burchell, developer advocate for data science at JetBrains, returns to the show to discuss modern sentiment analysis in Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Drupal Advent Calendar day 20 - Navigation james Fri, 12/20/2024 - 09:00

It’s day 20 of the Drupal Advent Calendar, and today we’re looking at the admin UI Navigation. Joining us today are Pablo López and Matthew Oliveira, so let’s look into it…

The aim of the Navigation track is to provide a better site management experience for Drupal users. It does not provide a specific recipe or feature to Drupal CMS. Navigation is a core experimental module. However, the Navigation track provides key integration points to Drupal CMS that will help other tracks to highlight their features in the new Navigation left sidebar.

The navigation sidebar provides an improved interface for site builders and content creators

Since Navigation has replaced Toolbar in Drupal CMS…

Tags

• Navigation
• Admin UI
• Administration
• New Admin Toolbar
• Drupal Starshot

Recap of the CKEditor webinar: Explore AI tools, templates, and collaboration features to level up Drupal content editing. Watch the video now!

Peter Wang has been pushing Python forward since the early days of its data science roots. We're lucky to have him back on the show. We're going to talk about the Anaconda Toolbox for Excel as well as many other trends and topics that are hot in the Python space right now. I'm sure you'll enjoy listening to the two of us exchanging our takes on the topics and trends.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/bluehost'>Bluehost</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Peter on BSky</b>: <a href="https://bsky.app/profile/wang.social?featured_on=talkpython" target="_blank" >@wang.social</a><br/> <b>Michael on BSky</b>: <a href="https://bsky.app/profile/mkennedy.codes?featured_on=talkpython" target="_blank" >@mkennedy.codes</a><br/> <b>Michael's Curated BSky Starter List</b>: <a href="https://bsky.app/starter-pack/mkennedy.codes/3lbdnupl26e2x?featured_on=talkpython" target="_blank" >bsky.app</a><br/> <b>Python Blsky Starter Pack List</b>: <a href="https://blueskydirectory.com/starter-packs/all?q=python&featured_on=talkpython" target="_blank" >blueskydirectory.com</a><br/> <br/> <b>Anaconda Toolbox for Microsoft Excel</b>: <a href="https://www.anaconda.com/products/anaconda-toolbox?featured_on=talkpython" target="_blank" >anaconda.com</a><br/> <b>JupyterLite</b>: <a href="https://jupyter.org/try-jupyter/lab/?featured_on=talkpython" target="_blank" >jupyter.org</a><br/> <b>8 of the Biggest Excel Mistakes of All Time</b>: <a href="https://blog.hurree.co/8-of-the-biggest-excel-mistakes-of-all-time?featured_on=talkpython" target="_blank" >blog.hurree.co</a><br/> <b>The Five Demons of Python Packaging PyBay talk</b>: <a href="https://www.youtube.com/watch?v=qA7NVwmx3gw&ab_channel=SFPython" target="_blank" >youtube.com</a><br/> <b>PEP 759</b>: <a href="https://peps.python.org/pep-0759/?featured_on=talkpython" target="_blank" >peps.python.org</a><br/> <b>TIOBE Index</b>: <a href="https://www.tiobe.com/tiobe-index/?featured_on=talkpython" target="_blank" >tiobe.com</a><br/> <b>pyscript</b>: <a href="https://pyscript.net/?featured_on=talkpython" target="_blank" >pyscript.net</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=H2mFIZnVBok" target="_blank" >youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/489/anaconda-toolbox-for-excel-and-more-with-peter-wang" target="_blank" >talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

In recent weeks we have been working on transferring LabPlot’s documentation to a new format.

We decided to move the documentation from the DocBook and MediaWiki format to the Sphinx/reStrcutredText framework. In our perception Sphinx offers a user-friendly and flexible way to create and manage documentation. Easy math typing and code formatting also come along. Additionally, Sphinx supports basic syntax checks, and modern documentation practices, such as versioning and integration with various output formats like HTML, PDF and ePub.

The new user’s manual is available on a dedicated page: https://docs.labplot.org. Please check it out and let us know what you think.

The manual still needs to be supplemented with new content, so we encourage you to contribute to the documentation, e.g. by fixing and adding new sections, updating images, as collaborative efforts can lead to a more comprehensive resource for everyone. Please check the Git repository dedicated to the documentation to find more details on how to help make it better.

Fast Sketch Cleanup plugin Introduction

We started this project with the intent of providing users a tool helpful in inking sketches. It is based on a research article by Simo & Sierra published in 2016, and it uses neural networks (now commonly called simply AI) to work. The tool has been developed in partnership with Intel and it’s still considered experimental, but you can already use it and see the results.

In the section below there are some real life examples of use cases and the results from the plugin. The results vary, but it can be used for extracting faint pencil sketches from photos, cleaning up lines, and comic book inking.

Regarding the model used in the tool, we trained it ourselves. All the data in the dataset is donated from people who sent their pictures to us themselves and agreed on this specific use case. We haven’t used any other data. Moreover, when you use the plugin, it processes locally on your machine, it doesn’t require any internet connection, doesn’t connect to any server, and no account is required either. Currently it works only on Windows and Linux, but we’ll work on making it available on MacOS as well.

Use cases

It averages the lines into one line and creates strong black lines, but the end result can be blurry or uneven. In many cases however it still works better than just using a Levels filter (for example in extracting the pencil sketch). it might be a good idea to use Levels filter after using the plugin to reduce the blurriness. Since the plugin works best with white canvas and grey-black lines, in case of photographed pencil sketches or very light sketch lines, it might be a good idea to use Levels also before using the plugin.

Extracting photographed pencil sketch

This is the result of the standard procedure of using Levels filter on a sketch to extract the lines (which results in a part of the image getting the shadow):

sketch_girl_original_procedure_comparison_small1843×1209 165 KB

The sketch was drawn by Tiar (link to KA profile)

This is the procedure using the plugin with SketchyModel (Levels → plugin → Levels):

sketch_girl_new_procedure_comparison_small1843×2419 267 KB

Comparison (for black lines):

sketch_girl_procedures_comparison_small1920×1260 215 KB

Another possible result is to just stop at the plugin without forcing black lines using Levels, which results in a nicer, more pencil-y look while keeping the lower part of the page still blank:

sketch_girl_after_plugin_small1536×2016 161 KB

Comic book-like inking

Picture of a man made by BeARToys

Here in the pictures above you can see the comic book style inking. The result, which is a bit blurry compared to the original, can be further enhanced by using a Sharpen filter. The dragon was sketched by David Revoy (CC-BY 4.0).

Cleaning up lines

Examples of sketches I made and the result of the plugin, showing the strong and weak points of the plugin. All of the pictures below were made using the SketchyModel.

flower_0011209×739 46.5 KB

flower_001_detail681×456 22.1 KB

portrait_man_portrait_2_comparison_2_small1305×505 139 KB

portrait_man_portrait_2_detail646×1023 26.6 KB

All of the pictures above painted by Tiar (link to KA profile)

On the pictures below, on the scales of the fish, you can see how the model discriminates lighter lines and enhances the stronger lines, making the scales more pronounced. In theory you could do that using the Levels filter, but in practice the results would be worse, because the model takes into account local strength of the line.

fish_square_sketchy_comparison_small1920×968 156 KB

Picture of the fish made by Christine Garner (link to portfolio)

How to use it in Krita

To use the Fast Sketch Cleanup plugin in Krita, do the following:

1. Prepare Krita:

   1. On Windows:

      1. Either in one package: download Krita 5.3.0-prealpha with Fast Sketch Cleanup plugin already included: https://download.kde.org/unstable/krita/5.3.0-prealpha-fast-sketch/krita-x64-5.3.0-prealpha-cdac9c31.zip

      2. Or separately:

         1. Download portable version of Krita 5.2.6 (or similar version - should still work)
         2. Download separately the Fast Sketch Cleanup plugin here: https://download.kde.org/stable/krita/FastSketchPlugin-1.0.2/FastSketchPlugin1.0.2.zip
         3. Unzip the file into krita-5.2.6/ folder (keeping the folder structure).
         4. Then go to Settings → Configure Krita → Python Plugin Manager, enable Fast Sketch Cleanup plugin, and restart Krita.

   2. On Linux:

      1. Download the appimage: https://download.kde.org/unstable/krita/5.3.0-prealpha-fast-sketch/krita-5.3.0-prealpha-cdac9c31c9-x86_64.AppImage

2. (Optional) Install NPU drivers if you have NPU on your device (practically only necessary on Linux, if you have a very new Intel CPU): Configurations for Intel® NPU with OpenVINO™ — OpenVINO™ documentation (note: you can still run the plugin on CPU or GPU, it doesn’t require NPU)

3. Run the plugin:

   1. Open or create a white canvas with grey-white strokes (note that the plugin will take the current projection of the canvas, not the current layer).
   2. Go to Tools → Fast Sketch Cleanup
   3. Select the model. Advanced Options will be automatically selected for you.
   4. Wait until it finishes processing (the dialog will close automatically then).
   5. See that it created a new layer with the result.

Advice for processing

Currently it’s better to just use the SketchyModel.xml, in most cases it works significantly better than the SmoothModel.xml.

You need to make sure the background is pretty bright, and the lines you want to keep in the result are relatively dark (either somewhat dark grey or black; light grey might result in many missed lines). It might be a good idea to use a filter like Levels beforehand.

After processing, you might want to enhance the results with either Levels filter or Sharpen filter, depending on your results.

Technology & Science behind it Unique requirements

First unique requirement was that it had to work on canvases of all sizes. That meant that the network couldn’t have any dense/fully or densely connected linear layers that are very common in most of the image processing neural networks (which require input of a specific size and will produce different results for the same pixel depending on its location), only convolutions or pooling or similar layers that were producing the same results for every pixel of the canvas, no matter the location. Fortunately, the Simo & Sierra paper published in 2016 described a network just like that.

Another challenge was that we couldn’t really use the model they created, since it wasn’t compatible with Krita’s license, and we couldn’t even really use the exact model type they described, because one of those model files would be nearly as big as Krita, and the training would take a really long time. We needed something that would work just as well if not better, but small enough that it can be added to Krita without making it twice as big. (In theory, we could do like some other companies and make the processing happen on some kind of a server, but that wasn’t what we wanted. And even if it resolved some of our issues, it would provide plenty of its own major challenges. Also, we wanted for our users to be able to use it locally without a reliance on our servers and the internet). Moreover, the model had to be reasonably fast and also modest in regards to RAM/VRAM consumption.

Moreover, we didn’t have any dataset we could use. Simo & Sierra used a dataset, where the expected images were all drawn using a constant line width and transparency, which meant that the results of the training had those qualities too. We wanted something that looked a bit more hand-drawn, with varying line-width or semi-transparent ends of the lines, so our dataset had to contain those kinds of images. Since we haven’t been aware of any datasets that would match our requirements regarding the license and the data gathering process, we asked our own community for help, here you can read the Krita Artists thread about it: https://krita-artists.org/t/call-for-donation-of-artworks-for-the-fast-line-art-project/96401 .

The link to our full dataset can be found below in the Dataset section.

Model architecture

All main layers are either convolutional or deconvolutional (at the end of the model). After every (de)convolutional layer except for the last one there is a ReLu activation layer, and after the last convolution there is a sigmoid activation layer.

Python packages used: Pillow, Numpy, PyTorch and Openvino

Numpy is a standard library for all kinds of arrays and advanced array operations and we used Pillow for reading images and converting them into numpy arrays and back. For training, we used PyTorch, while in the Krita plugin we used Openvino for inference (processing through the network).

Using NPU for inference

This table shows the result of benchmark_app, which is a tool that’s provided with Intel’s python package openvino. It tests the model in isolation on random data. As you can see, the NPU was several times faster than the CPU on the same machine.

On the other hand, introducing NPU added a challenge: the only models that can run on NPU are static models, meaning the input size is known at the time of saving the model to file. To solve this, the plugin first cuts the canvas into smaller parts of a specified size (which depends on the model file), and then proceeds to process all of them and finally stitch the results together. To avoid artifacts on the areas next to the stitching, all of the parts are cut with a little bit of a margin and the margin is later cut off.

How to train your own model

To train your own model, you’ll need some technical skills, pairs of pictures (input and the expected output) and a powerful computer. You might also need quite a lot of space on your hard drive, though you can just remove unnecessary older models if you start having issues with lack of space.

Drivers & preparation

You’ll need to install Python3 and the following packages: Pillow, openvino, numpy, torch. For quantization of the model you will also need nncf and sklearn. If I missed anything, it will complain, so just install those packages it mentions too.

If you’re on Windows, you probably have drivers for NPU and dedicated GPU. On Linux, you might need to install NPU drivers before you’ll be able to use it: https://docs.openvino.ai/2024/get-started/configurations/configurations-intel-npu.html .

Moreover if you want to use iGPU for training (which might still be significantly faster than on CPU), you’ll probably need to use something like IPEX which allows PyTorch to use an “XPU” device, which is just your iGPU. It’s not tested or recommended since I personally haven’t been able to use it because my Python version was higher than the instruction expects, but the instruction is here: https://pytorch-extension.intel.com/installation?platform=gpu&version=v2.5.10%2Bxpu .
The sanity check for the installation is as follows:
python3 -c "import torch; import intel_extension_for_pytorch as ipex; print(f'Packages versions:'); print(f'Torch version: {torch.__version__}'); print(f'IPEX version: {ipex.__version__}'); print(f'Devices:'); print(f'Torch XPU device count: {torch.xpu.device_count()}'); [print(f'[Device {i}]: {torch.xpu.get_device_properties(i)}') for i in range(torch.xpu.device_count())];"
It should show more than 0 devices with some basic properties.

If you manage to get XPU device working on your machine, you’ll still need to edit the training scripts so they’ll able to use it: https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/getting_started.html (most probably you’ll just need to add this line:
import intel_extension_for_pytorch as ipex
to the script on the very top, just underneath “import torch”, and use “xpu” as the device name when invoking the script, and it should work. But as I said, the scripts hasn’t been tested for that.

Dataset

You’ll need some pictures to be able to train your model. The pictures must be in pairs, every pair must contain a sketch (input) and a lineart picture (expected output). The better quality of the dataset, the better the results.

Before training, it’s best if you augment the data: that means the pictures are rotated, scaled up or down, and mirrored. Currently the data augmentation script also performs an inversion with the assumption that training on inverted pictures would bring the results faster (considering that black means zero means no signal, and we’d like that to be the background, so the models learn the lines, not the background around lines).

How to use the data augmentation script is explained below in the detailed instruction for the training part.

Here’s the dataset that we used (please read the license carefully if you want to use it): https://files.kde.org/krita/extras/FastSketchCleanupPluginKritaDataset.zip

Choice of model and other parameters

For quick results, use tooSmallConv; if you have more time and resources, typicalDeep might be a better idea. If you have access to a powerful GPU machine, you might try original or originalSmaller, which represent the original description of the model from the SIGGRAPH article by Simo-Sierra 2016, and a smaller version of it.

Use adadelta as the optimizer.

You can use either blackWhite or mse as the loss function; mse is classic, but blackWhite might lead to faster results since it lowers the relative error on the fully white or fully black areas (based on the expected output picture).

Training

1. Clone the repository at https://invent.kde.org/tymond/fast-line-art (at 33869b6)
   git clone https://invent.kde.org/tymond/fast-line-art.git

2. Then, prepare the folder:

   • Create a new folder for the training.
   • In the folder, run:
     python3 [repository folder]/spawnExperiment.py --path [path to new folder, either relative or absolute] --note "[your personal note about the experiment]"

3. Prepare data:

   • If you have existing augmented dataset, put it all in data/training/ and data/verify/, keeping in mind that paired pictures in ink/ and sketch/ subfolders must have the exact same names (for example if you have sketch.png and ink.png as data, you need to put one in sketch/ as picture.png and another in ink/ as picture.png to be paired).
   • If you don't have existing augmented dataset:
     1. Put all your raw data in data/raw/, keeping in mind that paired pictures should have the exact same names with added prefix either ink_ or sketch_ (for example if you have picture_1.png being the sketch picture and picture_2.png being the ink picture, you need to name them sketch_picture.png and ink_picture.png respectively.)
     2. Run the data preparer script:
        python3 [repository folder]/dataPreparer.py -t taskfile.yml
        That will augment the data in the raw directory in order for the training to be more successful.

4. Edit the taskfile.yml file to your liking. The most important parts you want to change are:

   • model type - code name for the model type, use tinyTinier, tooSmallConv, typicalDeep or tinyNarrowerShallow
   • optimizer - type of optimizer, use adadelta or sgd
   • learning rate - learning rate for sgd if in use
   • loss function - code name for loss function, use mse for mean squared error or blackWhite for a custom loss function based on mse, but a bit smaller for pixels where the target image pixel value is close to 0.5

5. Run the training code:
   python3 [repository folder]/train.py -t taskfile.yml -d "cpu"

   On Linux, if you want it to run in a background, add “&” at the end. If it runs in a foreground, you can pause the training just by pressing ctrl+C, and if it runs in a background, find a process id (using either “jobs -l” command or “ps aux | grep train.py” command, the first number would be the process id) and kill it using “kill [process id]” command. Your results will still be in the folder, and you’ll be able to resume the training using the same command.

6. Convert the model to an openvino model:
   python3 [repository folder]/modelConverter.py -s [size of the input, recommended 256] -t [input model name, from pytorch] -o [openvino model name, must end with .xml]

7. Place both the .xml and .bin model files in your Krita resource folder (inside pykrita/fast_sketch_cleanup subfolder) alongside other models to use them in the plugin.

Merging dictionaries in Python is usually as simple as a single pipe character.

Table of contents

1. Merging dictionaries with | in Python
2. Merging dictionaries with the update method
3. Merging dictionaries with **
4. The difference between | and **
5. Handling duplicate keys when merging
6. The | operator performs a "union" between dictionaries
7. Join dictionaries with the | operator

Merging dictionaries with | in Python

First let's talk about the simplest way to merge dictionaries, which will usually be all that you need.

Here are two dictionaries:

>>> context = {"language": "en", "timezone": "UTC"} >>> more_context = {"title": "Home", "breadcrumbs": ["Home"]}

We'd like to make a new third dictionary that combines these two. This new dictionary should contain all the key-value pairs from both dictionaries.

The easiest way to do this is to use the pipe (|) operator:

>>> new_context = context | more_context

This made a new dictionary that contains all the items from both of the two initial dictionaries:

>>> new_context {'language': 'en', 'timezone': 'UTC', 'title': 'Home', 'breadcrumbs': ['Home']}

Using the | operator is basically the same as making a new empty dictionary, and then looping over all the items in the first dictionary and then all the items in the second dictionary, and adding all of them to the new dictionary:

>>> new_context = {} >>> for key, value in context.items(): ... new_context[key] = value ... >>> for key, value in more_context.items(): ... new_context[key] = value ... >>> new_context {'language': 'en', 'timezone': 'UTC', 'title': 'Home', 'breadcrumbs': ['Home']} Merging dictionaries with the update method

What if we wanted to …

Read the full article: https://www.pythonmorsels.com/merging-dictionaries/

The FSF SysOps team has been hard at work over the past six months on quite a few major projects.

The FSF SysOps team has been hard at work over the past six months on quite a few major projects.

Implementing LAPACK routines for numerical computation in web applications