Feeds

Everyone wants to improve their lives over time, including those areas that make them feel like they are on top of the latest trends and staying caught up on new developments. The same happens with your software too, why fall behind from new updates? Updating your websites, software, and other essential operational business ongoings is important. 

But sometimes what happens is your team doesn’t support this not-so-big task. They believe upgrading the website is pointless because it will look the same. It becomes difficult to convince your team but we got these practical tips so your team will not deny migrating from Drupal 7 to 10. Let’s see how you can convince your team members to migrate to Drupal 10 from Drupal 7. 

IMAP and OAuth2 Integrations with Microsoft 365 2024-05-13, by Dariusz Suchojad

Overview

This is the first in a series of articles about automation of and integrations with Microsoft 365 cloud products using Python and Zato.

We start off with IMAP automation by showing how to create a scheduled Python service that periodically pulls latest emails from Outlook using OAuth2-based connections.

IMAP and OAuth2

Microsoft 365 requires for all IMAP connections to use OAuth2. This can be challenging to configure in server-side automation and orchestration processes so Zato offers an easy way that lets you read and send emails without a need for getting into low-level OAuth2 details.

Consider a common orchestration scenario - a business partner sends automated emails with attachments that need to be parsed, some information needs to be extracted and processed accordingly.

Before OAuth2, an automation process would receive from Azure administrators a dedicated IMAP account with a username and password.

Now, however, in addition to creating an IMAP account, administrators will need to create and configure a few more resources that the orchestration service will use. Note that the password to the IMAP account will never be used.

Administrators need to:

• Register an Azure client app representing your service that uses IMAP
• Grant this app a couple of Microsoft Graph application permissions:
• Mail.ReadWrite
• Mail.Send

Next, administrators need to give you a few pieces of information about the app:

• Application (client) ID
• Tenant (directory) ID
• Client secret

Additionally, you still need to receive the IMAP username (an e-mail address). It is just that you do not need its corresponding password.

In Dashboard

The first step is to create a new connection in your Zato Dashboard - this will establish an OAuth2-using connection that Zato will manage and your Python code will not have to do anything else, all the underlying OAuth2 tokens will keep refreshing as needed, the platform will take care of everything.

Having received the configuration details from Azure administrators, you can open your Zato Dashboard and navigate to IMAP connections:

Fill out the form as below, choosing "Microsoft 365" as the server type. The other type, "Generic IMAP" is used for the classical case of IMAP with a username and password:

Change the secret and click Ping to confirm that the connection is configured correctly:

In Python

Use the code below to receive emails. Note that it merely needs to refer to a connection definition by its name and there is no need for any usage of OAuth2 here:

# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class MyService(Service): def handle(self): # Connect to a Microsoft 365 IMAP connection by its name .. conn = self.email.imap.get('My Automation').conn # .. get all messages matching filter criteria ("unread" by default).. for msg_id, msg in conn.get(): # .. and access each of them. self.logger.info(msg.data)

This is everything that is needed for integrations with IMAP using Microsoft 365 although we can still go further. For instance, to create a scheduled job to periodically invoke the service, go to the Scheduler job in Dashboard:

In this case, we decide to have a job that runs once per hour:

As expected, clicking OK will suffice for the job to start in background. It is as simple as that.

Next steps

• Read more about using Python in API integrations
• Start the tutorial which will guide you how to design and build Python API services for automation and integrations

More blog posts➤

Dear Readers,

Much has happened in a week. The biggest open-source event in North America, DrupalCon Portland 2024 has come to a close. A convergence of technology and community weaving tales of innovations gathering Drupalers worldwide has successfully concluded. Among the many highlights of the DrupalCon Portland 2024, some stole the limelight. The introduction of the Drupal Starshot initiative definitely tops the list!

What is Drupal Starshot?

It is a long overdue strategic overhaul aimed at improving Drupal's functionality and usability.  This project introduces a new "Drupal CMS" that seamlessly integrates core modules with popular community modules through a Recipes API, facilitating easier setups. Additionally, the "Experience Builder" tool, built with React, is designed to enhance page-building capabilities. Dries Buytaert propelled the Drupal Starshot rocket during DrupalCon with the help of John. F. Kennedy's "moonshot" speech. 

"We choose to go to the moon, not because that will be easy, but because it will be hard" - JFK

From learning the Soviet Union had put a man in space, the space war began. It was followed by eight years of failed attempts from the part of the US before they finally succeeded in launching Apollo 11. Dries had chosen the perfect analogy to fuel Drupal Starshot because the competition in the CMS landscape is much similar to that of the space war and failure is not an option. He reiterates that if Drupal needs to stay relevant, it needs to have its moonshot moment and Drupal Starshot is the moonshot Drupal needed.

With Drupal 8, it was more or less like Drupal had abandoned small and mid-websites for enterprise sites, but this has adversely affected its market. Additionally, Drupal's sub-standard page-building capabilities have been finally identified as the primary reason for Drupal's shrinking adoption. Now, there is only one way to go about it, make the fundamental changes urgently enhancing Drupal core with additional components tailored to everyday users of the interface.

The path that has been declared by Dries is easier said than done. But this is a race Drupal can't afford to lose because Drupal has a strong legacy to safeguard. It was born to be open and has been the defender of the open web for the past 23 years.

"Some people feel like we're falling behind, and no one wants our competitors to win because they aim to colonize a web that's less open, more closed, less creative, maybe more expensive, less accessible, and more proprietary." - Dries Buytaert

The web is moving forward it is up to Drupal and its community to decide whether they want to be a part of it. The need for change is evident and Drupal Starshot is the way to a rejuvenation. So join the Drupal Starshot initiative and make your pledge today. A comprehensive and detailed account of Driesnote that introduced Starshot can be found here.

With that let's move on to the important stories of last week.

DrupalCon Portland came up with many insights and announcements. Elma John, our sub-editor has gathered the key insights from DrupalCon Portland 2024. Drupalers are excited about DrupalCon North America  2025 declared to be held in Atlanta from March 24 to March 27. The dates for DrupalCon Singapore have been confirmed: November 18th to 21st, 2024. The event is to be held at the prestigious Pan Pacific Parkroyal Collection in Marina Bay.

Michael Anello, co-owner of DrupalEasy, has received the prestigious 2024 Aaron Winborn Award at DrupalCon Portland 2024. The award, presented by the Drupal Community Working Group, honors individuals who have shown exceptional commitment and contribution to the Drupal community.

The Women in Drupal Luncheon headed by the female stalwarts of Four Kitchens was a huge success. The event titled, "The Lonely Only: Women in Tech and Navigating a Male-Dominated Landscape" featured Sebastianna Skalisky, Laura Johnson, Jenna Harris, and Shanice Ortiz in a compelling discussion about the dimunitive presence of women in the historically made-dominated tech landscape. Most of these women have been the only women in their department and even at their company. For detailed insights into this session, please read my article, "Women in Drupal Luncheon at DrupalCon Portland 2024: A Convergence for Change."

A supportive initiative bloomed at DrupalCon Portland this year, introducing green sunflower lanyards to support individuals with hidden disabilities. These lanyards marked a significant step towards making Drupal events more accessible and comfortable for everyone. 

Jesus Manuel Olivas, CEO of Octahedroid, unveiled a new Headless Drupal content structuring approach improving the balance between developers, site builders, content editors, and marketers at DrupalCon. The system called Drupal as a Composable CMS, introduces three key modules: the Composable Module for better content adaptability, a Decoupled Preview Iframe for real-time content previews, and a Visual Editor for streamlined editing.

In other news, The DropTimes has published a new study revealing a global trend of governments favoring free and open-source software (FOSS) for their official websites. Team TDT analyzed the websites of 194 countries to understand the extent of this trend and uncover which content management systems (CMSs) are preferred for official government communication. Interestingly, Drupal emerged as a top choice when analyzing the top 100 country websites based on popularity (Tranco rank), Additionally, WordPress remains the most widely used FOSS option globally.

Kazima Abbas, a sub-editor with TDT had the opportunity to sit with Tim Hestenes Lehnen, Chief Technology Officer at the Drupal Association and Author of Young Adult Fiction, to talk about his latest novel, 'Fog & Fireflies'. 

DrupalCon Barcelona 2024 Calls for Artists to Enter Drawing Contest. Participants are encouraged to create an original drawing of Mercè, the celebrated figure of Barcelona's annual festival, in any artistic form they prefer. The winning design will earn a spot on the official event stickers, giving artists a unique platform to showcase their work. 

LagoonCon Portland 2024 offered a night of tech insights and connections. The conference featured three sessions dealing with Lagoon, Open Web Alliance, and Large Language Models. The free event also had a Lagoon round table discussion. Read more here. 

Events in the Drupal community are always a matter of celebration. It brings together Drupalers from various walks of life and forms a support system with Drupal at the core. Here is a list of Drupal events you can attend this week.

The 2024 Drupal Developer Survey, led by Michael Richardson, provided a comprehensive overview of the global Drupal community. With 648 developers from 65 countries, including significant contributions from the United States, France, and India, the survey showcased a mature developer base, with 76% aged between 30 and 49.

James Shields has launched the 2024 Drupal Advent Calendar, inviting contributions from the Drupal community. This year's theme, "People of Drupal," aims to highlight individuals who have significantly impacted the Drupal ecosystem. In another update, Promote Drupal committee member Emma Horrell has launched a comprehensive initiative to demystify Drupal's specialized language, known as "Drupalisms," to make the platform more accessible, especially to newcomers. The new initiative is named "De-jargoning Drupal".

1xINTERNET has announced its support for Drupal Starshot, promising to contribute with its expertise in pre-configured Drupal applications. Their existing CMS, Try Drupal, showcases their capabilities and will integrate with Starshot to further enrich the Drupal community and promote widespread adoption.

We acknowledge that there are more stories to share. However, due to selection constraints, we must pause further exploration for now.

To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.

Thank you,
Sincerely
Alka Elizabeth
Sub-editor, The DropTimes.

A new revision of digiKam Recipes is available for your reading pleasure. The new version covers the auto tagging feature introduced in digiKam 8.3 and explains how to run digiKam in a container. If you bought the book through Gumroad, you’ll find the new revision in the Library section. The book purchased through Google Play should be updated automatically to the latest version. If you have problems getting the latest revision of the book, contact the author at dmpop@cameracode.

Thoughts on Drupal's new Starshot Initiative mherchel Sun, 05/12/2024 - 18:16

The Kubuntu Team are thrilled to announce significant updates to KubuQA, our streamlined ISO testing tool that has now expanded its capabilities beyond Kubuntu to support Ubuntu and all its other flavors. With these enhancements, KubuQA becomes a versatile resource that ensures a smoother, more intuitive testing process for upcoming releases, including the 24.04 Noble Numbat and the 24.10 Oracular Oriole.

What is KubuQA?

KubuQA is a specialized tool developed by the Kubuntu Team to simplify the process of ISO testing. Utilizing the power of Kdialog for user-friendly graphical interfaces and VirtualBox for creating and managing virtual environments, KubuQA allows testers to efficiently evaluate ISO images. Its design focuses on accessibility, making it easy for testers of all skill levels to participate in the development process by providing clear, guided steps for testing ISOs.

New Features and Extensions

The latest update to KubuQA marks a significant expansion in its utility:

• Broader Coverage: Initially tailored for Kubuntu, KubuQA now supports testing ISO images for Ubuntu and all other Ubuntu flavors. This broadened coverage ensures that any Ubuntu-based community can benefit from the robust testing framework that KubuQA offers.
• Support for Latest Releases: KubuQA has been updated to include support for the newest Ubuntu release cycles, including the 24.04 Noble Numbat and the upcoming 24.10 Oracular Oriole. This ensures that communities can start testing early and often, leading to more stable and polished releases.
• Enhanced User Experience: With improvements to the Kdialog interactions, testers will find the interface more intuitive and responsive, which enhances the overall testing experience.

Call to Action for Ubuntu Flavor Leads

The Kubuntu Team is keen to collaborate closely with leaders and testers from all Ubuntu flavors to adopt and adapt KubuQA for their testing needs. We believe that by sharing this tool, we can foster a stronger, more cohesive testing community across the Ubuntu ecosystem.

We encourage flavor leads to try out KubuQA, integrate it into their testing processes, and share feedback with us. This collaboration will not only improve the tool but also ensure that all Ubuntu flavors can achieve higher quality and stability in their releases.

Getting Involved

For those interested in getting involved with ISO testing using KubuQA:

• Download the Tool: You can find KubuQA on the Kubuntu Team Github.
• Join the Community: Engage with the Kubuntu community for support and to connect with other testers. Your contributions and feedback are invaluable to the continuous improvement of KubuQA.

Conclusion

The enhancements to KubuQA signify our commitment to improving the quality and reliability of Ubuntu and its derivatives. By extending its coverage and simplifying the testing process, we aim to empower more contributors to participate in the development cycle. Whether you’re a seasoned tester or new to the community, your efforts are crucial to the success of Ubuntu.

We look forward to seeing how different communities will utilise KubuQA to enhance their testing practices. And by the way, have you thought about becoming a member of the Kubuntu Community? Join us today to make a difference in the world of open-source software!

I've been resisting the Wikipedia ads about “we don't run ads, give us money“ for over a decade now (mostly since WMF already has tons of cash and use very little on it to actually improve Wikipedia), and now they are jumping on the AI/LLM hype. It does not help.

(Of course you should pick your charities yourself. I've donated to Signal Foundation in the past even though I think they could run things somewhat cheaper if they didn't insist on all-cloud, and archive.org is basically, for better or for worse, the collective memory of the Internet by now, at least unless they get bankrupted by the ongoing lawsuit from some overly silly book lending. Not to mention that in a day and age where there's a certain Eurovision-participating country and another banned-from-Eurovision country both causing tons of civilian suffering and casualities these days, perhaps there are non-tech charities that are also important.)

I have not been able to walk since February 18, 2023.

When people ask me how I'm doing, this is the first thing that comes to mind. "Well, you know, the usual, but also I still can't walk," I think to myself.

If I dream at night, I often see myself walking or running. In conversation, if I talk about going somewhere, I'll imagine walking there. Even though it's been over a year, I remember walking to the bus, riding to see my friends, going out for brunch, cooking community dinners.

But these days, I can't manage going anywhere except by car, and I can't do the driving, and I can't dis/assemble and load my chair. When I'm resting in bed and follow a guided meditation, I might be asked to imagine walking up a staircase, step by step. Sometimes, I do. Other times, I imagine taking a little elevator in my chair, or wheeling up ramps.

I feel like there is little I can say that can express the extent of what this illness has taken from me, but it's worth trying. To an able-bodied person, seeing me in a power wheelchair is usually "enough." One of my acquaintances cried when they last saw me in person. But frankly, I love my wheelchair. I am not "wheelchair-bound"—I am bed-bound, and the wheelchair gets me out of bed. My chair hasn't taken anything from me.

***

In October of 2022, I was diagnosed with myalgic encephalomyelitis.

Scientists and doctors don't really know what myalgic encephalomyelitis (ME) is. Diseases like it have been described for over 200 years.1 It primarily affects women between the ages of 10-39, and the primary symptom is "post-exertional malaise" or PEM: debilitating, disproportionate fatigue following activity, often delayed by 24-72 hours and not relieved by sleep. That fatigue has earned the illness the misleading name of "Chronic Fatigue Syndrome" or CFS, as though we're all just very tired all the time. But tired people respond to exercise positively. People with ME/CFS do not.2

Given the dearth of research and complete lack of on-label treatments, you may think this illness is at least rare, but it is actually quite common: in the United States, an estimated 836k-2.5m people3 have ME/CFS. It is frequently misdiagnosed, and it is estimated that as many as 90% of cases are missed,4 due to mild or moderate symptoms that mimic other diseases. Furthermore, over half of Long COVID cases likely meet the diagnostic criteria for ME,5 so these numbers have increased greatly in recent years. That is, ME is at least as common as rheumatoid arthritis,6 another delightful illness I have. But while any doctor knows what rheumatoid arthritis is, not enough7 have heard of "myalgic encephalomylitis."

Despite a high frequency and disease burden, post-viral associated conditions (PASCs) such as ME have been neglected for medical funding for decades.8 Indeed, many people, including medical care workers, find it hard to believe that after the acute phase of illness, severe symptoms can persist. PASCs such as ME and Long COVID defy the typical narrative around common illnesses. I was always told that if I got sick, I should expect to rest for a bit, maybe take some medications, and a week or two later, I'd get better, right? But I never got better.

These are complex, multi-system diseases that do not neatly fit into the Western medical system's specializations. I have seen nearly every specialty because ME/CFS affects nearly every system of the body: cardiology, nephrology, pulmonology, neurology, opthalmology, and, many, many more. You'd think they'd hand out frequent flyer cards, or a medical passport with fun stamps, but nope. Just hundreds of pages of medical records. And when I don't fit neatly into one particular specialist's box, then I'm sent back to my primary care doctor to regroup while we try to troubleshoot my latest concerning symptoms. "Sorry, can't help you. Not my department."

With little available medical expertise, a lot of my disease management has been self-directed in partnership with primary care. I've read hundreds of articles, papers, publications, CME material normally reserved for doctors. It's truly out of necessity, and I'm certain I would be much worse off if I lacked the skills and connections to do this; there are so few ME/CFS experts in the US that there isn't one in my state or any adjacent state.9 So I've done a lot of my own work, much of it while barely being able to read. (A text-to-speech service is a real lifesaver.) To facilitate managing my illness, I've built a mental model of how my particular flavour of ME/CFS works based on the available research I've been able to read and how I respond to treatments. Here is my best attempt to explain it:

• After a severe (non-COVID) infection, an ongoing interaction between my immune system and my metabolism have stopped my body from being able to do aerobic respiration.10
• I don't know why or how, but my mitochondria don't work properly anymore.11
• This means that if I use too much energy, my body isn't able to make enough energy to catch up, and I have severe symptoms over the next few days as my body tries to manage the consequences.
• Those symptoms aren't limited to fatigue: I've developed flu-like symptoms and even fevers, limbs so heavy they felt paralyzed, tachycardia in response to even the slightest activity.

The best way I have learned to manage this is to prevent myself from doing activities where I will exceed that aerobic threshold by wearing a heartrate monitor,12 but the amount of activity that permits in my current state of health is laughably restrictive. Most days I'm unable to spend more than one to two hours out of bed.

Over time, this has meant worsening from a persistent feeling of tiredness all the time and difficulty commuting into an office or sitting at a desk, to being unable to sit at a desk for an entire workday even while working from home and avoiding physically intense chores or exercise without really understanding why, to being unable to leave my apartment for days at a time, and finally, being unable to stand for more than a minute or two or walk.

But it's not merely that I can't walk. Many folks in wheelchairs are able to live excellent lives with adaptive technology. The problem is that I am so fatigued, any activity can destroy my remaining quality of life. In my worst moments, I've been unable to read, move my arms or legs, or speak aloud. Every single one of my limbs burned, as though I had caught fire. Food sat in my stomach for hours, undigested, while my stomach seemingly lacked the energy to do its job. I currently rely on family and friends for full-time caretaking, plus a paid home health aide, as I am unable to prep meals, shower, or leave the house independently. This assistance has helped me slowly improve from my poorest levels of function.

While I am doing better than I was at my worst, I've had to give up essentially all of my hobbies with physical components. These include singing, cooking, baking, taking care of my houseplants, cross-stitching, painting, and so on. Doing any of these result in post-exertional malaise so I've had to stop; this reduction of activity to prevent worsening the illness is referred to as "pacing." I've also had to cut back essentially all of my volunteering and work in open source; I am only cleared by my doctor to work 15h/wk (from bed) as of writing.

***

CW: severe illness, death, and suicide (skip this section)

The difficulty of living with a chronic illness is that there's no light at the end of the tunnel. Some diseases have a clear treatment path: you take the medications, you complete the procedures, you hit all the milestones, and then you're done, perhaps with some long-term maintenance work. But with ME, there isn't really an end in sight. The median duration of illness reported in one 1997 study was over 6 years, with some patients reporting 20 years of symptoms.13 While a small number of patients spontaneously recover, and many improve, the vast majority of patients are unable to regain their baseline function.14

My greatest fear since losing the ability to walk is getting worse still. Because, while I already require assistance with nearly every activity of daily living, there is still room for decline. The prognosis for extremely ill patients is dismal, and many require feeding tubes and daily nursing care. This may lead to life-threatening malnutrition;15 a number of these extremely severe patients have died, either due to medical neglect or suicide.16 Extremely severe patients cannot tolerate light, sound, touch, or cognitive exertion,17 and often spend most of their time lying flat in a darkened room with ear muffs or an eye mask.18

This is all to say, my prognosis is not great.

But while I recognize that the odds aren't exactly in my favour, I am also damn stubborn. (A friend once cheerfully described me as "stubbornly optimistic!") I only get one shot at life, and I do not want to spend the entirety of it barely able to perceive what's going on around me. So while my prognosis is uncertain, there's lots of evidence that I can improve somewhat,19 and there's also lots of evidence that I can live 20+ years with this disease. It's a bitter pill to swallow, but it also means I might have the gift of time—something that not all my friends with severe complex illnesses have had.

I feel like I owe it to myself to do the best I can to improve; to try to help others in a similar situation; and to enjoy the time that I have. I already feel like my life has been moving in slow motion for the past 4 years—there's no need to add more suffering. Finding joy, as much as I can, every day, is essential to keep up my strength for this marathon. Even if it takes 20 years to find a cure, I am convinced that the standard of care is going to improve. All the research and advocacy that's been happening over the past decade is plenty to feel hopeful about.20 Hope is a discipline,21 and I try to remind myself of this on the hardest days.

***

I'm not entirely sure why I decided to write this. Certainly, today is International ME/CFS Awareness Day, and I'm hoping this post will raise awareness in spaces that aren't often thinking about chronic illnesses. But I think there is also a part of me that wants to share, reach out in some way to the people I've lost contact with while I've been treading water, managing the day to day of my illness. I experience this profound sense of loss, especially when I think back to the life I had before. Everyone hits limitations in what they can do and accomplish, but there is so little I can do with the time and energy that I have. And yet, I understand even this precious little could still be less. So I pace myself.

Perhaps I can inspire you to take action on behalf of those of us too fatigued to do the advocacy we need and deserve. Should you donate to a charity or advocacy organization supporting ME/CFS research? In the US, there are many excellent organizations, such as ME Action, the Open Medicine Foundation, SolveME, the Bateman Horne Center, and the Workwell Foundation. I am also happy to match any donations through the end of May 2024 if you send me your receipts. But charitable giving only goes so far, and I think this problem deserves the backing of more powerful organizations.

Proportionate government funding and support is desperately needed. It's critical for us to push governments22 to provide the funding required for research that will make an impact on patients' lives now. Many organizers are running campaigns around the world, advocating for this investment. There is a natural partnership between ME advocacy and Long COVID advocacy, for example, and we have an opportunity to make a great difference to many people by pushing for research and resources inclusive of all PASCs. Some examples I'm aware of include:

• [US] A Long COVID "moonshot" effort, drafted in April 2024 by Senator Sanders. Contact your representatives and encourage them to support it. Ask them to ensure the effort includes support for ME/CFS and other related conditions and PASCS.
• [Canada] National ME/FM Action recommended participation in public budget consultation processes and made a submission. Though the 2024 consultation is closed, you can still contact your MP to improve awareness and advocate for budgetary support. It may also be worth contacting your MLA to advocate for better PASC care in provincial clinics.
• [UK] Action for ME is advocating for a cross-Government Delivery Plan for ME/CFS care in England. Contact your MP and encourage them to support it.
• [Other] MEAction also regularly hosts government advocacy campaigns in the US, UK, and Scotland.

But outside of collective organizing, there are a lot of sick individuals out there that need help, too. Please, don't forget about us. We need you to visit us, care for us, be our confidantes, show up as friends. There are a lot of people who are very sick out here and need your care.

I'm one of them.

After the unfortunate and somewhat surprising shutdown of the Open Collective Foundation (OCF), htop and Performance Co-Pilot (PCP) have migrated to Hack Club.

Initially founded to improve STEM education, support high school computer science clubs and firmly founded in the hacker culture, Hack Club have created a US IRS approved 501(c)(3) charity that provides what Open Collective did/does1 and more at a flat 7% fee of the project income. Nathan Scott organized these moves with Paul Spitler. Many thanks!

We considered other options for the projects, e.g. Gentoo has moved to Software in the Public Interest (SPI) and I know SPI quite well as they were created initially to host Debian. But PCP moved from SPI to OCF in 2021. Open Collective has a European branch that seems independent of the dissolved US foundation. But all-in-all Hack Club seemed the best fit.

You can find the new fiscal sponsorship and donation landing pages at:

htophttps://hcb.hackclub.com/htop/https://hcb.hackclub.com/donations/start/htop PCPhttps://hcb.hackclub.com/pcp/https://hcb.hackclub.com/donations/start/pcp

1. Open Collective as in the fancy "manage your project donations and reimbursements" website still continues to run but the foundation of the same name that provided the actual fiscal sponsorship (i.e. managing the funds) got dissolved. It's ... complicated. ↩

Companion guide to the Python packaging tutorial.

This is not an overview of packaging, nor a history of the tooling. The intended audience is an author of a simple package who merely wants to publish it on the package index, without being forced to make uninformed choices.

Build backends

The crux of the poor user experience is choosing a build backend. The reader at this stage does not know what a “build backend” is, and moreover does not care.

The 4 backends in the tutorial are described here in their presented order. An example snippet of a pyproject.toml file is included, mostly assuming defaults, with a couple common options:

• dynamic version
• package data for type information

hatchling

- 83 kB with 4 dependencies

requires = ["hatchling"] build-backend = "hatchling.build" [tool.hatch.build.targets.sdist] include = ["<package>/*"] [tool.hatch.version] path = "<package>/__init__.py"

Part of - not to be confused with - the project manager Hatch.

The source distribution section is included because by default hatchling ostensibly includes all files that are not ignored. However, it only abides by the root .gitignore. It will include virtual environments, if not named .venv. For a project that advocates sensible defaults, this is surprising behavior and a security flaw. Even if the issue is fixed, it will presumably include untracked files and clearly omissible directories such as .github.

setuptools

- 894 kB with 0 dependencies

[build-system] requires = ["setuptools>=61.0"] build-backend = "setuptools.build_meta" [tool.setuptools] packages = ["<package>"] [tool.setuptools.dynamic] version = {attr = "<package>.__version__"} [tool.setuptools.package-data] <package> = ["py.typed"]

The original build tool, and previously the de facto standard. It is no longer commonly included in Python distributions, so they are all on equal footing with respect to needing installation.

Setuptools requires explicitly specifying the package, as well as any package data. It also includes legacy “.egg” and “setup.cfg” files, which a modern user will not be familiar with.

flit-core

- 63 kB with 0 dependencies

[build-system] requires = ["flit-core>=3.4"] build-backend = "flit_core.buildapi"

Part of the Flit tool for publishing packages.

Flit automatically supports dynamic versions (and descriptions), and includes the source directory with data files in the source distribution.

pdm-backend

- 101 kB with 0 dependencies

requires = ["pdm-backend"] build-backend = "pdm.backend" [tool.pdm] version = {source = "file", path = "<package>/__init__.py"}

Part of - not to be confused with - the project manager PDM.

PDM automatically includes the source and test directories, with data files, in the source distribution.

Evaluations Popularity and endorsements

The popularity of Setuptools should be discounted because of its history. The popularity of Hatchling and PDM-backend is clearly influenced by their respective parent projects. PDM has significantly less downloads than the others, but they are all popular enough to expect longevity.

Setuptools, Hatch, and Flit are all under the packaging authority umbrella, though as the previously cited article points out, PyPA affiliation does not indicate much.

The tutorial “defaults” to Hatchling, which presumably is not intended as an endorsement, but will no doubt be interpreted as such.

Size and dependencies

Setuptools is by far the largest; no surprise since it is much more than a build backend. Hatchling is the only one with dependencies, but the 3 modern ones seem appropriately lightweight.

File selection

Wheels have a standard layout, but source distributions do not. Whether sdist should include docs and tests is a matter of debate.

There was a time when open source software meant “distributed with an open source license”, so the source distribution was the primary way to acquire the code. This all seems anachronistic in the age of distributed version control and public collaboration. Not to mention wheels are zip files which have the source code.

One piece of advice is that the sdist should be buildable. Generated portable files could be included, thereby not needing the tools that generate them. But for a simple (read pure) Python project, that is not particularly relevant.

There is another issue with backends creating different artifacts when using their own build commands. This rundown only evaluated python -m build.

Metadata

The modern 3 implicitly support data files. All 4 support dynamic versioning in some manner. Then again, maybe the __version__ attribute is no longer the leading convention among the 7 options for single-sourcing the version. Now that importlib.metadata is no longer provisional, is that preferred?

Recommendations

It would be disingenuous to not end with recommendations, since the refusal to - in a document titled tool recommendations - is the problem. The PyPA endorses pip, build, and twine as standard tools, even though there are alternatives.

Author’s disclosures: I am a long-time Python developer of several packages, and a couple with extension modules. I use no project management tools, and am not affiliated with any of these projects.

1. flit-core - No criticisms. The dynamic version and description feature are a plus; not having any flit-specific sections feels like less coupling.
2. pdm-backend - No criticisms. A natural choice if one wants tests in the source distribution.
3. hatchling - The file selection issue is significant. Users need a warning that they should include an sdist section and check their tarballs. Many are going to have unnecessarily large distributions, and someone with a local secrets directory - whether ignored or untracked - is going to have a seriously bad day.
4. setuptools - Requires the most customization and is perpetually handicapped by backwards compatibility. The only advantage setuptools had was being already installed. It may be time to disavow it for new projects without extension modules.

My projects currently use setuptools for purely historically reasons. For new projects, I would likely use flit-core. I may switch-over existing projects, though there is really no incentive to.

Unless a standard emerges, of course.

Epilogue

A meta case could be made for Flit(-core) as well: that its limited scope and independence from a project manager is itself an asset. Whereas choosing Hatch(ling) or PDM(-backend) feels like picking a side. Flit can position itself as the minimalist choice for those who resent having to choose.

Contributing to Debian is part of Freexian’s mission. This article covers the latest achievements of Freexian and their collaborators. All of this is made possible by organizations subscribing to our Long Term Support contracts and consulting services.

P.S. We’ve completed over a year of writing these blogs. If you have any suggestions on how to make them better or what you’d like us to cover, or any other opinions/reviews you might have, et al, please let us know by dropping an email to us. We’d be happy to hear your thoughts. :)

Salsa CI updates & GSoC candidacy, by Santiago Ruano Rincon

In the context of Google Summer of Code (GSoC), Santiago continued the mentoring work, following the applications of three of the candidates. This work started in March, but Aquila Macedo, Ahmed Siam and Piyush Raj continued in April to propose and review MRs. For example, Update CI pipeline to utilize specific blhc image per release and Remove references to buster-backports by Aquila, or the reviews the candidates made to Document the structure of the different components of the pipeline (see below).

Unfortunately, the Salsa CI project didn’t get any slot from the GSoC program in the end.

Along with the Salsa CI related work, Santiago improved the documentation of Salsa CI, to make it easier for newcomers (as the GSoC candidates) or people willing to fork the project to understand its internals. Documentation is an aspect where a lot of improvements can be made.

OpenSSH option review, by Colin Watson

In light of last month’s xz-utils backdoor, Colin did an extensive review of some of the choices in Debian’s OpenSSH packaging. Some work on this has already been done (removing uses of libsystemd and reducing tcp-wrappers linkage); the next step is likely to be to start work on the plan to split out GSS-API key exchange again.

Miscellaneous contributions

• Utkarsh Gupta started to put together and kickstart the bursary team ahead of DebConf 24, to be held in Busan, South Korea.
• Utkarsh Gupta reviewed some MRs and docs for the bursary team for the DC24 website.
• Helmut Grohne sent patches for 19 cross build failures and submitted a gcc patch removing LIMITS_H_TEST upstream.
• Helmut sent 8 bug reports with 3 patches related to the /usr-move.
• Helmut diagnosed why /dev/stdout is not accessible in sbuild --mode=unshare.
• Helmut diagnosed the time64-induced glibc FTBFS.
• Helmut sent patches for fixing initramfs triggers on firmware removal.
• Thorsten Alteholz uploaded foo2zjs and fixed two bugs, one related to /usr-merge. Likewise the upload of cups-filters (from the 1.x branch) fixed three bugs. In order to fix an RC bug in cpdb-backends-cups, which was updated to the 2.x branch, the new package libcupsfilters has been introduced. Last but not least an upload of hplip fixed one RC bug and an upload of gutenprint fixed two of them. All of these RC bugs were more or less related to the time_t transition.
• Santiago continued to work in the DebConf organization tasks, including some for the DebConf 24 Content Team, and looking to build a local community for DebConf 25.
• Stefano Rivera made a couple of uploads of dh-python to Debian, and a few other general package update uploads.
• Stefano did some winding up of DebConf 23 finances, including closing bursary claims and recording the amounts spent on travel bursaries.
• Stefano opened DebConf 24 registration, which always requires some last-minute work on the website.
• Colin released man-db 2.12.1.
• Colin fixed a regression in groff’s PDF output.
• In the Python team, Colin fixed build/autopkgtest failures in seven packages, and updated ten packages to new upstream versions.

In this rather lengthy post I talk a bit about the current issues with icons for the KDE applications I work on or use.

Let’s start with looking at what I mean with KDE applications and what the current state is, up to KDE Frameworks 6.2 and current KDE Gear 24.02. Then let’s see what will be improved in future releases.

What do I mean with ‘KDE Applications’ #

If I speak about ‘KDE Applications’ here I talk about applications like Kate, Dolphin, Okular and others like that.

This means applications developed with Qt and KDE Frameworks that integrate well with the KDE Plasma desktop but are not restricted to it.

Many of this applications not just aim to work well on Linux & BSD or other open source operating systems but are ported and working well on the rather different Windows and macOS desktop. Some even are successful since years in the official Windows Store.

The above applications are part of the KDE Gear releases, but the described issues and solutions naturally are not restricted to stuff released with that.

What most of these applications have in common is that they rely on rather large parts of our Frameworks. With that they depend at least indirectly on an icon set that covers large parts of what our default icon set Breeze provides. Even if you use no icons from that icon set yourself in your application, just using the standard actions or many widgets/dialogs from Frameworks will rely on some subset of Breeze.

Current State of Icons per Desktop or Platform #

When talking about the current situation of icons that depends largely on the desktop or platform you are running the KDE application on.

Let’s take a look at some (I for sure miss some that are common or loved, that doesn’t mean I disregard them, I just want to limit the scope).

KDE Plasma on Linux/BSD with Wayland/X11 #

If you just aim to run on the KDE Plasma desktop with your Qt and KDE Frameworks based application, all is fine with icons, there is no problem.

The KDE project did their job, at least for Kate I never did have any issues with icons on Plasma.

Below a screenshot of Kate 24.02 running on Plasma 6. All icons are there, they are properly re-colored for the dark theme, too, including not just the used Breeze icons but for example the small Git icons in the left sidebar that Kate has bundled.

This is the vanilla state each user will get if Kate is installed on Plasma (and the dark theme is used). There are no patches done during building to achieve that nor is there any extra user configuration necessary.

Microsoft Windows #

If you run Kate on Windows, the icon situation is good, too, if you use our Windows Store variant or get at build done via Craft.

See below what the current nightly of Kate looks like in some Windows 11 VM (I just started it from the unpacked ZIP, no setup needed).

In the Craft build descriptions we do some patches to ensure the Breeze icons are bundled as library and the application links with that. In addition we ensure with some more patching that our own icon engine is used to allow for the proper recoloring.

If you don’t do that patching you will end up with close to no icons or for dark theme black on black icons.

Apple’s macOS #

The situation on macOS is the same as on Windows.

If you go with a Craft build of Kate, you will end up with something like below.

All icons are there and even application provided icons like our Git one are properly recolored.

Without the Craft patches Kate has more or less no icons like on Windows.

Haiku #

After covering Plasma and the two large closed-source desktop operating systems, as a small excursion, look how Kate (the KF5 based version) looks if installed on Haiku with the package they provide.

Kate looks ok, system icons intermixed with Breeze as fallback icons.

GNOME #

For testing this, I installed the latest Fedora Workstation in a VM. I have done no user configuration beside what the installer and initial setup wizard asked and then just installed the Kate package. The shell was even helpful to ask to do that after you just tried to start the not installed Kate.

Most icons not there, not that nice. For details about that read this post, we don’t need to re-iterate this again.

If you think: that is just Kate, let us just try Okular.

One thing that can be at least solved easily is that the icons are gone, we just install the Breeze icon set as package.

Looks ok, system icons intermixed with Breeze as fallback icons just like on Haiku. Not stylish but usable.

I was unable to trigger Kate or Okular to adjust to the dark mode GNOME provides, therefore I can not test if we end up with black on black icons there, but it is likely, as the fallback is just Breeze.

MATE #

Kate and Dolphin 24.02 on MATE with dark mode on NixOS, normal system packages, Breeze icons is installed.

System icons intermixed with Breeze as fallback icons, looks not that nice. Breeze icons not readable, as recoloring is not working.

Xfce #

Kate and Dolphin 24.02 on Xfce with dark mode on NixOS, normal system packages, Breeze icons is installed.

Same mix and unreadable state as on MATE.

Enlightenment #

Kate 24.02 on Enlightenment with dark mode on NixOS, normal system packages, Breeze icons is installed.

Just unreadable icons, beside out own Git icon and the few colored ones.

Summary: What’s up with Icons today #

The icons in KDE applications do look perfect on KDE Plasma. That should be no real surprise as many people working on these applications will test them there and KDE Frameworks and Qt are well tested on Plasma, too.

The icons look fine on Windows and macOS, too, at least for applications that got properly ported, but only thanks to patches we do in Craft. If you just grab e.g. Kate’s and the needed frameworks sources from our normal repositories, you don’t get that.

If the maintainers of the port for some OS do care, like the Haiku people, KDE applications can look fine there.

On other desktop environments it doesn’t look that great out of the box.

Unlike for the other operating systems, there the same packages without extra patches are running.

Whereas that works perfect on Plasma, we rely too much that the desktop environment running provides an icon set that has a similar coverage and naming as Breeze. As we don’t hard depend on the Breeze icons for our applications, it can even happen that just no fitting icons are there per default.

Even if that can be solved with some better package dependencies, you still end up with a patchwork look and without a Qt platform theme plugin that handles the needed recoloring to make dark mode feasible.

Getting it fixed #

Fortunately, just because the status quo is not that nice, it must not stay that way.

We have more or less all needed parts to fix the situation, we did already fix it during the porting to Windows and macOS.

We just never pushed to get this stuff done on Linux and Co.

How did we solve it there?

• We have the Breeze icon set as Qt resource inside a library and link with that. That makes them a hard build and runtime dependency and easy to deploy.
• We ensure the icon engine we have in our KIconThemes framework is there and used.
• We enforce the Breeze Qt style. (this is not really icon related, but ensures an usable look’n’feel, too)

The first and the last thing are easy to do on Linux and Co., too, even with still allowing the user to override the icon set and style, but still defaulting to Breeze.

The second point is harder, as that requires at the moment a few hacks and is not 100% as good as going the Qt platform theme plugin route we use inside Plasma.

For KDE Frameworks 6.3 we worked to get that done.

See our meta issue on our GitLab instance covering that topic.

All is not perfect, we will need to get some Qt API to fully do that, but the current state is already usable.

Here a comparison with the state as we have it now in our released software compared to with the state in the current master branch on an Cinnamon desktop.

The left side is the current Kate 24.02, the right side the current master build of Kate with master Frameworks.

The hard dependency to the Breeze icon library is done in KIconThemes, if you link to that, you are guaranteed that you have Breeze icons. You can naturally just link to only the Breeze icon library on your own.

The ensuring that the proper icon engine is done with some new API in KIconThemes that application developers must opt-in for. The same for the Qt style setup, there we have API in KConfigWidgets.

For Kate the concrete changes can be found here. They are minimal and even remove some platform specific code for the style setup.

Including fallback code for pre 6.3 Frameworks compatibility of the style setting, the basic idea is:

#include <KIconTheme> #define HAVE_STYLE_MANAGER __has_include(<KStyleManager>) #if HAVE_STYLE_MANAGER #include <KStyleManager> #endif int main(...) { /** * trigger initialisation of proper icon theme */ #if KICONTHEMES_VERSION >= QT_VERSION_CHECK(6, 3, 0) KIconTheme::initTheme(); #endif QApplication yourAppInstance(...); #if HAVE_STYLE_MANAGER /** * trigger initialisation of proper application style */ KStyleManager::initStyle(); #else /** * For Windows and macOS: use Breeze if available * Of all tested styles that works the best for us */ #if defined(Q_OS_MACOS) || defined(Q_OS_WIN) QApplication::setStyle(QStringLiteral("breeze")); #endif #endif ... }

In the long run, once 6.3 is the minimal version the application depends on, this is just:

#include <KIconTheme> #include <KStyleManager> int main(...) { /** * trigger initialisation of proper icon theme */ KIconTheme::initTheme(); QApplication yourAppInstance(...); /** * trigger initialisation of proper application style */ KStyleManager::initStyle(); ... }

At the moment KIconTheme::initTheme() is still a bit hacky until we have proper Qt API, but that is not visible for the API user.

If we get this properly done in our applications, that will not just solve the current issue for running them in other desktop environments.

With that API in use and the now already upstreamed patches, one can build vanilla Frameworks and Kate on Windows and macOS and the icons will just work in the resulting application bundles and you get an usable style out of the box if Breeze is there.

Help Wanted! #

We have now some API to help our applications to be more usable on non-Plasma installations and Windows and macOS.

We still need to make use of it and we need to improve the implementation and upstream to Qt the needed extra API to make it a real 100% replacement to what we do with the Plasma integration plugin.

If you have time to help us, show up on our meta issue.

Not just coding is needed, we for example have still a few icons that don’t recolor well, help to fix that is wanted, too.

Feedback #

You can provide feedback on the matching KDE Social, reddit or Hacker News post.

Learned a few things about xdg and mimetype registration in the last week that could be helpful to have condensed in a single place.

No Need to Ship a Mailcap Mime File

If you already ship a .desktop file (that is what ends up in /usr/share/applications/) which has a MimeType declared, there is no need to also ship a mailcap file (that is what ends up in /usr/lib/mime/packages/). Some triggers will do the conversion work for you. See also Debian Policy 4.9.

Reverse DNS Naming Convention for .desktop Files

Seems to be a closely guarded secret, maybe mainly known inside the Gnome world, but it's in the spec. Also not very widely known inside Debian if I look at my local system as not very representative sample.

Your hicolor Theme App Icon can be a Mime Type Icon as Well

In case you didn't know the hicolor icon theme is the default fallback theme. Many of us already install application icons e.g. in /usr/share/icons/hicolor/48x48/apps/ which is used in conjunction with the Icon field in the .desktop file to locate the application icon. Now the next step, and there it seems quite of few us miss out, is to create a symlink to also provide a mime type icon, so it's displayed in graphical file managers for the application data files. The schema here is simple: Take the MimeType e.g. application/x-vymand replace the / with a - and use that as file name in e.g. /usr/share/icons/hicolor/48x48/mimetypes/. In the vym case that is /usr/share/icons/hicolor/48x48/mimetypes/application-x-vym.png. If you have one use a scalable .svg file instead of .png.

This seems to be an area where Debian lacks a bit of tooling to automatically convert application icons to all the different sizes and install it in all the appropriate places. What is already there is a trigger to run gtk-update-icon-cache when you install new icons into one of the icon theme folder so they're picked up.

No Priority or Order in .desktop Files

Likely something that hapens on all my fresh installations: Libreoffice is installed and xdg-open starts to open pdf files with Libreoffice instead of evince. Now I've to figure out again to run xdg-mime default org.gnome.Evince.desktop application/pdf to change that (at least for my user). Background here is that the desktop file spec explicitly mandates "Priority for applications is handled external to the .desktop files.". That's why we got in addition to all of that mimeapps.list files. And now, after running the xdg-mime command from above, we've a ~/.config/mimeapps.list defining

[Default Applications] application/pdf=org.gnome.Evince.desktop

Debian as whole seems to be not very keen on shipping something like a sensible default mimeapps.list outside of desktop environment specific ones. A quick search gave me just

$ apt-file search mimeapps.list cinnamon-desktop-data: /usr/share/applications/x-cinnamon-mimeapps.list gdm3: /usr/share/gdm/greeter/applications/mimeapps.list gnome-session-common: /usr/share/applications/gnome-mimeapps.list plasma-workspace: /usr/share/applications/kde-mimeapps.list sxmo-utils: /usr/share/applications/mimeapps.list sxmo-utils: /usr/share/sxmo/xdg/mimeapps.list

While it's a bit anoying to run into that pdf vs Libreoffice thing every now and then, it's maybe better to not have long controversial threads about default pdf viewer, like the ones we already had about the default MTA choices. And while we're at it: everyone using Libreoffice should give a virtual hug to rene@ for taming that beast since 2010 and OpenOffice.org before.

I’m a little late in reporting on this topic, but Ruff put out an update in April 2024 that includes a hand-written recursive descent parser. This update is in version 0.4.0 and newer.

Ruff’s new parser is >2x faster, translating to a 20-40% speedup for all linting and formatting invocations. Ruff’s announcement includes some statistics to show improvements that are worth checking out.

What’s This New Parser?

I’ve never tried writing a code parser, so I’ll have to rely on Ruff’s announcement to explain this. Basically, when you are doing static analysis, you will turn the source code into Abstract Syntax Trees (ASTs), which you can then analyze. Python has an AST module built in for this purpose. Ruff is written in Rust, though, so their AST analyzer is also written in Rust.

The original parser was called a generated parser, specifically LALRPOP. The parser requires a grammar to be defined in a Domain Specific Language (DSL), which is then converted into executable code for the generator.

Ruff’s new hand-written parser is a recursive descent parser. Follow that link to Wikipedia to learn all the nitty gritty details.

Their team created a hand-written parser to give them more control and flexibility over the parsing process, making it easier to work on the many weird edge cases they need to support. They also created a new parser to make Ruff faster and provide better error messages and error resilience.

Wrapping Up

Ruff is great and makes linting and formatting your Python code so much faster. You can learn much more about Ruff in my other articles on this topic:

• An Intro to Ruff – An Extremely Fast Python Linter

• The Ruff Formatter – Python’s Fastest Formatter!

• Episode 23 – The Ruff Formatter with Charlie Marsh

The post Ruff – The Fastest Python Linter and Formatter Just Got Faster! appeared first on Mouse Vs Python.

Hello Planet KDE! I will be participating in Google Summer of Code this year adding Python support to a subset of the KDE Frameworks. You can follow my progress on this blog.

15 reasons I am excited about Drupal's new Starshot initiative

Drupal project lead Dries Buytaert just unveiled the Drupal Starshot Initiative and I couldn't be more excited! Starshot is all about putting users first. Enhancing Drupal core with additional components tailored to everyday users of the interface. Technically, it builds on Drupal core's robust foundation, utilizing Automatic Updates/Package Manager for seamless installation and maintenance, Recipes for the base system and on-demand additions, and Project Browser to help with recipe and extension discovery.

As I was talking to people about Starshot at DrupalCon Portland I realized it is easy to overlook how fundamental the changes proposed are in terms of how Drupal will innovate and the benefits end users will enjoy. At the same time those that did not have time to watch the whole keynote had understandable misconceptions about its nature. Some were concerned the fundamental changes are happening in the architecture, or a fork / parallel project is being set up. That is not where/how Starshot revolutionizes Drupal though. To address those misconceptions, I already worked on answers to frequently asked questions and published on the Starshot page.

This blog post will focus on the benefits that I see. I had the chance to participate in two hours of Starshot BoFs and then an hour long Q&A session at DrupalCon Portland. I also covered more technical details of Starshot's architecture in my Drupal 11 talk (including the most popular question on what the Launch button might lead to). Finally I had countless conversations with people at the event. Maybe confirmation bias, but there were only a couple people I talked to that were entirely skeptical. On the other hand I got very different viewpoints on what will the benefits be depending on who I talked to.

I think all of those are great, so I compiled them. Let's see 15 different reasons why I am super excited about Starshot and how it is very different from previous initiatives.

Gábor Hojtsy Sat, 05/11/2024 - 13:49

This week a lot of work that has been in progress for weeks got merged! So check out the free goodies! And isn’t that amazing? Free stuff day after day, week after week. No price tag, no ads, no spying, no activation, no subscription, no nonsense. Just good work donated to the public. And not only from KDE, but the software stack we rely on, the distributions that make our software available, and on and on! We really live in an amazing time, folks.

New Features

By default, Dolphin now selects everything in a folder when you double-click on its view background, and also lets you configure it to do other things instead–up to and including running custom terminal commands! (George Florea Bănuș, Dolphin 24.08. Link)

Elisa now lets you shuffle the playlist contents by album, not just by track (Bart De Vries, Elisa 24.08. Link)

System Settings now features a page where you can turn on and configure remote login based on the Remote Desktop Protocol (Akseli Lahtinen and me: Nate Graham, Plasma 6.1. Link 1 and link 2)

Final UI is still a bit in flux; you’ll notice that the merge request at the second link isn’t merged yet, and there are some obvious misalignments

On Wayland, KWin can now be configured to pull color profile information from the monitor’s EDID metadata where present. Note that color profile information in EDID metadata is often wrong, so use this setting with caution. The feature includes inline help text to make you aware of this, too. (Xaver Hugl, Plasma 6.2. Link 1, link 2, link 3, and link 4)

UI Improvements

It’s now more obvious how you end a screen recording in Spectacle: the “currently recording” icon it shows in the System Tray now animates to get your attention a bit more, and Spectacle also sends a system notification to tell you about it (Noah Davis, Spectacle 24.08. Link)

When the clock disappears on Plasma’s lock screen, the cursor does too, which makes it possible to use the screen locker as a true screensaver if you give it a wallpaper plugin that has some kind of animated effect (Someone amazing, Plasma 6.0.4. Link)

It’s now obvious how you close Plasma 6’s fancy new panel configuration dialog: it has a “Done” button in the corner! (Taro Tanaka and Me: Nate Graham, Plasma 6.1. Link):

When you disconnect from a network while it’s showing the speed graph view, it now automatically switches over to the info view (Eugene Popov, Plasma 6.1. Link)

Smooth scrolling in KDE’s QML-based apps is now optional (though still on by default). It’s also possible that 3rd-party apps will eventually read and respect this setting, as I recently noticed Firefox does for our global animation speed setting (Nathan Misner, Plasma 6.2. Link 1 and link 2)

Small in-window dialogs in QtQuick-based software have gotten a visual overhaul to remove everything not visually necessary, which gives the text and buttons more focus (Carl Schwan, Frameworks 6.3. Link 1 and link 2)

Also not the final appearance, but this is the general design direction right now

Command bars in QtWidgets-based apps have also gotten a visual overhaul to match this more minimalistic style (Eugene Popov, Frameworks 6.3. Link):

Bug Fixes

Elisa no longer freezes when you open Party Mode while music is playing and the headerbar is collapsed or sized in certain ways (Pedro Nishiyama, Elisa 24.05. Link)

Fixed two longstanding issues that could cause Plasma to crash when it didn’t find all the screens it expected to find when waking up or booting the system (Marco Martin, Plasma 6.0.4 and 6.0.5. Link 1 and link 2)

Discover no longer misleadingly and incorrectly claims that apps with no licenses listed are proprietary (Harald Sitter, Plasma 6.0.5. Link)

Fixed a Plasma 6 regression that caused Discover to show annoying and ignorable error messages when viewing pages for content from store.kde.org (Harald Sitter, Plasma 6.0.5. Link)

The search/filter field in Plasma’s Printers widget now works (Mike Noe, Plasma 6.0.5. Link)

Fixed a Plasma 6 regression that caused panel widgets to overlap when you have an Activity Pager widget somewhere on a horizontal panel (Edo Friedman, Plasma 6.0.5. Link)

KWin is now more reliable about turning off screens in response to hardware and driver quirks that previously made this less than reliable with certain setups (Arsen Arsenović, Plasma 6.0.5. Link)

Config windows for Plasma’s System Monitor and System Tray widgets and the power profiles OSD no longer have mismatched colors for some UI controls and icons when using a mixed light/dark global theme such as Breeze Twilight (Akseli Lahtinen, Evgeniy Harchenko, and Nicolas Fella, Plasma 6.0.5. Link 1, link 2, and link 3)

Searching for something in Plasma’s Clipboard widget now returns a message with the correct text (“No matches”) when your search didn’t match anything (Thomas Duckworth, Plasma 6.0.5. Link)

Plasma’s Task Manager widget was internally refactored to simplify some old crusty code, which fixes two prominent layout glitches, including a Plasma 6 regression where tasks would overlap with adjacent widgets when in multi-row mode (Marco Martin, Plasma 6.1. Link 1 and link 2)

Other bug information of note:

• 3 Very high priority Plasma bugs (same as last week). Current list of bugs
• 38 15-minute Plasma bugs (up from 35 last week). Current list of bugs
• 133 KDE bugs of all kinds fixed over the last week Full list of bugs

Automation & Systematization

The plasma-workspace git repo has adopted a merge request template to guide people towards writing good commit messages, testing their changes, and including before-and-after screenshots. If this works out well, we’ll expend it elsewhere too (me: Nate Graham, link)

We now have a bug announcement bot that yells at us about the number of high and very high priority Plasma bugs, as well as the number of current known regressions (Ben Bonacci, link):

…And Everything Else

This blog only covers the tip of the iceberg! If you’re hungry for more, check out https://planet.kde.org, where you can find more news from other KDE contributors.

How You Can Help

The KDE organization has become important in the world, and your time and labor have helped to bring it there! But as we grow, it’s going to be equally important that this stream of labor be made sustainable, which primarily means paying for it. Right now the vast majority of KDE runs on labor not paid for by KDE e.V. (the nonprofit foundation behind KDE, of which I am a board member), and that’s a problem. We’ve taken steps to change this with paid technical contractors—but those steps are small due to growing but still limited financial resources. If you’d like to help change that, consider donating today!

Otherwise, visit https://community.kde.org/Get_Involved to discover other ways to be part of a project that really matters. Each contributor makes a huge difference in KDE; you are not a number or a cog in a machine! You don’t have to already be a programmer, either. I wasn’t when I got started. Try it, you’ll like it! We don’t bite!

Since the last two posts about this topic (part one, part two) there has been some more progress, so let’s take a look.

Brightness Control

In Plasma 6.0, when HDR is enabled, you get to choose the brightness of SDR content in the display settings:

This is however not convenient to change quickly and doesn’t apply to HDR content, which so far is always presented at 100% brightness. In Plasma 6.1, you’ll be able to use the same ways to control brightness as on laptops or with displays that support DDC/CI in SDR mode: Just drag the brightness slider, press keyboard shortcuts or scroll on the brightness icon in the system tray, and it’ll dim the whole screen like you’d expect.

Powerdevil unfortunately only supports controlling the brightness of a single screen right now, and that limitation also applies here. The API it uses to communicate with KWin is per display though, so once powerdevil supports multiple screens, it’ll work for HDR displays too.

More accurate colors without a Colorimeter

Many new displays have a color gamut much wider than sRGB, and assume that their input signal is fitting for their color gamut - which means that, unless you use an ICC profile, colors will be much more saturated than they should be. There’s a few ways to correct this:

• find an sRGB option in your monitor’s OSD. This is usually pretty accurate if it’s available, but also limits all apps to sRGB
• buy a Colorimeter and profile your display. That costs money though, and the display profiling situation on Linux isn’t in great shape at the moment (DisplayCAL on Flathub is 4 years old, my last attempt at building it on Fedora didn’t work, and it only works correctly on Xorg atm)
• find an ICC profile for your display on the Internet and use that, hoping the display doesn’t deviate too much from the one that was profiled

There is a fourth option though: Use the color information from the display’s EDID. In Plasma 6.1, you can simply select this in the display settings.

Note that it comes with some caveats too:

• the EDID only describes colors with the default display settings, so if you change the “picture mode” or similar things in the display settings, the values may not be correct anymore
• the manufacturer may not measure every panel and just put generic values for the display model into the EDID
• the manufacturer may put completely wrong values in there (which is why this is disabled by default)
• even when correct values are provided, ICC profiles have much more detailed information on the display’s behavior than the EDID can contain

So if you care about color accuracy, this is not a way out of getting a Colorimeter and profiling your display… but if you just have two screens and you’re annoyed that one of them has much more intense colors than the other, this option is an easy and fast way to fix it.

Gamescope

Joshua Ashton implemented a new backend in gamescope, that uses Wayland subsurfaces to forward content to the host compositor instead of compositing it all into one image, and includes direct support for the frog_color_management_v1 protocol. The result of this is that with a new enough gamescope you don’t have to use any Vulkan layers to have gamescope pass HDR content to KWin, and you don’t have to use the --hdr-debug-force-output option anymore. If you want to play a game in HDR, you can now just put

gamescope -W 5120 -H 1440 --hdr-enabled --fullscreen %command%

into its launch options in Steam (with width and height adjusted to your screen ofc) and you’re done.

The backend also reduces the image copies made in comparison to the previously default SDL backend; the game’s buffers are directly passed to the host compositor, in most cases even while overlays are visible.

GPU Drivers

The driver situation is something I’ve been kind of ignoring in previous posts, but it’s obviously pretty important. First, let’s talk about the KMS API for setting a display into HDR mode. It consists of two parts:

• the HDR_OUTPUT_METADATA property, which compositors use to set mostly brightness related metadata about the image, like which transfer function is used and which brightness and color values the content roughly contains
• the Colorspace property, which compositors use to set the colorspace of the image, so that the display interprets the colors correctly

When you enable HDR in the system settings in Plasma, KWin will set HDR_OUTPUT_METADATA to the Perceptual Quantizer transfer function, and the brightness and mastering display properties to almost exactly what the display’s EDID says is optimal. This is done independently of the actual image content or what apps on the screen tell the compositor, to prevent the display from doing dumb things like dimming down just because an HDR video you opened claims it wants to display a supernova in your living room.

The one exception to that is the max_cll value - the maximum brightness of any pixel on the screen. It’s set to the maximum average brightness the display can show, because my Samsung C49RG94SSR monitor reduces the backlight brightness below SDR levels if you set the max_cll value it claims is ideal… With that one exception, this strategy has worked without issues so far.

On the Colorspace front, KWin sets the property to Default in SDR mode, and to BT2020_RGB when HDR is enabled. Sounds simple enough, but it’s of course actually more complicated.

Like any API that didn’t actually get used in practice for a very long time (if ever), both the API and the implementations were and are quite broken. The biggest issues I’ve seen so far are:

• AMD’s implementation of the Colorspace property for DisplayPort was broken, which caused colors to be washed out in HDR mode (fixed in Linux 6.8)
• the NVidia driver doesn’t force a modeset when HDR_OUTPUT_METADATA changes the transfer function or Colorspace changes its value, which causes temporary glitches when enabling HDR on some displays
• the Intel driver claims to support both properties for HDR laptop displays, but the implementation is missing entirely (this is being worked on)
• the Colorspace property implementations from Intel and NVidia cause washed out colors on many displays, because the API requires the compositor to change the property value depending on whether or not communication with the display uses RGB or YUV encoding… which the compositor doesn’t actually know anything about. The AMD implementation works around this by translating the property to the correct value in the kernel
• multiple laptop- or display specific issues that you can look up in the bug trackers if you want to

To summarize, if you want to use HDR, it’s best to use an AMD GPU with either kernel 6.8, or if you really must use an older kernel, HDMI. Even then, you might still see issues in some cases though - if you do, please make bug reports about it! Neither driver nor compositor developers can fix what they don’t know is broken.

What’s next?

As always, for every bit of progress made or feature implemented, there’s ten more upcoming exciting things that could be talked about or worked on… but the big next topic is offloading of color management tasks to the GPU’s scanout hardware, to save power and improve performance. Next week I’ll be attending the 2024 Linux Display Next hackfest, which will focus on exactly that, so stay tuned!

 

I've run simulations, tens of thousands of them at a time, over and over as we developed chips. In one project I noticed that I could predict the final result after only a small number of results were in which allowed me to halt the rest of the simulations, or make advance preparations for the final result.

I looked it up at the time and, indeed, there is an equation where if you want the pass rate of a "large" population to within a given accuracy, it will give you the minimum sample size to use.

To some, that's all gobledygook so I'll try to explain with some code.

Explanation

Lets say you have a large randomised population of pass/fails or ones and zeroes:

from random import sample
population_size = 100_000    # must be large, > 65K?sample_size = 123           # Actualp_hat = 0.5                  # Population pass rate, 0.5 == 50%
_ones = int(population_size*p_hat)_zeroes = population_size - _onespopulation = [1] * _ones + [0] * _zeroes

And that we take a sample from it and compute the pass rate of that single, smaller sample

def random_sample() -> list[int]:    return sample(population, k=sample_size)
pass_rate = (sum(random_sample())  # how many ones             / sample_size)      # convert to a pass rate
print(pass_rate)  # e.g. 0.59027552674230146

Every time we run that pass_rate expression we get a different value. It is random. We need to run that pass_rate calculation many times to get an idea of what the likely pass_rate would be when using the given sample size:

runs_per_sample = 10_000     # each sample size is run this many times ~2500
pass_rates = [sum(random_sample()) / sample_size              for _ in range(runs_per_sample)]

I have learnt that the question to ask is not how many times the sample pass rate was the population pass rate, but to define an acceptable margin of error, (say 5%) and say we want to be confident that pass rates be within that margin a certain percentage of the time

epsilon = 0.05               # target margin of error in sample pass rate. 0.05 == +/-5%
p_hat_min, p_hat_max = p_hat * (1 - epsilon), p_hat * (1 + epsilon)in_range_count = sum(p_hat_min <= pass_rate < p_hat_max                     for pass_rate in pass_rates)sample_confidence_level = in_range_count / runs_per_sampleprint(f"{sample_confidence_level = }")  # = 0.4054

So for a sample size of 123 we could expect the pass rate of the sample to be within 5% of the actual pass rate of the population, 0.5 or 50%, onlr o.4 or 40% of the time!

We need more!

What is actually done is we state what we think the population pass rate is, p_hat, (choose closer to 50% if you are unsure); the margin of error around p_hat we want, epsilon, usually +/-5% or +/-3%; and the confidence_level in hitting within the pass_rates margin of error.

There are calculators that will then give you n, the size of the sample needed to satisfy those condition.

Doing it myself

 I calculated for one specific sample size, above. Obviousely, if I calculated pass_rates over a range of sample_sizes, and increqasing runs_per_sample, I could search out the sample size needed.

That is done in my next program. I have to switch to using the numpy library for its speed and sample_size becomes a range.

When the pass rate confidence levels are calculated I end up with a list of confidence levels for increasing sample sizes that are usually not increasing due to the randomness, e.g.

range_hits = [...0.94, 0.95, 0.93, 0.954, ... 0.95, 0.96, 0.95, 0.96, 0.96, 0.97, ...]  # confidence levels

The range of sample_size corresponding to the first occurrence>= the requested confidence level, and the last occorrence of a confidence level < the requested confidence level in then slightly widened and the runs_per_sample increased on another iteration to get a better result.

Here's a sample of the output I get when searching

Sample output$ time python3 sample_search.py 2098 <= n <= 2610  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(50, 5000, 512), 250)
2013 <= n <= 2525  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(2013, 2694, 256), 500)
1501 <= n <= 2781  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1501, 3037, 256), 500)
1757 <= n <= 2013  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(221, 4061, 256), 500)
1714 <= n <= 1970  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1714, 2055, 128), 1000)
1458 <= n <= 2098  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1458, 2226, 128), 1000)
1586 <= n <= 1714  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(818, 2738, 128), 1000)
1564 <= n <= 1692  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1564, 1735, 64), 2000)
1500 <= n <= 1564  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1436, 1820, 64), 2000)
1553 <= n <= 1585  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1489, 1575, 32), 4000)
1547 <= n <= 1579  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1547, 1590, 16), 8000)
1547 <= n <= 1579  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1515, 1611, 16), 8000)
1541 <= n <= 1581  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1541, 1584, 8), 16000)
1501 <= n <= 1533  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1501, 1621, 8), 16000)
1501 <= n <= 1533  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1389, 1538, 8), 16000)
1503 <= n <= 1575  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1495, 1677, 8), 16000)
1503 <= n <= 1535  For p_hat, epsilon, confidence_level =(0.5, 0.05, 0.95)  Using population_size, sample_size, runs_per_sample =(100000, range(1491, 1587, 4), 32000)
 Use a sample n = 1535 to predict population pass rate of 50.0% +/-5% with a confidence level of 95%.
real    3m49.023suser    3m49.022ssys     0m1.361s

My Code# -*- coding: utf-8 -*-"""Created on Wed May  8 14:04:17 2024
@author: paddy"""# %%from random import sampleimport numpy as np

def sample_search(population_size, sample_size, p_hat, epsilon, confidence_level, runs_per_sample) -> range:    """    Arguments with example values:        population_size = 100_000           # must be large, > 65K?        sample_size = range(1400, 1750, 16) # (+min, +max, +step)        p_hat = 0.5                         # Population pass rate, 0.5 == 50%        epsilon = 0.05                      # target margin of error in sample pass rate. 0.05 == +/-5%        confidence_level = 0.95             # sample to be within p_hat +/- epsilon, 0.95 == 95% of the time.        runs_per_sample = 10_000            # each sample size is run this many times ~2500        Return:        min,max range for the sample size, n, satisfying inputs.        """        def create_1_0_array(population_size=100_000, p_hat=0.5) -> np.ndarray:        "Create numpy array of ones and zeroes with p_hat% as ones"        ones = int(population_size*p_hat + 0.5)        array10 = np.zeros(population_size, dtype=np.uint8)        array10[:ones] = 1                return array10

    def rates_of_samples(population: np.ndarray, sample_size_range: range, runs_per_sample: int                        ) -> list[list[float]]:        "Pass rates for range of sample sizes repeated runs_per_sample times."        # many_samples_many_rates = [( np.random.shuffle(population),         # shuffle every *run*        #                             [population[:s_count].sum() / s_count  # The pass rate for samples        #                             for s_count in sample_size_range]           #                         )[1]                                     # drop the shuffle        #                         for _ in range(runs_per_sample)]         # Every run
        many_samples_many_rates = [[ np.random.shuffle(population),         # shuffle every *sample*                                     [population[:s_count].sum() / s_count  # The pass rate for samples                                     for s_count in sample_size_range]                                      ][1]                                     # drop the shuffle                                   for _ in range(runs_per_sample)]         # Every run
        return list(zip(*many_samples_many_rates))      # Transpose to by_sample_size_then_runs

    population = create_1_0_array(population_size, p_hat)    by_sample_size_then_runs = rates_of_samples(population, sample_size, runs_per_sample)
    # Pass rates within target
    target_pass_range = tmin, tmax = p_hat * (1 - epsilon), p_hat * (1 + epsilon)  # Looking for rates within the range
    range_hits = [sum(tmin <= sample_pass_rate < tmax for sample_pass_rate in single_sample_size)                for single_sample_size in by_sample_size_then_runs]    runs_for_confidence_level = confidence_level * runs_per_sample
    for n_min, conf in zip(sample_size, range_hits):        if conf >= runs_for_confidence_level:            break    else:        n_min = sample_size.start
    for n_max, conf in list(zip(sample_size, range_hits))[::-1]:        if conf <= runs_for_confidence_level:            n_max += sample_size.step  # back a step            break    else:        n_max = sample_size.stop        if (n_min + sample_size.step) >= n_max and sample_size.step > 1:        # Widen        n_max = n_max + sample_size.step + 1
    return range(n_min, n_max, sample_size.step)

def min_max_mid_step(from_range: range) -> tuple[int, int, float, int]:    "Extract from **increasing** from_range the min, max, middle, step"    mn, st = from_range.start, from_range.step        # Handle range where start == stop    mx = from_range.stop    for mx in from_range:        pass        md = (mn + mx) / 2        return mn, mx, md, st    def next_sample_size(new_samples, last_samples,                     runs_per_sample,                     widener=1.33  # Widen range by                    ):    n_min, n_max, n_mid, n_step = min_max_mid_step(new_samples)    l_min, l_max, l_mid, l_step = min_max_mid_step(last_samples)        # Next range of samples computes in names with prefix s_        increase_runs = True    if n_max == l_max:        # Major expand of high end        s_max = l_max + (l_max - l_min)        increase_runs = False    else:        s_max = (n_mid +( n_max - n_mid)* widener)
    if n_min == l_min:        # Major expand of low end        s_min = max(1, l_min + (l_min - l_max))        increase_runs = False    else:        s_min = (n_mid +( n_min - n_mid)* widener)        s_min, s_max = (max(1, int(s_min)), int(s_max + 0.5))    s_step = n_step    if s_min == s_max:        if s_min > 2:            s_min -= 1        s_max += 1            if increase_runs or n_max == n_min:        runs_per_sample *= 2        if n_max == n_min:            s_step = 1        else:            s_step = max(1, (s_step + 1) // 2)  # Go finer        next_sample_range = range(max(1, int(s_min)), int(s_max + 0.5), s_step)        return next_sample_range, runs_per_sample
# %%
if __name__ == '__main__':
    population_size = 100_000           # must be large, > 65K?    sample_size = range(50, 5_000, 512) # Increasing!    p_hat = 0.50                        # Population pass rate, 0.5 == 50%    epsilon = 0.05                      # target margin of error in sample pass rate. 0.05 == +/-5%    confidence_level = 0.95             # sample to be within p_hat +/- epsilon, 0.95 == 95% of the time.    runs_per_sample = 250               # each sample size is run this many time at start, ~250    max_runs_per_sample = 35_000
    while runs_per_sample < max_runs_per_sample:        new_range = sample_search(population_size, sample_size, p_hat, epsilon, confidence_level, runs_per_sample)        n_min, n_max, n_mid, n_step = min_max_mid_step(new_range)        print(f"{n_min} <= n <= {n_max}")        print(f"  For {p_hat, epsilon, confidence_level =}\n"            f"  Using {population_size, sample_size, runs_per_sample =}\n")                    sample_size, runs_per_sample = next_sample_size(new_range, sample_size, runs_per_sample)
print(f" Use a sample n = {n_max} to predict population pass rate of {p_hat*100.:.1f}% +/-{epsilon*100.:.0f}% "      f"with a confidence level of {confidence_level*100.:.0f}%.")

END.

 

Learn about workflow configuration and customizations to empower your website’s content approver and publisher roles.