Feeds

Kay Hayen: Nuitka this week #15

Planet Python - Mon, 2024-01-29 18:00

This is a weekly update, or at least it’s supposed to be of what’s going on in Nuitka land, for you to learn about ongoing developments and important changes to the project.

In this issue, I am first going to cover a bit of backlog from news update missed in the past, but also covering very exciting changes from this week.

Contents

Nuitka Action

A GitHub Action is a component used in GitHub workflows. These are yaml driven configurations that can cause GitHub to do automatic building of your software.

Many of the more professional users build their binaries as part of GitHub workflows, and Nuitka and Nuitka commercial are both used in that way a lot. Many times people do it on their own, i.e. install Nuitka by hand, and call it by hand, which is kind of not the preferred way for many people.

Enter the great Nuitka Action which was originally created by Jim Kring, who handed over the maintenance of it to the Nuitka organization that has further refined it. This was a great contribution that makes it everything easier for Nuitka users on GitHub if they want to use it.

- name: Build Executable uses: Nuitka/Nuitka-Action@main with: nuitka-version: main script-name: kasa_cli onefile: true - name: Upload Artifacts uses: actions/upload-artifact@v3 with: name: ${{ runner.os }} Build path: | build/*.exe build/*.bin build/*.app/**/*

Options of Nuitka are exposed as yaml attributes. The documentation of this mapping could be very much enhanced, but basically it’s just dropping the -- part from e.g. --onefile and for toggles, you say true.

Now one interesting limitation of GitHub Action, I have come across this week and that is that it’s not easily possible to specify an option twice. For some values in Nuitka, that however is necessary. Where module names are acceptable, a , separation os supported, but with file paths, we don’t do that, e.g. not with --include-data-dir (note --include-package-data is much better to use) but that is the one it came up for.

But now we support splitting by new-line from GitHub actions for everything that produces a list value as a Nuitka option. See below for a very nice example of how the | in Yaml makes that even easy to read.

- name: Build Executable uses: Nuitka/Nuitka-Action@main with: nuitka-version: main script-name: kasa_cli onefile: true include-data-dir: | source_path_dir1=dest_path_dir1 source_path_dir2=dest_path_dir2 source_path_dir3=dest_path_dir3

Note

This works with Nuitka 2.0 or higher.

The Nuitka-Action is permanently refined. Just today we updated its caching action to latest, and there is an ongoing activity to improve options. We started to generate options from Nuitka help output directly, so that it is easier to add support for new Options in Nuitka, and to generally make them more consistent.

Community

On the Discord server, you can get in touch with an ever more vibrant community of Nuitka users. You are welcome to join us on the Discord server for Nuitka community where you can hang out with the developers and ask questions. It’s not intended as an interactive manual. You are supposed to read the docs for yourself first. And issues are best reported to GitHub.

I am also now occasionally on the Python Discord server. Mostly when I get summoned to answer questions that my community thinks make sense, and have been awarded the community role there, which is pretty nice. I seem to make new connections there.

Optimization Work

For me this is extremely exciting, this has been on my nerves for a long time, and I didn’t have the time to figure it out. Now for the scalability work, I wanted to make sure the algorithm used for loop type analysis is actually going to be sustainable, before I optimize the implementation to scale better.

And low and behold, one of my oldest code examples, the one I mean to demonstrate C type performance from Python code with, has failed to get proper results for a long time now. But this changed this week and it’s part of the 2.0 release, making it my mind worth the bump itself. Checkout this annotated code.

# Initially the value of undefined "i" is "NUITKA_NINT_UNASSIGNED" # in its indicator part. The C compiler will remove that assignment # as it's only checked in the assignment coming up. i = 0 # Assignment from a constant, produces a value where both the C # and the object value are value. This is indicated by a value # of "NUITKA_NINT_BOTH_VALID". The code generation will assign # both the object member from a prepared value, and the clong # member to 0. # For the conditional check, "NUITKA_NINT_CLONG_VALID" will # always be set, and therefore function will resort to comparing # that clong member against 9 simply, that will always be very # fast. Depending on how well the C compiler can tell if an overflow # can even occur, such that an object might get created, it can even # optimize that statically. In this case it probably could, but we # do not rely on that to be fast. while i < 9: # RICH_COMPARE_LT_CBOOL_NINT_CLONG # Here, we might change the type of the object. In Python2, # this can change from ``int`` to ``long``, and our type # analysis tells us that. We can consider another thing, # not "NINT", but "NINTLONG" or so, to special case that # code. We ignore Python2 here, but multiple possible types # will be an issue, e.g. list or tuple, float or complex. # So this calls a function, that returns a value of type # "NINT" (actually it will become an in-place operation # but lets ignore that too). # That function is "BINARY_OPERATION_ADD_NINT_NINT_CLONG"(i, 1) # and it is going to check if the CLONG is valid, add the one, # and set to result to a new int. It will reset the # "NUITKA_NINT_OBJECT_VALID" flag, since the object will not be # bothered to create. i = i + 1 # Since "NUITKA_INT_OBJECT_VALID" not given, need to create the # PyObject and return it. return i

Now that the loop analysis works, I will be much happier to make the value trace collection faster. I will describe it when I do it. From here on for optimization, the C type NINT needs to be created and code generation for the branching helper functions be added, and then the should see this perform perfectly.

Functions like RICH_COMPARE_LT_CBOOL_NINT_CLONG will look like this. We do not yet have RICH_COMPARE_LT_CBOOL_LONG_CLONG which it will fall back to, but we did RICH_COMPARE_LT_CBOOL_INT_CLONG for Python2 a while ago, and we could expand that no problem.

extern bool RICH_COMPARE_LT_CBOOL_NINT_CLONG(nuitka_long *operand1, long operand2) { if (operand1->validity & NUITKA_LONG_VALUE_VALID) { return operand1->long_value < operand2; } else { return RICH_COMPARE_LT_CBOOL_LONG_CLONG(operand1->long_object, operand2); } }

Once I get to that, performance will get a hot topic. From there then, adding sources of type information, be it profile guided compilation, be it type annotations, be it ever better compile time type inference, will start to make a lot more sense.

Nuitka 2.0

The 2.0 release has been made. I am going to announce it separately. I am usually waiting for a few days, to settle potentially regressions. This time older C compiler support needed a fixup, there is always something. And then I announce it when I feel that the regressions are gone and that new users will not encounter obvious breakage at all.

Technical Writer

When I first launched Nuitka commercial, I needed to get myself financially supported, dropping my day job after 20 years. I am willing to say that has happened.

Now as you all know, Nuitka is technically excellent. I cannot say the same thing of the documentation. Large parts of Nuitka commercial are still not even publicly described. The User Manual of Nuitka is good, but not nearly as good as it should be. The website is kind of unorganized. It’s pretty obvious that my talent is not there. I have received plenty of help over the years, but it’s not been enough to match Nuitka and Nuitka commercial outstanding quality.

So, what I did this week, after seeing that my financial projection for the coming years seems to allow it, is to attempt and hire people on a free lancing basis. The first step is a technical writer. She will know very little of Python and even the terminal, but she will know how to organize and improve the content of Nuitka.

It will take time for her to get there and this is very fresh.

Nuitka Website as a DevContainer

As a first necessary step to make it easier to contribute to the Nuitka documentation, the website repo, has gained DevContainer configuration. It will install a small Ubuntu via docker (or podman if you configured Visual Code like that), and run the pipenv environment and a daemon to open the website.

The docs for that are spotty right now, and the Technical Writer that is using that, is tasked to improve this right now.

It should become really easy that way to contribute enhancements to the documentation.

I have yet to figure out how to handle the release matching documentation vs. website documentation for user manual. But the idea is certainly that the Nuitka documentation is edited on the website.

Nuitka Website Refinements

With the DevContainer the need for translation and staging sites is gone. The Nuitka/Website has been disbanded, since it was only used to control access to “live” rendered branches of the website, that are no more.

As part of the DevContainer process, the website build was changed to Python 3.10 so that Ubuntu image is easier to use (was Debian 3.9 so far). The used tools got all upgraded, and many small improvements came out of it. Links got checked after the upgrade, finding a few broken ones, and the translation dropdown is now only present when there are actual translations. Previously e.g. all posts were having them, which made no sense.

Making the container smoother to use will be an ongoing process. How to integration Nuitka auto-format in an easy fashion is still being looked at.

Nuitka Package Configuration

So I added a post that explains variables, but the one for parameters, I still need to do that and also update the actual reference documentation.

Teasers

Future TWN still have a lot to talk about, we will speak about Nuitka-Python (our own Python fork with incredible capabilities), about Nuitka-Watch (our way of making sure Nuitka works with PyPI packages and hot-fixes to not regress), about compilation reports as a new feature, Windows AV stuff, onefile improvements, and so on and so on. I got interesting stuff for many weeks. Limiting myself for now or I will never publish this.

Twitter and Mastodon

I should be more active there, although often I fail due to not wanting to talk about unfinished things, so actually I do not post there as much.

And lets not forget, having followers make me happy. So do re-tweets. Esp. those, please do them.

Help Wanted

Nuitka definitely needs more people to work on it. I hope the technical writer will aid us in better laying how ways for you to help.

System Message: INFO/1 (/home/nuitka-buildslave/slave/site-main-update/build/doc/posts/nuitka-this-week-15.rst, line 7); backlink

Duplicate implicit target name: “help wanted”.

If you are interested, I am tagging issues help wanted and there is a bunch, and very likely at least one you can help with.

Categories: FLOSS Project Planets

Matthew Palmer: Why Certificate Lifecycle Automation Matters

Planet Debian - Mon, 2024-01-29 17:30

If you’ve perused the ActivityPub feed of certificates whose keys are known to be compromised, and clicked on the “Show More” button to see the name of the certificate issuer, you may have noticed that some issuers seem to come up again and again. This might make sense – after all, if a CA is issuing a large volume of certificates, they’ll be seen more often in a list of compromised certificates. In an attempt to see if there is anything that we can learn from this data, though, I did a bit of digging, and came up with some illuminating results.

The Procedure

I started off by finding all the unexpired certificates logged in Certificate Transparency (CT) logs that have a key that is in the pwnedkeys database as having been publicly disclosed. From this list of certificates, I removed duplicates by matching up issuer/serial number tuples, and then reduced the set by counting the number of unique certificates by their issuer.

This gave me a list of the issuers of these certificates, which looks a bit like this:

/C=BE/O=GlobalSign nv-sa/CN=AlphaSSL CA - SHA256 - G4 /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Domain Validation Secure Server CA /C=GB/ST=Greater Manchester/L=Salford/O=Sectigo Limited/CN=Sectigo RSA Organization Validation Secure Server CA /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 /C=US/ST=Arizona/L=Scottsdale/O=Starfield Technologies, Inc./OU=http://certs.starfieldtech.com/repository//CN=Starfield Secure Certificate Authority - G2 /C=AT/O=ZeroSSL/CN=ZeroSSL RSA Domain Secure Site CA /C=BE/O=GlobalSign nv-sa/CN=GlobalSign GCC R3 DV TLS CA 2020

Rather than try to work with raw issuers (because, as Andrew Ayer says, The SSL Certificate Issuer Field is a Lie), I mapped these issuers to the organisations that manage them, and summed the counts for those grouped issuers together.

The Data Insert obligatory "not THAT data" comment here

The end result of this work is the following table, sorted by the count of certificates which have been compromised by exposing their private key:

IssuerCompromised Count Sectigo170 ISRG (Let's Encrypt)161 GoDaddy141 DigiCert81 GlobalSign46 Entrust3 SSL.com1

If you’re familiar with the CA ecosystem, you’ll probably recognise that the organisations with large numbers of compromised certificates are also those who issue a lot of certificates. So far, nothing particularly surprising, then.

Let’s look more closely at the relationships, though, to see if we can get more useful insights.

Volume Control

Using the issuance volume report from crt.sh, we can compare issuance volumes to compromise counts, to come up with a “compromise rate”. I’m using the “Unexpired Precertificates” colume from the issuance volume report, as I feel that’s the number that best matches the certificate population I’m examining to find compromised certificates. To maintain parity with the previous table, this one is still sorted by the count of certificates that have been compromised.

IssuerIssuance VolumeCompromised CountCompromise Rate Sectigo88,323,0681701 in 519,547 ISRG (Let's Encrypt)315,476,4021611 in 1,959,480 GoDaddy56,121,4291411 in 398,024 DigiCert144,713,475811 in 1,786,586 GlobalSign1,438,485461 in 31,271 Entrust23,16631 in 7,722 SSL.com171,81611 in 171,816

If we now sort this table by compromise rate, we can see which organisations have the most (and least) leakiness going on from their customers:

IssuerIssuance VolumeCompromised CountCompromise Rate Entrust23,16631 in 7,722 GlobalSign1,438,485461 in 31,271 SSL.com171,81611 in 171,816 GoDaddy56,121,4291411 in 398,024 Sectigo88,323,0681701 in 519,547 DigiCert144,713,475811 in 1,786,586 ISRG (Let's Encrypt)315,476,4021611 in 1,959,480

By grouping by order-of-magnitude in the compromise rate, we can identify three “bands”:

  • The Super Leakers: Customers of Entrust and GlobalSign seem to love to lose control of their private keys. For Entrust, at least, though, the small volumes involved make the numbers somewhat untrustworthy. The three compromised certificates could very well belong to just one customer, for instance. I’m not aware of anything that GlobalSign does that would make them such an outlier, either, so I’m inclined to think they just got unlucky with one or two customers, but as CAs don’t include customer IDs in the certificates they issue, it’s not possible to say whether that’s the actual cause or not.

  • The “Regular” Leakers: Customers of SSL.com, GoDaddy, and Sectigo all have compromise rates in the 1-in-hundreds-of-thousands range. Again, the low volumes of SSL.com make the numbers somewhat unreliable, but the other two organisations in this group have large enough numbers that we can rely on that data fairly well, I think.

  • The Low Leakers: Customers of DigiCert and Let’s Encrypt are at least three times less likely than customers of the “regular leakers” to lose control of their private keys. Good for them!

Now we have some useful insights we can think about.

Why Is It So? If you don't know who Professor Julius Sumner Miller is, I highly recommend finding out

All of the organisations on the list, with the exception of Let’s Encrypt, are what one might term “traditional” CAs. To a first approximation, it’s reasonable to assume that the vast majority of the customers of these traditional CAs probably manage their certificates the same way they have for the past two decades or more. That is, they generate a key and CSR, upload the CSR to the CA to get a certificate, then copy the cert and key… somewhere. Since humans are handling the keys, there’s a higher risk of the humans using either risky practices, or making a mistake, and exposing the private key to the world.

Let’s Encrypt, on the other hand, issues all of its certificates using the ACME (Automatic Certificate Management Environment) protocol, and all of the Let’s Encrypt documentation encourages the use of software tools to generate keys, issue certificates, and install them for use. Given that Let’s Encrypt has 161 compromised certificates currently in the wild, it’s clear that the automation in use is far from perfect, but the significantly lower compromise rate suggests to me that lifecycle automation at least reduces the rate of key compromise, even though it doesn’t eliminate it completely.

Sidebar: ACME Does Not Currently Rule The World

It is true that all of the organisations in this analysis also provide ACME issuance workflows, should customers desire it. However, the “traditional CA” companies have been around a lot longer than ACME has, and so they probably acquired many of their customers before ACME existed.

Given that it’s incredibly hard to get humans to change the way they do things, once they have a way that “works”, it seems reasonable to assume that most of the certificates issued by these CAs are handled in the same human-centric, error-prone manner they always have been.

If organisations would like to refute this assumption, though, by sharing their data on ACME vs legacy issuance rates, I’m sure we’d all be extremely interested.

Explaining the Outlier

The difference in presumed issuance practices would seem to explain the significant difference in compromise rates between Let’s Encrypt and the other organisations, if it weren’t for one outlier. This is a largely “traditional” CA, with the manual-handling issues that implies, but with a compromise rate close to that of Let’s Encrypt.

We are, of course, talking about DigiCert.

The thing about DigiCert, that doesn’t show up in the raw numbers from crt.sh, is that DigiCert manages the issuance of certificates for several of the biggest “hosted TLS” providers, such as CloudFlare and AWS. When these services obtain a certificate from DigiCert on their customer’s behalf, the private key is kept locked away, and no human can (we hope) get access to the private key. This is supported by the fact that no certificates identifiably issued to either CloudFlare or AWS appear in the set of certificates with compromised keys.

When we ask for “all certificates issued by DigiCert”, we get both the certificates issued to these big providers, which are very good at keeping their keys under control, as well as the certificates issued to everyone else, whose key handling practices may not be quite so stringent.

It’s possible, though not trivial, to account for certificates issued to these “hosted TLS” providers, because the certificates they use are issued from intermediates “branded” to those companies. With the crt.sh psql interface we can run this query to get the total number of unexpired precertificates issued to these managed services:

SELECT SUM(sub.NUM_ISSUED[2] - sub.NUM_EXPIRED[2]) FROM ( SELECT ca.name, max(coalesce(coalesce(nullif(trim(cc.SUBORDINATE_CA_OWNER), ''), nullif(trim(cc.CA_OWNER), '')), cc.INCLUDED_CERTIFICATE_OWNER)) as OWNER, ca.NUM_ISSUED, ca.NUM_EXPIRED FROM ccadb_certificate cc, ca_certificate cac, ca WHERE cc.CERTIFICATE_ID = cac.CERTIFICATE_ID AND cac.CA_ID = ca.ID GROUP BY ca.ID ) sub WHERE sub.name ILIKE '%Amazon%' OR sub.name ILIKE '%CloudFlare%' AND sub.owner = 'DigiCert';

The number I get from running that query is 104,316,112, which should be subtracted from DigiCert’s total issuance figures to get a more accurate view of what DigiCert’s “regular” customers do with their private keys. When I do this, the compromise rates table, sorted by the compromise rate, looks like this:

IssuerIssuance VolumeCompromised CountCompromise Rate Entrust23,16631 in 7,722 GlobalSign1,438,485461 in 31,271 SSL.com171,81611 in 171,816 GoDaddy56,121,4291411 in 398,024 "Regular" DigiCert40,397,363811 in 498,732 Sectigo88,323,0681701 in 519,547 All DigiCert144,713,475811 in 1,786,586 ISRG (Let's Encrypt)315,476,4021611 in 1,959,480

In short, it appears that DigiCert’s regular customers are just as likely as GoDaddy or Sectigo customers to expose their private keys.

What Does It All Mean?

The takeaway from all this is fairly straightforward, and not overly surprising, I believe.

The less humans have to do with certificate issuance, the less likely they are to compromise that certificate by exposing the private key.

While it may not be surprising, it is nice to have some empirical evidence to back up the common wisdom.

Fully-managed TLS providers, such as CloudFlare, AWS Certificate Manager, and whatever Azure’s thing is called, is the platonic ideal of this principle: never give humans any opportunity to expose a private key. I’m not saying you should use one of these providers, but the security approach they have adopted appears to be the optimal one, and should be emulated universally.

The ACME protocol is the next best, in that there are a variety of standardised tools widely available that allow humans to take themselves out of the loop, but it’s still possible for humans to handle (and mistakenly expose) key material if they try hard enough.

Legacy issuance methods, which either cannot be automated, or require custom, per-provider automation to be developed, appear to be at least four times less helpful to the goal of avoiding compromise of the private key associated with a certificate.

Humans Are, Of Course, The Problem No thanks, Bender, I'm busy tonight

This observation – that if you don’t let humans near keys, they don’t get leaked – is further supported by considering the biggest issuers by volume who have not issued any certificates whose keys have been compromised: Google Trust Services (fourth largest issuer overall, with 57,084,529 unexpired precertificates), and Microsoft Corporation (sixth largest issuer overall, with 22,852,468 unexpired precertificates). It appears that somewhere between “most” and “basically all” of the certificates these organisations issue are to customers of their public clouds, and my understanding is that the keys for these certificates are managed in same manner as CloudFlare and AWS – the keys are locked away where humans can’t get to them.

It should, of course, go without saying that if a human can never have access to a private key, it makes it rather difficult for a human to expose it.

More broadly, if you are building something that handles sensitive or secret data, the more you can do to keep humans out of the loop, the better everything will be.

Your Support is Appreciated

If you’d like to see more analysis of how key compromise happens, and the lessons we can learn from examining billions of certificates, please show your support by buying me a refreshing beverage. Trawling CT logs is thirsty work.

Appendix: Methodology Limitations

In the interests of clarity, I feel it’s important to describe ways in which my research might be flawed. Here are the things I know of that may have impacted the accuracy, that I couldn’t feasibly account for.

  • Time Periods: Because time never stops, there is likely to be some slight “mismatches” in the numbers obtained from the various data sources, because they weren’t collected at exactly the same moment.

  • Issuer-to-Organisation Mapping: It’s possible that the way I mapped issuers to organisations doesn’t match exactly with how crt.sh does it, meaning that counts might be skewed. I tried to minimise that by using the same data sources (the CCADB AllCertificates report) that I believe that crt.sh uses for its mapping, but I cannot be certain of a perfect match.

  • Unwarranted Grouping: I’ve drawn some conclusions about the practices of the various organisations based on their general approach to certificate issuance. If a particular subordinate CA that I’ve grouped into the parent organisation is managed in some unusual way, that might cause my conclusions to be erroneous. I was able to fairly easily separate out CloudFlare, AWS, and Azure, but there are almost certainly others that I didn’t spot, because hoo boy there are a lot of intermediate CAs out there.

Categories: FLOSS Project Planets

Talking Drupal: Talking Drupal #435 - UI Suite initiative

Planet Drupal - Mon, 2024-01-29 14:00

Today we are talking about web design and development, from a group of people with one thing in common… We love Drupal. This is episode #435 UI Suite initiative.

For show notes visit: www.talkingDrupal.com/435

Topics
  • Elevator Pitch
  • What do you mean by implementing design systems
  • Is this to thel themers or site builders
  • What is the expected outcome
  • The project page says initiative, but this is not on the community initiatives page, is this an intitiative
  • How does this implement public design systems
  • Does this work with Single Directory Components
  • Youtube Channel
  • Getting involved
  • Roadmap
  • Use cases
  • Do you see this going into core
Resources Guests

Pierre Dureau - LinkedIn pdureau

Hosts

Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi

MOTW Correspondent

Chris Wells - chrisfromredfin

  • Brief description:
    • Have you ever wanted simple but accessible dropdown navigation on your Drupal website? There’s a module for that.
  • Module name/project name:
  • Brief history
    • How old: created in Nov 2022 by Jay Huskins of Redfin Solutions
    • Versions available: 2.1.5 which works with Drupal 9 and 10
  • Maintainership
    • Actively maintained, most recent commit is less than a month ago
    • No Test coverage at this point
    • Documentation - via README.md
    • Number of open issues: 2, neither of which are bugs
  • Usage stats:
    • 29 sites
  • Maintainer(s):
    • Jay Huskins, with support from Redfin
  • Module features and usage
    • Provides a new menu block that includes markup for disclosure buttons, that make it easier for users on assistive devices to navigate drop-down menus
    • Activating the button will toggle the visibility of nested menu items
    • Also includes ARIA labels to provide clarity around what the buttons do
    • Worth mentioning that Drupal’s Olivero theme also includes disclosure buttons in its mWe’ll also cover Disclosure Menu as our module of the week.
    • Guest Introenus, but there isn’t a straightforward way to pull the menus implemented there into your custom theme
    • Also had a write-up in the Drupal advent calendar we talked about last month, so listeners can also check that out if they want more information
Categories: FLOSS Project Planets

The Drop Times: Wrapping Up the First Month of 2024

Planet Drupal - Mon, 2024-01-29 12:09

Dear Reader,

As we bid farewell to the inaugural month of the new year, we find ourselves standing at the intersection of introspection and eagerness. With its promise of renewal and a clean slate, January has set the stage for the unfolding chapters of the coming months. It's a time marked by resolutions, goal-setting, and embracing the boundless possibilities.

In reflecting on this initial chapter, let's take a moment to assess our achievements, acknowledge progress, and eagerly anticipate the challenges and triumphs on the horizon. The journey of the year has only just begun, and the unwritten pages of our story beckon with anticipation.

As January gracefully transitions into February, the closure of the Global Contribution Weekend in the Drupal community symbolizes a significant close to an impactful month. Here is a compilation of Drupal Contribution Weekend celebrations that occurred across the globe. Nico Grienauer has an explainer of what this event is

As we bid farewell to January, the Drupal Community sets its sights on a month steeped in celebration. Often hailed as the month of love, February encourages us to bask in the warmth of connections and expressions of affection. With numerous global events, Drupal is poised to embrace the heartwarming spirit of togetherness.

Now, let's shift focus and explore some of the latest news stories and articles we covered last week.
Kazima recently engaged in a compelling interview with Fran Garcia-Linares, Senior Drupal Developer at the Drupal Association. During the interview, Fran discussed the Association's strategic initiatives, including migrating from Drupal 7 to Drupal 10 and spearheading the GitLab Acceleration initiative. Additionally, Kazima delivered comprehensive coverage of the latest developments and insights from Drupal Mountain Camp 2024.

Acquia has exciting webinars lined up. On February 7, 2024, they will host "Mastering Your DAM Strategy and Governance Plan for 2024 and Beyond," aiding businesses in navigating asset demands through digital asset management (DAM) solutions. Additionally, on January 29, 2024, at 12:15 pm EST, Acquia will delve into "Navigating the AI Labyrinth: Security & Compliance in Modern DXPs and CMSs."

Jesse Loseberg leads an A11yTalk session on February 08, 2024, at 01:30 PM (ET), titled "The Accessibility Mindset: Moving Beyond Remediating, Fixing, and Reacting." and at the Florida DrupalCamp on February 24, 2024, from 1:00 pm to 1:45 pm, Steve Wirt of CivicActions explores "Unraveling the Clues to Effective Drupal Site Documentation," tailored for developers, project managers, content designers, system architects, and UX designers.

Droptica's webinar, "The Future After Drupal 7: Exploring Alternatives and New Opportunities," is on February 22 at 3:00 pm CEST. Save the date for Drupalcamp Rennes on March 28-30, 2024, promising three days of Drupal Camp goodness. Stay updated with this week's Drupal events by checking the published list here.

NERD Summit 2024 has extended its session proposal deadline until all 40 session slots are filled. Additionally, the event offers sponsorship opportunities for keynote talks. For more details, check the provided link. DrupalSouth Sydney 2024, happening at the Sydney Masonic Centre from March 20 to 22, is a significant gathering for the Australian and New Zealand Drupal community. GitHub has opened its certification program to the public, offering individuals worldwide a chance to enhance their skills and credentials. Learn more at this link.

Splash Awards Australia and New Zealand are calling for submissions in 2024. Find more details here. Explore the details of the upcoming DrupalCon Portland, which features a dedicated marketing track. Find more information here.

Varbase 9.1.1 is now available, offering a patch (bugfix) release for Varbase 9 suitable for production sites. Created by Rajab Natshah, this release addresses issues and ensures smoother functionality. Also the PHP development team has released PHP 8.2.15 and PHP 8.3.2, introducing important updates and improvements to the widely used scripting language.

There are more stories available out there. But the compulsion to limit the selection of stories is forcing us to put a hard break on further exploration—happy Drupaling folks.

To get timely updates, follow us on LinkedIn, Twitter and Facebook.

Thank you,

Sincerely
Elma John
Sub-editor, TheDropTimes.

Categories: FLOSS Project Planets

Steinar H. Gunderson: bcachefs boot tweaks

Planet Debian - Mon, 2024-01-29 12:09

Following my previous foray into bcachefs-on-/ booting, I whipped up some patches to make multidevice root filesystems boot in Debian:

  • bcachefs-tools: Enable the Rust parts in the Debian package, and upgrade to latest git HEAD.
  • klibc: Add bcachefs detection and UUID extraction to fstype (probably not needed, blkid is used as a fallback and seems to work fine)
  • initramfs-tools: Don't rewrite root=UUID=12345678 back to a /dev device for bcachefs (which would find only an arbitrary device in the filesystem, not all of them); keep the UUID= around so that mount.bcachefs can find everything.
  • grub: Enable support in grub-mkconfig for reading UUIDs from bcachefs, and to generally understand the multi-device /dev syntax.

With all of these applied, and then the 6.7 kernel and the right fstab setups, I can seemingly boot just fine. Still no installer support, of course, but that would be a bit premature.

There's also a minor quibble in libzstd-dev; if it's fixed, it's possible to take the sid-built bcachefs-tools .deb and install directly on bookworm, which has a value in itself, given that mount.bcachefs is not built in bookworm.

Categories: FLOSS Project Planets

mandclu: Prompt Engineering: Get the Most From Your Drupal Site's AI Integration

Planet Drupal - Mon, 2024-01-29 09:39
Mon, 01/29/2024 - 09:39 Prompt Engineering: Get the Most From Your Drupal Site's AI Integration

A recent analysis by the IMF estimates that 40% of jobs globally, or 60% of jobs in advanced economies, will be impacted by the growing adoption of AI. How will it impact your role? In my opinion the surest way to secure your place in these transformed economies will be by including AI in the toolset you use to stay competitive.

Tags
Categories: FLOSS Project Planets

Real Python: Python Exceptions: An Introduction

Planet Python - Mon, 2024-01-29 09:00

A Python program terminates as soon as it encounters an error. In Python, an error can be a syntax error or an exception. In this tutorial, you’ll see what an exception is and how it differs from a syntax error. After that, you’ll learn about raising exceptions and making assertions. Then, you’ll get to know all the exception-related keywords that you can use in a try … except block to fine-tune how you can work with Python exceptions.

In this tutorial, you’ll learn how to:

  • Raise an exception in Python with raise
  • Debug and test your code with assert
  • Handle exceptions with try and except
  • Fine-tune your exception handling with else and finally

You’ll get to know these keywords by walking through a practical example of handling a platform-related exception. Finally, you’ll also learn how to create your own custom Python exceptions.

Get Your Code: Click here to download the free sample code that shows you how exceptions work in Python.

Take the Quiz: Test your knowledge with our interactive “Python Exceptions: An Introduction” quiz. Upon completion you will receive a score so you can track your learning progress over time:

Take the Quiz »

Understanding Exceptions and Syntax Errors

Syntax errors occur when the parser detects an incorrect statement. Observe the following example:

Python Traceback >>> print(0 / 0)) File "<stdin>", line 1 print(0 / 0)) ^ SyntaxError: unmatched ')' Copied!

The arrow indicates where the parser ran into the syntax error. Additionally, the error message gives you a hint about what went wrong. In this example, there was one bracket too many. Remove it and run your code again:

Python >>> print(0 / 0) Traceback (most recent call last): File "<stdin>", line 1, in <module> ZeroDivisionError: division by zero Copied!

This time, you ran into an exception error. This type of error occurs whenever syntactically correct Python code results in an error. The last line of the message indicates what type of exception error you ran into.

Instead of just writing exception error, Python details what type of exception error it encountered. In this case, it was a ZeroDivisionError. Python comes with various built-in exceptions as well as the possibility to create user-defined exceptions.

Raising an Exception in Python

There are scenarios where you might want to stop your program by raising an exception if a condition occurs. You can do this with the raise keyword:

You can even complement the statement with a custom message. Assume that you’re writing a tiny toy program that expects only numbers up to 5. You can raise an error when an unwanted condition occurs:

Python low.py number = 10 if number > 5: raise Exception(f"The number should not exceed 5. ({number=})") print(number) Copied!

In this example, you raised an Exception object and passed it an informative custom message. You built the message using an f-string and a self-documenting expression.

When you run low.py, you’ll get the following output:

Python Traceback Traceback (most recent call last): File "./low.py", line 3, in <module> raise Exception(f"The number should not exceed 5. ({number=})") Exception: The number should not exceed 5. (number=10) Copied!

The program comes to a halt and displays the exception to your terminal or REPL, offering you helpful clues about what went wrong. Note that the final call to print() never executed, because Python raised the exception before it got to that line of code.

With the raise keyword, you can raise any exception object in Python and stop your program when an unwanted condition occurs.

Debugging During Development With assert

Before moving on to the most common way of working with exceptions in Python using the try … except block, you’ll take a quick look at an exception that’s a bit different than the others.

Read the full article at https://realpython.com/python-exceptions/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

The Drop Times: Drupal's Future Excites and Warrants Consideration: Boyan Borisov

Planet Drupal - Mon, 2024-01-29 08:59
Dive into an illuminating conversation with Boyan Borisov, VP of Digital Solutions Europe at FFW, as he shares his journey as a Drupal enthusiast and his experiences with digital transformation. Discover his insights on Drupal's roadmap, transitioning to a unified multi-brand ecosystem, and the remarkable evolution of the Layout Builder.
Categories: FLOSS Project Planets

PyCharm: PyCharm 2024.1 EAP 2 Is Out!

Planet Python - Mon, 2024-01-29 08:58

The second EAP build of PyCharm 2024.1 has landed and is now available for you to explore.  

You can download this new build from our website, through the free Toolbox App, or via snaps for Ubuntu.

Download PyCharm 2024.1 EAP

This new build introduces a reworked Terminal tool window and brings an ability to run or debug both client and server in a single npm configuration. Take a look!

Revamped Terminal tool window

PyCharm 2024.1 EAP 2 brings an overhauled terminal with both visual and functional enhancements to make terminal-based tasks simpler and more convenient. This update both improves the tool visually and expands its feature set.

The new Terminal tool window seamlessly integrates with the new UI, aligning it with the IDE’s refreshed look-and-feel, and it comes complete with a new color scheme that enhances readability.

One standout improvement is the presentation of each command in a distinct block. This makes it easy to identify the start and end of each one, enhancing the overall clarity of output. Easily navigate between blocks using the arrow keys or switch the focus between the prompt and output with the ⌘↑ / Ctrl + ↑ and  ⌘↓ / Ctrl + ↓ keyboard shortcuts.

We introduced a convenient command history with filtering options, making navigation through recently executed commands a breeze.

The updated terminal supports only Bash, Zsh, and PowerShell (currently only for Windows 11). We are actively working on supporting more shell integrations.

Run or debug both client and server in a single npm configuration

You can now use the same npm configuration to run and debug both the server and client sides of your application. The new Browser / Live Edit tab in the Run/Debug Configurations editor allows you to set up a run configuration to open a browser automatically after launch. If needed, you can attach the debugger to the opened browser right away.

These are the most notable updates for this week. For the full list of the implemented changes, please consult the release notes. 

Try out these new features and let us know what you think in the comments below or on X (formerly Twitter). If you encounter any bugs, please report them via our issue tracker.

Happy developing!

Categories: FLOSS Project Planets

LN Webworks: Top 9 Essential Drupal Caching Tips For Faster Website Performance

Planet Drupal - Mon, 2024-01-29 07:07

Ever found yourself wondering, 'Why is my Drupal site crawling at a snail's pace?' or contemplating ways to turbocharge your Drupal website performance? Essentially, what are the most efficient, budget-friendly, and quickest solutions to tackle those performance bottlenecks that are hindering your website's speed?

Fortunately, if your website is powered by Drupal 8 (and we assume it is), you're in for a treat with one of the most sophisticated caching systems available. The key lies in embracing Drupal caching best practices and leveraging its full potential.

Top 9 Essential Drupal Caching Tips

Let’s explore the top 9 Drupal caching best practices that promise to be the game-changer for your page load time. 

Categories: FLOSS Project Planets

Brian Okken: pytest 8 is here

Planet Python - Mon, 2024-01-29 07:00
pytest 8.0.0 was released on 17-Jan-2024, and I’m pretty excited about it. I’m not going to cover all fo the changes, I’ll just highlight a few. For full set of changes, see the pytest changelog: Changes in 8.0.0rc1 Changes in 8.0.0rc2 Changes in 8.0.0 Version Compatibility Dropped support for Python 3.7, as it reached EOL last June. Features and Improvements Improved Diffs Improved diffs that pytest prints when an assertion fails, including:
Categories: FLOSS Project Planets

Russell Coker: Thinkpad X1 Yoga Gen3

Planet Debian - Mon, 2024-01-29 06:23

I just bought myself a Thinkpad X1 Yoga Gen3 for $359.10. I have been quite happy with the Thinkpad X1 Carbon Gen5 I’ve had for just over a year (apart from my mistake in buying one with lost password) [1] and I normally try to get more use out of a computer than that. If I divide total cost by the time that I’ve had it working that comes out to about $1.30 per day. I would pay more than that for a laptop and I have paid much more than that for laptops in the past, but I prefer not to. I was initially tempted to buy a new Thinkpad by the prices of high end X1 devices dropping, this new Yoga has 16G of RAM and a 2560*1440 screen – that’s a good upgrade from 8G with 1920*1080. The CPU of my new Thinkpad is a quad core i5-8350U that rates 6226 [2] and is a decent upgrade from the dual core i5-6300U that rates 3239 [3] although that wasn’t a factor as I found the old CPU fast enough.

The Yoga Gen3 has a minimum weight of 1.4Kg and mine might not be the lightest model in the range while the old Carbon weighs 1.14Kg. I can really feel the difference. It’s also slightly larger but fortunately still fits in the pocket of my Scottware jacket.

The higher resolution screen and more RAM were not sufficient to make me want to spend some money. The deciding factor is that as I’m working on phones with touch screens it is a benefit to use a laptop with a touch screen so I can do more testing. The Yoga I bought was going cheap because the touch part of the touch screen is broken but the stylus still works, this is apparently a common failure mode of the Yoga.

The Yoga has a brighter screen than the Carbon and seems to have better contrast. I think Lenovo had some newer technology for that generation of laptops or maybe my Carbon is slightly defective in that regard. It’s a hazard of buying second hand that if something basically works but isn’t quite as good as it should be then you will never know.

I’m happy with this purchase and I recommend that everyone who buys laptops secondhand the way I do only get 1440p or better displays. I’ve currently got the Kitty terminal emulator [4] setup with 9 windows that each have 103 or 104 columns and 26 or 28 rows of text. That’s a lot of terminals on a laptop screen!

Related posts:

  1. Thinkpad X1 Carbon Gen5 Gen1 Since February 2018 I have been using a Thinkpad...
  2. USB-PD and GaN A recent development is cheap Gallium Nitride based power...
  3. More About the Thinkpad X301 Last month I blogged about the Thinkpad X301 I got...
Categories: FLOSS Project Planets

Kyle Einecker|True Summit: Introducing the Search Web Components Module: A Better Way to Build Search Experiences

Planet Drupal - Mon, 2024-01-29 05:21
<p>Since Drupal 8 was released in 2015, I've launched my fair share of sites, and one common feature of those sites was a search page. Working mainly with healthcare and e-commerce businesses during that time, the sites I launched often required not just a site search but a catalog search, a find a location search, or a find a doctor search. With those additional searches came designs and requirements that I needed to convince Views, Search API, and Facets to do. Search API is great, Views is great, Facets is great, but combined, they've never worked as expected for me. Every time I build one of those experiences, it's a fight to meet the expectations of site owners. And I'm not alone. When you go to the Facets issue queue, you'll find many issues and comments reporting issues with facets.</p><p>Today, I am bundling up all my hard won search experience to introduce an easier, better way to build search experiences with Drupal. Search Web Components will take a whole class of search challenges and make it something Drupal just does instead of being a complex and time consuming part of a project.</p>
Categories: FLOSS Project Planets

Doug Hellmann: virtualenvwrapper 6.1.0 - hook scripts in project directories

Planet Python - Mon, 2024-01-29 03:48
What’s new in 6.1.0? source project-dir/.virtualenvwrapper/predeactivate when deactivating source project_dir/.virtualenvwrapper/postactivate during activation
Categories: FLOSS Project Planets

Russ Allbery: Review: Bluebird

Planet Debian - Sun, 2024-01-28 21:20

Review: Bluebird, by Ciel Pierlot

Publisher: Angry Robot Copyright: 2022 ISBN: 0-85766-967-2 Format: Kindle Pages: 458

Bluebird is a stand-alone far-future science fiction adventure.

Ten thousand years ago, a star fell into the galaxy carrying three factions of humanity. The Ascetics, the Ossuary, and the Pyrites each believe that only their god survived and the other two factions are heretics. Between them, they have conquered the rest of the galaxy and its non-human species. The only thing the factions hate worse than each other are those who attempt to stay outside the faction system.

Rig used to be a Pyrite weapon designer before she set fire to her office and escaped with her greatest invention. Now she's a Nightbird, a member of an outlaw band that tries to help refugees and protect her fellow Kashrini against Pyrite genocide. On her side, she has her girlfriend, an Ascetic librarian; her ship, Bluebird; and her guns, Panache and Pizzazz. And now, perhaps, the mysterious Ginka, a Zazra empath and remarkably capable fighter who helps Rig escape from an ambush by Pyrite soldiers.

Rig wants to stay alive, help her people, and defy the factions. Pyrite wants Rig's secrets and, as leverage, has her sister. What Ginka wants is not entirely clear even to Ginka.

This book is absurd, but I still had fun with it.

It's dangerous for me to compare things to anime given how little anime that I've watched, but Bluebird had that vibe for me: anime, or maybe Japanese RPGs or superhero comics. The storytelling is very visual, combat-oriented, and not particularly realistic. Rig is a pistol sharpshooter and Ginka is the type of undefined deadly acrobatic fighter so often seen in that type of media. In addition to her ship, Rig has a gorgeous hand-maintained racing hoverbike with a beautiful paint job. It's that sort of book.

It's also the sort of book where the characters obey cinematic logic designed to maximize dramatic physical confrontations, even if their actions make no logical sense. There is no facial recognition or screening, and it's bizarrely easy for the protagonists to end up in same physical location as high-up bad guys. One of the weapon systems that's critical to the plot makes no sense whatsoever. At critical moments, the bad guys behave more like final bosses in a video game, picking up weapons to deal with the protagonists directly instead of using their supposedly vast armies of agents. There is supposedly a whole galaxy full of civilizations with capital worlds covered in planet-spanning cities, but politics barely exist and the faction leaders get directly involved in the plot.

If you are looking for a realistic projection of technology or society, I cannot stress enough that this is not the book that you're looking for. You probably figured that out when I mentioned ten thousand years of war, but that will only be the beginning of the suspension of disbelief problems. You need to turn off your brain and enjoy the action sequences and melodrama.

I'm normally good at that, and I admit I still struggled because the plot logic is such a mismatch with the typical novels I read. There are several points where the characters do something that seems so monumentally dumb that I was sure Pierlot was setting them up for a fall, and then I got wrong-footed because their plan worked fine, or exploded for unrelated reasons. I think this type of story, heavy on dramatic eye-candy and emotional moments with swelling soundtracks, is a lot easier to pull off in visual media where all the pretty pictures distract your brain. In a novel, there's a lot of time to think about the strategy, technology, and government structure, which for this book is not a good idea.

If you can get past that, though, Rig is entertainingly snarky and Ginka, who turns out to be the emotional heart of the book, is an enjoyable character with a real growth arc. Her background is a bit simplistic and the villains are the sort of pure evil that you might expect from this type of cinematic plot, but I cared about the outcome of her story. Some parts of the plot dragged and I think the editing could have been tighter, but there was enough competence porn and banter to pull me through.

I would recommend Bluebird only cautiously, since you're going to need to turn off large portions of your brain and be in the right mood for nonsensically dramatic confrontations, but I don't regret reading it. It's mostly in primary colors and the emotional conflicts are not what anyone would call subtle, but it delivers a character arc and a somewhat satisfying ending.

Content warning: There is a lot of serious physical injury in this book, including surgical maiming. If that's going to bother you, you may want to give this one a pass.

Rating: 6 out of 10

Categories: FLOSS Project Planets

Gushing about KDE applications

Planet KDE - Sun, 2024-01-28 19:00

This a lazy and anti-rant post… I want to shine a light on the fantastic KDE software that I use daily. You can do similar things with GNOME and whatever else, but that’s for someone else to write. I have some bias because I have contributed to several of these applications, but that doesn’t detract from the point that I depend on them daily.

Screenshot of KMail from kde.org

I check my work and personal mail using KMail. I’m one of those lucky few that checks my mail from two IMAP-compliant servers, so I steer clear from Outlook/GMail. I keep track of tasks, events and meetings using Merkuro. I can keep tabs on my calendar since the time applet is synced thanks to Akonadi. I really enjoy and use the integration between these Akonadi applications, such as accepting invitations to meetings which are automatically recorded into my calendar.

My work uses Rocket.Chat, and I use Ruqola for interacting with that:

Screenshot of Ruqola from kde.org

Even when not working, I still use KDE software! One of them is drawing, and I use Krita for that (which is a great application in general, you should use it!) It’s completely replaced Procreate and Clip Studio Paint which I used before. I really like it’s integrated brush engines and default brush set, along with all of it’s built-in functionality like animation support. I even use Krita when sketching on-the-go or in bed now instead of Procreate, since my Lenovo Yoga runs Linux and KDE Plasma. When I edit videos, my program of choice is Kdenlive (which is a great application in general, you should use it!) It does everything I want it to do, honestly I have very little trouble with it but my needs are minimal.

Screenshot of Kdenlive from kde.org

My primary chat platform is Matrix, so of course I use NeoChat as my preferred client everywhere I can. I chose Mastodon as my Twitter-replacement, and I use Tokodon so much that I don’t even open up their web interface anymore! The less I have to run in the browser the better, in my opinion.

Screenshot of Tokodon from kde.org

There’s also lots of small utilities that I use, such as Spectacle for screenshots and quick screen recordings. I use Kate for all of my text editing, such as this post right now! Gwenview is my image viewer of choice too. Of course I use Okular for reading the occasional PDF. Can’t forget about Okteta when I’m trying to dissect some binary file.

Screenshot of Kasts from kde.org

I even use KDE applications for consuming media, too. I use PlasmaTube to feed my YouTube addiction. I like to put videos in picture-in-picture and even added that to the next release, meaning I can stop using the web interface for Invidious. I have started listening to some podcasts, and have been using Kasts for those. I elarned recently that it even can sync via NextCloud!

Upcoming #

Here’s some software I recently learned about, and want to start using soon:

  • Accessibility Inspector, which is a KDE-based alternative to GNOME’s Accerciser.
  • Codevis, a code visualization program, I used it once before but I need to try it again.
  • Fielding, a REST API client. I plan to expand it’s features further so I don’t have to depend on Insomina.
  • Powerplant, something to help keep your plants. I only have one plant to keep right now, so this is the perfect time to learn how to use it!

Hope this sheds some light on my favorite applications, and gives you ideas for using them! I enjoy how fast these applications are, and how integrated and nice they all look together. I couldn’t go over every single one, but maybe I can expand in the future.

Categories: FLOSS Project Planets

Michael Ablassmeier: qmpbackup 0.28

Planet Debian - Sun, 2024-01-28 19:00

Over the last weekend i had some spare time to improve qmpbackup a little more, the new version:

  • Uses blockdev-backup QMP commands instead of the soon to be deprecated drive-backup.
  • Adds --compress option: target QCOW files data can be compressed.
  • Adds --speed-limit option to limit backup speed throughput.
  • Adds --include-raw option so attached raw devices can be backed up too.
  • Allows backing up virtual machines in paused or pre-running state.
  • Adds backup option copy to allow backing up virtual machines with older qcow image formats, that are not supporting persistent bitmaps.
  • Improved logging output.

and some minor code reworks. Hope its useful for someone.

Categories: FLOSS Project Planets

TestDriven.io: Working with Static and Media Files in Django

Planet Python - Sun, 2024-01-28 17:28
This article looks at how to work with static and media files in a Django project, locally and in production.
Categories: FLOSS Project Planets

TechBeamers Python: IndexError: List Index Out of Range in Python

Planet Python - Sun, 2024-01-28 13:39

An “IndexError: list index out of range” in Python typically occurs when you’re trying to access an index in a list that does not exist. This typically happens when attempting to access an index that is beyond the bounds of the list. In this guide, we’ll explore the causes of this error and discuss various […]

The post IndexError: List Index Out of Range in Python appeared first on TechBeamers.

Categories: FLOSS Project Planets

Niels Thykier: Annotating the Debian packaging directory

Planet Debian - Sun, 2024-01-28 11:45

In my previous blog post Providing online reference documentation for debputy, I made a point about how debhelper documentation was suboptimal on account of being static rather than online. The thing is that debhelper is not alone in this problem space, even if it is a major contributor to the number of packaging files you have to to know about.

If we look at the "competition" here such as Fedora and Arch Linux, they tend to only have one packaging file. While most Debian people will tell you a long list of cons about having one packaging file (such a Fedora's spec file being 3+ domain specific languages "mashed" into one file), one major advantage is that there is only "the one packaging file". You only need to remember where to find the documentation for one file, which is great when you are running on wetware with limited storage capacity.

Which means as a newbie, you can dedicate less mental resources to tracking multiple files and how they interact and more effort understanding the "one file" at hand. I started by asking myself how can we in Debian make the packaging stack more accessible to newcomers? Spoiler alert, I dug myself into rabbit hole and ended up somewhere else than where I thought I was going.

I started by wanting to scan the debian directory and annotate all files that I could with documentation links. The logic was that if debputy could do that for you, then you could spend more mental effort elsewhere. So I combined debputy's packager provided files detection with a static list of files and I quickly had a good starting point for debputy-based packages.

Adding (non-static) dpkg and debhelper files to the mix

Now, I could have closed the topic here and said "Look, I did debputy files plus couple of super common files". But I decided to take it a bit further. I added support for handling some dpkg files like packager provided files (such as debian/substvars and debian/symbols). But even then, we all know that debhelper is the big hurdle and a major part of the omission...

In another previous blog post (A new Debian package helper: debputy), I made a point about how debputy could list all auxiliary files while debhelper could not. This was exactly the kind of feature that I would need for this feature, if this feature was to cover debhelper. Now, I also remarked in that blog post that I was not willing to maintain such a list. Also, I may have ranted about static documentation being unhelpful for debhelper as it excludes third-party provided tooling.

Fortunately, a recent update to dh_assistant had provided some basic plumbing for loading dh sequences. This meant that getting a list of all relevant commands for a source package was a lot easier than it used to be. Once you have a list of commands, it would be possible to check all of them for dh's NOOP PROMISE hints. In these hints, a command can assert it does nothing if a given pkgfile is not present. This lead to the new dh_assistant list-guessed-dh-config-files command that will list all declared pkgfiles and which helpers listed them.

With this combined feature set in place, debputy could call dh_assistant to get a list of pkgfiles, pretend they were packager provided files and annotate those along with manpage for the relevant debhelper command. The exciting thing about letting debpputy resolve the pkgfiles is that debputy will resolve "named" files automatically (debhelper tools will only do so when --name is passed), so it is much more likely to detect named pkgfiles correctly too. Side note: I am going to ignore the elephant in the room for now, which is dh_installsystemd and its package@.service files and the wide-spread use of debian/foo.service where there is no package called foo. For the latter case, the "proper" name would be debian/pkg.foo.service.

With the new dh_assistant feature done and added to debputy, debputy could now detect the ubiquitous debian/install file. Excellent. But less great was that the very common debian/docs file was not. Turns out that dh_installdocs cannot be skipped by dh, so it cannot have NOOP PROMISE hints. Meh...

Well, dh_assistant could learn about a new INTROSPECTABLE marker in addition to the NOOP PROMISE and then I could sprinkle that into a few commands. Indeed that worked and meant that debian/postinst (etc.) are now also detectable.

At this point, debputy would be able to identify a wide range of debhelper related configuration files in debian/ and at least associate each of them with one or more commands.

Nice, surely, this would be a good place to stop, right...?

Adding more metadata to the files

The debhelper detected files only had a command name and manpage URI to that command. It would be nice if we could contextualize this a bit more.

Like is this file installed into the package as is like debian/pam or is it a file list to be processed like debian/install. To make this distinction, I could add the most common debhelper file types to my static list and then merge the result together.

Except, I do not want to maintain a full list in debputy. Fortunately, debputy has a quite extensible plugin infrastructure, so added a new plugin feature to provide this kind of detail and now I can outsource the problem! I split my definitions into two and placed the generic ones in the debputy-documentation plugin and moved the debhelper related ones to debhelper-documentation. Additionally, third-party dh addons could provide their own debputy plugin to add context to their configuration files.

So, this gave birth file categories and configuration features, which described each file on different fronts. As an example, debian/gbp.conf could be tagged as a maint-config to signal that it is not directly related to the package build but more of a tool or style preference file. On the other hand, debian/install and debian/debputy.manifest would both be tagged as a pkg-helper-config. Files like debian/pam were tagged as ppf-file for packager provided file and so on.

I mentioned configuration features above and those were added because, I have had a beef with debhelper's "standard" configuration file format as read by filearray and filedoublearray. They are often considered simple to understand, but it is hard to know how a tool will actually read the file. As an example, consider the following:

  • Will the debhelper use filearray, filedoublearray or none of them to read the file? This topic has about 2 bits of entropy.
  • Will the config file be executed if it is marked executable assuming you are using the right compat level? If it is executable, does dh-exec allow renaming for this file? This topic adds 1 or 2 bit of entropy depending on the context.
  • Will the config file be subject to glob expansions? This topic sounds like a boolean but is a complicated mess. The globs can be handled either by debhelper as it parses the file for you. In this case, the globs are applied to every token. However, this is not what dh_install does. Here the last token on each line is supposed to be a directory and therefore not subject to globs. Therefore, dh_install does the globbing itself afterwards but only on part of the tokens. So that is about 2 bits of entropy more. Actually, it gets worse...
    • If the file is executed, debhelper will refuse to expand globs in the output of the command, which was a deliberate design choice by the original debhelper maintainer took when he introduced the feature in debhelper/8.9.12. Except, dh_install feature interacts with the design choice and does enable glob expansion in the tool output, because it does so manually after its filedoublearray call.

So these "simple" files have way too many combinations of how they can be interpreted. I figured it would be helpful if debputy could highlight these difference, so I added support for those as well.

Accordingly, debian/install is tagged with multiple tags including dh-executable-config and dh-glob-after-execute. Then, I added a datatable of these tags, so it would be easy for people to look up what they meant.

Ok, this seems like a closed deal, right...?

Context, context, context

However, the dh-executable-config tag among other are only applicable in compat 9 or later. It does not seem newbie friendly if you are told that this feature exist, but then have to read in the extended description that that it actually does not apply to your package.

This problem seems fixable. Thanks to dh_assistant, it is easy to figure out which compat level the package is using. Then tweak some metadata to enable per compat level rules. With that tags like dh-executable-config only appears for packages using compat 9 or later.

Also, debputy should be able to tell you where packager provided files like debian/pam are installed. We already have the logic for packager provided files that debputy supports and I am already using debputy engine for detecting the files. If only the plugin provided metadata gave me the install pattern, debputy would be able tell you where this file goes in the package. Indeed, a bit of tweaking later and setting install-pattern to usr/lib/pam.d/{name}, debputy presented me with the correct install-path with the package name placing the {name} placeholder.

Now, I have been using debian/pam as an example, because debian/pam is installed into usr/lib/pam.d in compat 14. But in earlier compat levels, it was installed into etc/pam.d. Well, I already had an infrastructure for doing compat file tags. Off we go to add install-pattern to the complat level infrastructure and now changing the compat level would change the path. Great. (Bug warning: The value is off-by-one in the current version of debhelper. This is fixed in git)

Also, while we are in this install-pattern business, a number of debhelper config files causes files to be installed into a fixed directory. Like debian/docs which causes file to be installed into /usr/share/docs/{package}. Surely, we can expand that as well and provide that bit of context too... and done. (Bug warning: The code currently does not account for the main documentation package context)

It is rather common pattern for people to do debian/foo.in files, because they want to custom generation of debian/foo. Which means if you have debian/foo you get "Oh, let me tell you about debian/foo ". Then you rename it to debian/foo.in and the result is "debian/foo.in is a total mystery to me!". That is suboptimal, so lets detect those as well as if they were the original file but add a tag saying that they are a generate template and which file we suspect it generates.

Finally, if you use debputy, almost all of the standard debhelper commands are removed from the sequence, since debputy replaces them. It would be weird if these commands still contributed configuration files when they are not actually going to be invoked. This mostly happened naturally due to the way the underlying dh_assistant command works. However, any file mentioned by the debhelper-documentation plugin would still appear unfortunately. So off I went to filter the list of known configuration files against which dh_ commands that dh_assistant thought would be used for this package.

Wrapping it up

I was several layers into this and had to dig myself out. I have ended up with a lot of data and metadata. But it was quite difficult for me to arrange the output in a user friendly manner.

However, all this data did seem like it would be useful any tool that wants to understand more about the package. So to get out of the rabbit hole, I for now wrapped all of this into JSON and now we have a debputy tool-support annotate-debian-directory command that might be useful for other tools.

To try it out, you can try the following demo:

In another day, I will figure out how to structure this output so it is useful for non-machine consumers. Suggestions are welcome. :)

Limitations of the approach

As a closing remark, I should probably remind people that this feature relies heavily on declarative features. These include:

  • When determining which commands are relevant, using Build-Depends: dh-sequence-foo is much more reliable than configuring it via the Turing complete configuration we call debian/rules.
  • When debhelper commands use NOOP promise hints, dh_assistant can "see" the config files listed those hints, meaning the file will at least be detected. For new introspectable hint and the debputy plugin, it is probably better to wait until the dust settles a bit before adding any of those.

You can help yourself and others to better results by using the declarative way rather than using debian/rules, which is the bane of all introspection!

Categories: FLOSS Project Planets

Pages