FLOSS Project Planets
Drupal Association blog: Navigating the Future 6 Months out from Drupal 7 Support Ending: Options for Your Drupal 7 Site
With Drupal 7’s (D7) end-of-life (EOL) in 6 months on January 5, 2025, organizations relying on D7 face critical decisions regarding the future of their websites. This article will help guide you through the paths you can take: migrating to modern Drupal, leveraging extended long-term support options, or staying on unsupported Drupal 7.
Update to Modern DrupalTransitioning from Drupal 7 to a newer version is crucial in future-proofing your digital presence. These versions embrace modern PHP standards, object-oriented programming, and Symfony components, providing a powerful foundation for your website. This upgrade allows you to access advanced features, enhanced performance, and ensures ongoing support and security updates.
Why Migrate to the Latest Versions?Modernization: Drupal 10 offers cutting-edge features and performance improvements, and an easy upgrade path to Drupal 11, releasing very soon.
Security: Continuous security updates protect your site from vulnerabilities.
Flexibility: Adopt contemporary coding standards and best practices.
Future-Proofing: Ensure compatibility with future updates and maintain a seamless digital experience.
Additionally, the upcoming release of Starshot, slated before the end of 2024, promises even more enhancements and features that will elevate your website's capabilities. By migrating now, your organization can seamlessly integrate these future advancements.
Migrating to newer versions can involve navigating significant architectural changes, and may require extensive modifications to custom modules and themes. However, tools like Drupal Rector and Retrofit on our DIY migration resources page can help make this process easier. The benefits of modernization, enhanced security, and future-proofing outweigh the initial investment in time, resources, and budget.
But you don’t have to do it yourself. There are a number of Drupal Certified Partners who can assist organizations in planning and implementing their migration.
Find the qualified company that is best for you: Certified Migration Partners.
Extended Security Support for Drupal 7To address the challenges of using unsupported software, the Drupal Association has established a program for supporting site owners who won't be able to migrate before the end of life date. The D7 Extended Security Support Program identifies existing Drupal Certified Partners who meet stringent standards and who the Drupal Association feels confident recommending.
With the end of support, the Drupal Security Team will no longer be involved in supporting Drupal 7. The Drupal Association recognizes that some site owners will not be in a position to migrate their site or need more time to do so. For many of these site owners, paying for extended support would be a good option.
Recognizing that the Drupal Security Team would not be officially involved in any such service, the Drupal Association created rigorous standards before certifying companies under this program. Some of these requirements include:
- Being a Drupal Certified Partner at the Gold tier or higher
- Employing a core security team member
- Experience in providing security and compatibility fixes
- History of reporting 2 or more CVEs and creating fixes for the same
- Willingness to enter in a service level agreement to ensure standards are being met
Find the company that will work best for you: D7 Extended Security Support Partners
Stay on Unsupported Drupal 7When Drupal 7 reaches its EOL, it will no longer receive new security updates, fixes, or official support from the Drupal community. While this option might seem cost-effective and leverages your team's stability and familiarity with Drupal 7, it comes with significant risks.
Without updates, your site will be vulnerable to new security exploits and non-compliance with standards such as FedRAMP, PCI-DSS, and HIPAA. Over time, tools and utilities supporting your Drupal 7 site may become incompatible with new versions of dependencies like PHP, and finding developers skilled in outdated technology could become increasingly difficult.
The Drupal Association does not recommend this option.
ConclusionOrganizations must carefully weigh their options as Drupal 7 approaches its EOL to ensure continued security, compliance, and compatibility. Embracing Drupal 10 and the upcoming Drupal 11 and Starshot release will position your organization for long-term success with access to the latest features, security updates, and a vibrant support community..
Greg Casamento: What Apple has forgotten...
When NeXT still existed and the black hardware was a thing, Steve Jobs made the announcement that OPENSTEP would be created and that the object model, not the operating system and not the hardware, was the important thing.
This is a concept that Apple has forgotten. With it's push towards Apple Silicon and a walled-garden, Apple has committed itself to the same pitfall that NeXT fell into. NeXT lacked the infrastructure to handle OPENSTEP running on multiple kinds of hardware, but the object model on different OSes was successful... this is evident in OPENSTEP1.1 for Solaris and OPENSTEP for NT.
GNUstep attempts to reach the same goal, but provides the APIs that are available with Cocoa. The object model IS the important thing and this is why GNUstep is so important. It breaks the walled garden and makes it possible for users to run their apps and tools on other operating systems. GNUstep HASN'T forgotten and we believe this is a core concept that Apple has left behind.
Samuel Henrique: Debian's curl now supports HTTP3
Starting with curl 8.0.0-2, you can now use HTTP3.
curl --http3-only https://example.comOr, if you would like to try it out in a container:
podman run debian:unstable apt install --update -y curl && curl --http3-only https://example.com(in case you haven't noticed, apt now has the --update option for the upgrade and install commands, although not available on stable yet)
Availability- Debian unstable - Since 2024-07-02
- Debian testing - Coming up between the second and third week of July 2024 (estimate if no issues are spotted).
- Debian 12/bookworm backports - As soon as the package gets to Debian testing, I'll upload it to bookworm.
- Debian 12/bookworm - Due to the mechanisms we have in place to make sure Debian stable is in fact stable, we will never be able to ship this in the regular repository. Users can make use of the backports repositories instead.
- Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.
HTTP3 is fresh new, well... not really, but at least fresh enough that I'm not aware of any other Linux distribution supporting it on curl, the reason is likely two-fold:
- OpenSSL is not there yet
OpenSSL still doesn't have proper HTTP3 support, and given that OpenSSL is so widely used, almost every curl distributor/packager will build curl with it and thus changing the TLS backend to something else is risky.
Unfortunately, proper support for the OpenSSL libcurl is unlikely to come anytime before the end of this year, the OpenSSL performance is not good enough yet as of version 3.3.
Daniel Stenberg has written about the state of this multiple times, most recently at HTTP/3 in curl mid 2024, if you're interested, I suggest reading through his other posts as well.
Some might have noticed that nginx does support HTTP3 through OpenSSL, although when you look closely, it's not exactly perfect:
An SSL library that provides QUIC support is recommended to build nginx, such as BoringSSL, LibreSSL, or QuicTLS. Otherwise, the OpenSSL compatibility layer will be used that does not support early data.
As you can see, they don't recommend using OpenSSL, and when doing so, you don't get complete support.
- HTTP3 support for GnuTLS/nghttp3/ngtcp2 is recent
The non-experimental support arrived back in October 2023, and so that's when I started seriously planning for this.
curl has been working on HTTP3 support for years, and so it did support other TLS backends before that, but out of them, the one most feasible for a distribution to ship would be GnuTLS, which gets HTTP3 support through ngctp2 and nghttp3.
The Debian curl package has historically shipped at least two variants of libcurl, an OpenSSL and a GnuTLS one.
The OpenSSL libcurl can't support HTTP3 for the reasons explained above, but the GnuTLS libcurl can (with ngtcp2 and nghtp3).
Debian packages can choose which version of libcurl to link against (without having to modify any upstream source code). Debian's "git" package being a famous example of a package that links against the GnuTLS libcurl.
Enabling HTTP3 on curl was done in three steps:
- Make sure all required dependencies fulfill the minimum requirements.
- Enable HTTP3 for GnuTLS libcurl.
- Change the libcurl used by the curl CLI, from OpenSSL to GnuTLS.
curl's HTTP3 support requires a somewhat recent version of nghttp3 and updating that required a transition (due to the SONAME bump), while we've also had months of freeze for transitions due to the time_t transition.
After the dependencies were in place, enabling HTTP3 for the GnuTLS libcurl was straightforward.
Then, for the last part, we had to switch the TLS backend used by the curl CLI. Doing the swap is also quite easy on the packaging level, but we have to consider the chances of this change breaking our users' environments.
Ensuring there are no breakagesThe first thing to consider regarding breakages is that this change is not going to be pushed directly to the current Debian stable releases, it will be present in the next stable release (13/trixie) but the current one will stick to the version that's already shipped.
Secondly, we have to consider the risk of losing the ability to use certain parameters from the curl CLI which could be limited to the OpenSSL backend. During curl-up 2024, the curl developers pointed out the existence of a page that lists the TLS related options and the backends they work with.
Analysing that page, ignoring all of the options that are suffixed with "BLOB" (only pertinent to the library, not the CLI), the only one left which is attention worthy is CURLOPT_ECH.
This experimental feature requires a special build of OpenSSL, as ECH is not yet supported in OpenSSL releases. In contrast ECH is supported by the latest BoringSSL and wolfSSL releases.
As it turns out, Encrypted Client Hello is experimental and it's not supported by the vanilla OpenSSL.
This was enough of an investigation for me to go ahead with the change. Noting that even in the worst case scenario (we find a horrible regression), we can rollback without having affected a single stable release.
Now that the package is on Debian unstable, the CI tests (autopkgtest) of every package that depends on curl is currently running, the results are compared against the migration-reference (in this case, the curl CLI with OpenSSL, before the change).
If everything goes right, curl with HTTP3 support will migrate to Debian testing in around 5 days. If we spot any issues, we'll have to solve them first and it's going to be hard to predict how long it takes, although it's fair to expect less than a month.
FeedbackFeel free to join the Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org
We have historically spoken Portuguese in the room but we'll switch to English
in case anyone joins.
It took us a bit longer than expected to be able to enable HTTP3, nonetheless it's still early enough to be excited about.
A lot of people were crucial to make this happen.
I should recognize in the first place, obviously, the curl developers and the developers of the supporting libraries: GnuTLS, nghttp3, ngtcp2. Participating in the curl-up 2024 conference helped me get motivated to push this through, besides becoming aware of the right documentation to research for impact.
On the Debian side, Sakirnth Nagarasa <sakirnth> was responsible for updating and taking care of the transition for nghttp3 and ngtcp2.
Also on the Debian side, I've got loads of help and support from the co-maintainers of the curl package: Sergio Durigan Junior <sergiodj> and Carlos Henrique Lima Melara <charles>.
Mike Gabriel: Polis - a FLOSS Tool for Civic Participation -- Initial Evaluation and Adaptation (episode 2/5)
Here comes the 2nd article of the 5-episode blog post series written by Guido Berhörster, member of staff at my company Fre(i)e Software GmbH.
Enjoy also this read on Guido's work on Polis,
Mike
- Introduction
- Initial evaluation and adaptation (this article)
- Issues extending Polis and adjusting our goals
- Creating (a) new frontend(s) for Polis
- Current status and roadmap
The Polis code base consists of a number of components, the administration and participation interfaces, a common web backend, and a statistics processing server. Both frontends and the backend are written in a mixture of JavaScript and TypeScript, only the statistics processing server is written in Clojure.
In case of self hosting the preferred method of deployment is via Docker containers using Docker Compose or any other orchestrator. The participation frontend for conversations can either be used as a standalone web page or be embedded via an iframe.
For our planned use case we initially defined the following goals:
- custom branding and the integration into different content management systems (CMS)
- better support for mobile devices
- mandatory authentication and support for a broader range of authentication methods, including self-hosted solutions and DigiD
- support for alternative email sending services
- GDPR compliance
After a preliminary evaluation of our own and consulting with Policy Lab UK who were also evaluating and testing Polis and had already made a range of improvements related to self-hosting as well as bug fixes and modernization changes we decided to take their work as a base for our adaptations with the intent of submitting generally useful changes back to the Polis project.
Subsequently, a number of changes were implemented, including the removal of hardcoded domain names, the elimination of unnecessary cookies and third-party requests, support for an alternative email sending service, and the option of disabling Facebook and X integration.
For the branding our approach was to add an option allowing websites which are embedding conversations in an iframe to load an alternative stylesheet for overriding the native Polis branding. For this to be practical we intended to use CSS custom properties for defining branding-related styles such as colors and fonts. That approach turned out to be problematic because although the Polis participation frontend stylesheet is generated via SCSS and some of the colors are parameterized, however, they are not used consistently throughout the SCSS stylesheets, unfortunately. In addition the frontend templates contain a large amount of hardcoded style attributes. While we succeeded in implementing user-defined stylesheets, it took a disproportionate amount of development resources to parameterize all used colors and fonts via CSS custom properties aggravated by the fact that the SCSS and template files are huge and contain many unused rules and code.
Ian Jackson: derive-deftly is nearing 1.x - call for review/testing
derive-deftly, the template-based derive-macro facility for Rust, has been a great success.
It’s coming up to time to declare a stable 1.x version. If you’d like to try it out, and have final comments / observations, now is the time.
Introduction to derive-deftlyHave you ever wished that you could that could write a new derive macro without having to mess with procedural macros?
You can!
derive-deftly lets you write a #[derive] macro, using a template syntax which looks a lot like macro_rules!:
use derive_deftly::{define_derive_deftly, Deftly}; define_derive_deftly! { ListVariants: impl $ttype { fn list_variants() -> Vec<&'static str> { vec![ $( stringify!( $vname ) , ) ] } } } #[derive(Deftly)] #[derive_deftly(ListVariants)] enum Enum { UnitVariant, StructVariant { a: u8, b: u16 }, TupleVariant(u8, u16), } assert_eq!( Enum::list_variants(), ["UnitVariant", "StructVariant", "TupleVariant"], );Statusderive-deftly has a wide range of features, which can be used to easily write sophisticated and reliable derive macros. We’ve been using it in Arti, the Tor Project’s reimplementation of Tor in Rust, and we’ve found it very useful.
There is comprehensive reference documentation, and more discursive User Guide for a more gentle introduction. Naturally, everything is fully tested.
Historyderive-deftly started out as a Tor Hackweek project. It used to be called derive-adhoc. But we renamed it because we found that many of the most interesting use cases were really not very ad-hoc at all.
Over the past months we’ve been ticking off our “1.0 blocker” tickets. We’ve taken the opportunity to improve syntax, terminology, and semantics. We hope we have now made the last breaking changes.
Plans - call for review/testingIn the near future, we plan to declare version 1.0. After 1.x, we intend to make breaking changes very rarely.
So, right now, we’d like last-minute feedback. Are there any wrinkles that need to be sorted out? Please file tickets or MRs on our gitlab. Ideally, anything which might imply breaking changes would be submitted on or before the 13th of August.
In the medium to long term, we have many ideas for how to make derive-deftly even more convenient, and even more powerful. But we are going to proceed cautiously, because we don’t want to introduce bad syntax or bad features, which will require difficult decisions in the future about forward compatibility.
comments
Drupal Association blog: Introducing Ripple Makers: our revamped Individual Membership program!
Fellow Drupalists!
We have exciting news. As a way to express our appreciation for our community, the Drupal Association is pleased to announce some changes to the Individual Membership program, now called Ripple Makers!
Our goal is to increase communication and transparency, and create more opportunities for engagement.
The Drupal Association is a United States incorporated 501(c)3 nonprofit organization. Donations to Ripple Makers are tax deductible, where applicable. We raise funding primarily through event registration, event sponsorship, the Drupal Certified Partner program, some grants, and individual contributions. We do not have a for profit parent corporation telling us what to do, nor do we ever charge to download Drupal. In fact, it is our mission to ensure that it will always be free.
Drupal has given members of our community opportunities beyond just a career. The values adopted by open source communities create innovation, collaboration, and creation across the globe. Drupal itself is recognized as a Digital Public Good. Drupal is a lever for change and makes a difference preserving the integrity of the internet.
At the heart of the Ripple Makers transformation is a commitment to fostering deeper connections with each and every member of our community, recognizing the invaluable role that sustaining donors play in advancing our shared mission and impact. By revamping our sustaining donors club to be more inclusive and communicative, we aim to create a culture of belonging where every donor feels valued, heard, and empowered to make a difference.
Ripple Makers is your chance to give back—and to celebrate the Drupal community and the open source ecosystem that helps it thrive. We invite you to make a monthly donation in support of the Drupal Association.
Sahil Dhiman: RTI to NPL Regarding Their NTP Infrastructure
I became interested in Network Time Protocol (NTP) last year after learning how fundamental this protocol is to the functioning of the global Internet. NTP helps synchronize clocks on devices over the Internet, which is essential for secure browsing, timestamping, keeping everyone in sync or just checking what time it is. Computers usually have a hardware real-time clock (RTC) but that deviates over time, so an occasional sync over NTP is required to keep the time accurate. Many network and IoT devices don’t have hardware RTC so have even more reliance on NTP.
Accurate time keeping starts with reference clocks like atomic clocks, GPS etc. Multiple government standard agencies host these reference clocks, which are regarded as Stratum 0. Stratum 1 servers are known as primary servers, and directly connect to Stratum 0 clocks for time. Stratum 1 servers then distribute time to Stratum 2 and further down the hierarchy. Computers typically connects to one or more Stratum 1/2/3… servers to get their time.
Someone has to host these public Stratum 1,2,3… NTP servers. That’s what NTP pool, a global effort by volunteers does. They provide NTP servers for the public to use. As of today, there are 4700+ servers in the pool which are free to use for anyone.
Now let’s come to the reason for writing this post. Indian Computer Emergency Response Team (CERT-In) in April 2022 released a set of cybersecurity directions which set the alarm bells ringing. Internet Society (and almost everyone else) wrote about it.
And then there was this specific section about NTP:
All service providers, intermediaries, data centres, body corporate and Government organisations shall connect to the Network Time Protocol (NTP) Server of National Informatics Centre (NIC) or National Physical Laboratory (NPL) or with NTP servers traceable to these NTP servers, for synchronisation of all their ICT systems clocks. Entities having ICT infrastructure spanning multiple geographies may also use accurate and standard time source other than NPL and NIC, however it is to be ensured that their time source shall not deviate from NPL and NIC.
CSIR-National Physical Laboratory (NPL) is the official timekeeper for India and hosts the only public Stratum 1 clock in India, according to NTP pool website. So I was naturally curious to know what kind of infrastructure they’re running for NTP. India has a Right to Information (RTI) Act which, like the Freedom of Information Act (FOIA) in the United States, gives citizens rights to request information from governmental entities, to which they have to respond in under 30 days. So last year, I filed two sets of RTI (one after the first reply came) inquiring about NPL’s public NTP server setup.
The first RTI had some generic questions:
First RTI. Click to enlargeThis gave a vague idea about the setup, so I sat down and came with some specific questions in the next RTI.
Second RTI. Click to enlargeFeel free to make your conclusions from it now. Bear in mind these were filled last year so things might have changed. Do let me know if you have more information about it.
Real Python: Working With JSON Data in Python
Since its introduction, JSON has rapidly emerged as the predominant standard for the exchange of information. Whether you want to transfer data with an API or store information in a document database, it’s likely you’ll encounter JSON. Fortunately, Python provides robust tools to facilitate this process and help you manage JSON data efficiently.
In this tutorial, you’ll learn how to:
- Understand the JSON syntax
- Convert Python data to JSON
- Deserialize JSON to Python
- Write and read JSON files
- Validate JSON syntax
- Prettify JSON in the terminal
- Minify JSON with Python
While JSON is the most common format for data distribution, it’s not the only option for such tasks. Both XML and YAML serve similar purposes. If you’re interested in how the formats differ, then you can check out the tutorial on how to serialize your data with Python.
Free Bonus: Click here to download the free sample code that shows you how to work with JSON data in Python.
Take the Quiz: Test your knowledge with our interactive “Working With JSON Data in Python” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Working With JSON Data in PythonIn this quiz, you'll test your understanding of working with JSON in Python. JSON has become the de facto standard for information exchange, and Python provides easy-to-use tools to handle JSON data.
Introducing JSONThe acronym JSON stands for JavaScript Object Notation. As the name suggests, JSON originated from JavaScript. However, JSON has transcended its origins to become language-agnostic and is now recognized as the standard for data interchange.
The popularity of JSON can be attributed to native support by the JavaScript language, resulting in excellent parsing performance in web browsers. On top of that, JSON’s straightforward syntax allows both humans and computers to read and write JSON data effortlessly.
To get a first impression of JSON, have a look at this example code:
JSON hello_world.json { "greeting": "Hello, world!" } Copied!You’ll learn more about the JSON syntax later in this tutorial. For now, recognize that the JSON format is text-based. In other words, you can create JSON files using the code editor of your choice. Once you set the file extension to .json, most code editors display your JSON data with syntax highlighting out of the box:
The screenshot above shows how VS Code displays JSON data using the Bearded color theme. You’ll have a closer look at the syntax of the JSON format next!
Examining JSON SyntaxIn the previous section, you got a first impression of how JSON data looks. And as a Python developer, the JSON structure probably reminds you of common Python data structures, like a dictionary that contains a string as a key and a value. If you understand the syntax of a dictionary in Python, you already know the general syntax of a JSON object.
Note: Later in this tutorial, you’ll learn that you’re free to use lists and other data types at the top level of a JSON document.
The similarity between Python dictionaries and JSON objects is no surprise. One idea behind establishing JSON as the go-to data interchange format was to make working with JSON as convenient as possible, independently of which programming language you use:
[A collection of key-value pairs and arrays] are universal data structures. Virtually all modern programming languages support them in one form or another. It makes sense that a data format that is interchangeable with programming languages is also based on these structures. (Source)
To explore the JSON syntax further, create a new file named hello_frieda.json and add a more complex JSON structure as the content of the file:
JSON hello_frieda.json 1{ 2 "name": "Frieda", 3 "isDog": true, 4 "hobbies": ["eating", "sleeping", "barking"], 5 "age": 8, 6 "address": { 7 "work": null, 8 "home": ["Berlin", "Germany"] 9 }, 10 "friends": [ 11 { 12 "name": "Philipp", 13 "hobbies": ["eating", "sleeping", "reading"] 14 }, 15 { 16 "name": "Mitch", 17 "hobbies": ["running", "snacking"] 18 } 19 ] 20} Copied!In the code above, you see data about a dog named Frieda, which is formatted as JSON. The top-level value is a JSON object. Just like Python dictionaries, you wrap JSON objects inside curly braces ({}).
In line 1, you start the JSON object with an opening curly brace ({), and then you close the object at the end of line 20 with a closing curly brace (}).
Note: Although whitespace doesn’t matter in JSON, it’s customary for JSON documents to be formatted with two or four spaces to indicate indentation. If the file size of the JSON document is important, then you may consider minifying the JSON file by removing the whitespace. You’ll learn more about minifying JSON data later in the tutorial.
Inside the JSON object, you can define zero, one, or more key-value pairs. If you add multiple key-value pairs, then you must separate them with a comma (,).
Read the full article at https://realpython.com/python-json/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Lullabot: Responsive HTML Tables: Presenting Data in an Accessible Way
Long gone are the days of using HTML tables for page layouts. That time almost seems shrouded in myth. One of the (many) reasons tables fell out of use was the advent of responsive design to meet the needs of different contexts and screen sizes. Tables are rigid. Their presence naturally hampers the responsiveness of a website to conform to a smaller screen size.
But tables are still useful. In some cases, they are necessary.
Real Python: Quiz: Python's Magic Methods: Leverage Their Power in Your Classes
In this quiz, you’ll test your understanding of Python’s Magic Methods.
By working through this quiz, you’ll revisit the concept of magic methods in Python, how they work, and how you can use them to customize the behavior of your classes.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
The Drop Times: Transforming Drupal Site Building: Lauri Timmanee on Experience Builder and Starshot Initiative
Samuel Henrique: Announcing wcurl: a curl wrapper to download files
Whenever you need to download files through the terminal and don't feel like using wget:
wcurl example.com/filename.txtManpage:
https://manpages.debian.org/unstable/curl/wcurl.1.en.html
- Debian unstable - Since 2024-07-02
- Debian testing - Coming up between the second and third week of July 2024.
- Debian 12/bookworm backports - As soon as the package gets to Debian testing, I'll upload it to bookworm.
- Debian 12/bookworm - Depends on whether Debian's release team will approve it, it could be available in the next point release.
- Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.
If you don't want to wait for the package update to arrive, you can always copy
the script and place it in your /usr/bin, the code is here:
https://github.com/Debian/wcurl/blob/main/wcurl
https://salsa.debian.org/debian/wcurl/-/blob/main/wcurl?ref_type=heads
Starting with curl version 8.8.0-2, the Debian's curl package now ships a wcurl executable.
wcurl is the solution for those who just need to download files without having to remember curl's parameters for things like automatically naming the files.
Some people, myself included, would fall back to using wget whenever there was a need to download a file. Sometimes even installing wget just for that usecase. After all, it's easier to remember "apt install wget" rather than "curl -L -O -C - ...".
wcurl consists of a simple shell script that provides sane defaults for the curl invocation, for when the use case is to just download files.
By default, wcurl will:
- Encode whitespaces in URLs;
- Download multiple URLs in parallel;
- Follow redirects;
- Automatically choose a filename as output;
- Perform retries;
- Resume from broken/interrupted downloads.
- Set the downloaded file timestamp to the value provided by the server, if available;
Example, to download a single file:
wcurl example.com/filename.txtIf you ever need to set a custom flag, you can make use of the -o/--opts wcurl option, anything set there will be passed to the curl invocation. Just beware that if you need to set any custom flags, it's likely you will be better served by calling curl directly. The -o/--opts options are there to allow for some flexibility in unforeseen circumstances.
The need for wcurlI've always felt a bit ashamed of not remembering curl's parameters for downloading a file and automatically naming it, having resorted to wget most of the times this was needed (even installing wget when it wasn't there, just for this). I've spoken to a few other experienced people I know and confirmed what could be obvious to others: a lot of people struggle with this.
Recently, the curl project released the results of 2024's curl survey, which also showed this is as a much needed feature, just look at some of the answers:
Q: Which curl command line option do you think needs improvement and how?-O, I really want wget like functionality where I don't have to specify the name
Downloading a file (like wget) could be improved - with automatic naming of the file
downloading files - wget is much cleaner
I wish the default behaviour when GETting a binary was to drop it on disk. That's the only reason 'wget foo.tgz" is still ingrained in my muscle memory .
Maybe have a way to download without specifying something in -o (the only reason i used wget still)
--remote-time should be default
--remote-name-all could really use a short flag
Q: If you miss support for something, tell us what!"Write the data to the file named in the URL (or in redirects if I'm feeling daring), and timestamp the file to the last-modified-date". This is the main reason I'm still using wget.
I can finally feel less bad about falling back to wget due to not remembering the parameters I want.
Idealization vs. realityI don't believe curl will ever change its default behavior in such a way that would accommodate this need, as that would have a side-effect of breaking things which expect the current behavior (the blast radius is literally the solar system).
This means a new executable needs to be shipped side-by-side with curl, an opportunity to start fresh and work with a more focused use case (to download files).
Ideally, this new executable would be maintained by the curl project, make use of libcurl under-the-hood, and be available everywhere. Nobody wants to worry if their systems have the tool or not, it should always be there.
Given I'm just a Debian Developer, with not as much free time as I wish, I've decided to write a simple shell script wrapper calling the curl CLI under-the-hood.
wcurl will come installed with the curl package from now on, and I will check with the release team about shipping it on the current Debian stable as well. Shipping wcurl in other distros will be up to them (Debian-derivatives should pick it up automatically, though).
We've tried to make it easy for anyone to ship this by using the curl license, keeping the script POSIX-compliant, and shipping a manpage.
Maybe if there's enough interest across distributions, someone might sign up for implementing this in upstream curl and increase its reach. I would be happy with the curl project reusing the wcurl name when that happens. It's unlikely that wcurl would be shipped by curl upstream as it is, assuming they would prefer a solution that uses libcurl direclty (more similar to curl the CLI, to maintain).
In the worst case, wcurl becomes a Debian-specific tool that only a few people are aware of, in the best case, it becomes the new go-to CLI tool for simply downloading files. I would be happy if at least someone other than me finds it useful.
Naming is hardWhen I started working on it, I was calling the new executable "curld" (stands for "curl download"), but then when discussing this in one of our weekly calls in the Debian Brasília community, it was mentioned that this could be confused for a daemon.
We then settled for the name "wcurl", which doesn't really stand for anything, but it's very easy to remember.
You know... "it's that wget alternative for when you want to use curl instead" :)
FeedbackI'm hosting the code on Github and Debian's GitLab instance, feel free to open an issue to provide feedback.
https://salsa.debian.org/debian/wcurl
https://github.com/Debian/wcurl
We also have a Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org
We have historically spoken Portuguese in the room but we'll switch to
English in case anyone joins.
The idea for wcurl came a few days before the curl-up conference 2024. I've been thinking a lot about developer productivity in the terminal lately, different tools and better defaults. Before curl-up, I was also thinking about packaging improvements for the curl package. I don't remember what exactly happened, but I likely had to download something and felt a bit ashamed of maintaining curl and not remembering the parameters to download files the way I wanted.
I first discussed this idea in the conference, where I asked the participants about it and there were no concerns raised, and some people said I should give it a go. Participating in curl-up was a really great experience and I'm thankful for the interactions I've had there.
On the Debian side, I've got reviews of the code and manpage by Sergio Durigan Junior <sergiodj>, Guilherme Puida Moreira <puida> and Carlos Henrique Lima Melara <charles>. Sergio ended up rewriting the tool to be POSIX-compliant (my version was written in bash), so he takes all the credit for the portability.
Calamares & some Distro Notes
Calamares is an indepdendent Linux distro-installer, and we just released Calamares 3.3.7. There’s a couple of known issues that need tracking down, but it is a slow process – one entirely dependent on how much time volunteers are able and willing to put into careful bug reporting (and reproduction) and then dealing with code to fix them. Anyway, here’s some semi-coherent notes about Calamares and distro’s and issues and things.
Installing KubuntuI recently had to install an “emergency PC” for my mom, and picked Kubuntu – fairly arbitrarily, I must say. The install-Kubuntu icon does not look like Calamares, and first comes up with some other selection dialog in a different style, so I was pleasantly surprised to see it using Calamares after that.
It picked up Dutch from geo-location, which was fine. Less fine is that the Dutch translations aren’t complete. I’m pretty sure I could fix that.
Installation was straightforward, although I always pick “erase whole disk” and click through nearly all of the defaults. No complicated installs for me, and it worked without a hitch.
Mom didn’t need the emergency PC after all, so now I do have a spare low-end desktop machine with Kubuntu. I might turn it into a try-real-installs box for a while.
Building Calamares on KDE NeonKDE Neon was a CI target for Calamares for a long time, but I switched it off in March 2024. Dependencies were a mess at the time, and the automatic CI builds were failing every night. Other distro’s didn’t have that struggle, so I put more CI weight on Fedora.
I did keep one of my desktop machines installed with KDE Neon, as a gaming machine. At least I can see what Plasma 6 is supposed to be like, as a reference for when we land it in FreeBSD. Untangling the package mess to return it to a development machine was just too much of a hassle.
At some point I removed Qt5 – and everything that depended on it – and then reinstalled some bits and did a pkcon update and whatever and the machine finally ended up in a workable state for development again, but don’t ask me exactly what I did and don’t talk to me about the upgrade experience.
Building Calamares on FreeBSDIt’s possible, just probably not very useful. Clang spits out mountains of warnings, which I occasionally try to address.
Not all of the warnings are all that useful – when dealing with command-line arguments, for instance. The relationship between argc and argv (conventional names for parameters) in main() is clear, but there’s nothing in the type of either to express that, so you get warnings like this one:
src/libcalamares/geoip/test_geoip.cpp:37:45: warning: unsafe buffer access [-Wunsafe-buffer-usage] 37 | QString selector = argc == 3 ? QString( argv[ 2 ] ) : QString(); Building Calamares on EndeavourOSThe live-ISO for EndeavourOS is one of my favorites. I don’t know why they ship an ISO with git in the live-image, but it means that I trivially have a system with a working Calamares configuration, which I can update with the latest version:
git clone http://github.com/calamares/calamares cd calamares sudo ./ci/deps-endeavouros.sh sudo pacman -Scc export CMAKE_ARGS="-DWITH_QT6=ON -DCMAKE_BUILD_TYPE=Debug" export BUILDDIR=build ./ci/build.shI have a VM with the live-ISO, but also two virtual disks attached. By doing the checkout and package download on one of the disks, and doing test-installs to the other, this is by far the easiest and most pleasant develop-and-test setup I have right now.
And the wallpaper is pretty nice, too.
Gaël Varoquaux: Skrub 0.2.0: tabular learning made easy
We just released skrub 0.2.0. This release markedly simplifies learning on complex dataframes.
model = tabular_learner(‘classifier’) Simple, yet solid default baselineThe highlight of the release is the tabular_learner function, which facilitates creating pipelines that readily perform machine learning on dataframes, adding preprocessing to a scikit-learn compatible learner. The function packs defaults and heuristics to transform all forms of dataframes to a representation that is well suited to a learner, and it can adapt these transformation: tabular_learner(HistGradientBoostingClassifier()) encodes categories differently than tabular_learner(LogisticRegression()).
The heuristics are tuned based on much benchmarking and experience shows that they give good tradeoffs. The default tabular_learner(‘classifier’) is often a strong baseline.
The benefit are visible in a really simple example:
>>> # First retrieve data >>> from skrub.datasets import fetch_employee_salaries >>> dataset = fetch_employee_salaries() >>> df = dataset.X >>> y = dataset.y >>> # The dataframe is a quite rich and complex dataframe, with various columns >>> dfWe can then easily build a learner that applies readily to this dataframe, without any transformation:
>>> from skrub import tabular_learner >>> learner = tabular_learner('regressor') >>> # The resulting learner can apply all the machine-learning conveniences (eg cross-validation) directly on the dataframe >>> from sklearn.model_selection import cross_val_score >>> cross_val_score(learner, df, y) array([0.89370447, 0.89279068, 0.92282557, 0.92319094, 0.92162666]) transformer = TableVectorizer() Making encoding complex dataframes easyBehind the hood, the work is done by the skrub.TableVectorizer(), a scikit-learn compatible transformer that facilitates combining multiple transformations on the different columns of a dataframe. The TableVectorizer is not new in the 0.2.0 release, but we have completely revamped its internals to cover really well edge cases. Indeed, one challenge is to make sure that nothing different or strange happens at test time. Actually, enforcing consistency between train-time and test-time transformation is the real value of skrub compared to using pandas or polars to do transformation.
Increasing support of polars Short-term goal of optimized support for pandas and polarsWe have implemented a new mechanism for supporting both pandas and polars. It has not been applied on all the codebase, hence the support is still imperfect. However, we are seeing increasing support for polars in skrub, and our goal in the short term is to provide rock-solid polar support.
Try skrub out! It’s still young, but in mind opinion, it provides a lot of value to tabular learning.
FSF Blogs: The Licensing & Compliance Team, running at full steam for your freedom
The Licensing & Compliance Team, running at full steam for your freedom
PyCoder’s Weekly: Issue #636 (July 2, 2024)
#636 – JULY 2, 2024
View in Browser »
In this video course, you’ll learn how to create graphical user interface (GUI) applications with Python and PyQt. Once you’ve covered the basics, you’ll build a fully functional desktop calculator that can respond to user events with concrete actions.
REAL PYTHON course
Umbra Space has released a data set consisting of satellite based radar images of shipping. This article from Mark shows you how to grab the data, visualize, and annotate it.
MARK LITWINTSCHIK
Logfire, by the makers of Pydantic, is an observability platform that will help you understand your app’s behavior with less code and time. Built on OpenTelemetry, it features user-friendly dashboards, SQL querying, and Python-specific integrations. Get started today →
PYDANTIC sponsor
This is a very detailed list of best practices for developing in Python. It includes tools, language features, application design, which libraries to use an more.
STUART ELLIS
In this tutorial, you’ll build a guitar synthesizer using the Karplus-Strong algorithm in Python. You’ll model vibrating strings, simulate strumming techniques, read musical notation and tablature, and apply audio effects. By the end, you’ll have created a digital guitar that can play any song. This tutorial was also discussed on Real Python Podcast Episode #210.
REAL PYTHON
The backdoor introduced to the xz-utils compression project through social engineering was one of the topics at the Python Language Summit. Participants discussed what can be done to prevent similar social engineering attacks on the Python source.
PYTHON SOFTWARE FOUNDATION
“With PropelAuth, I think I’ve spent about a day – total – on auth over the past year.” PropelAuth is easy to integrate and provides all the tools your team needs to manage your users - dashboards, user insights, impersonation, SSO and more →
PROPELAUTH sponsor
“Prettier” is a JavaScript based linting tool for templates. For folks not familiar with the world of npm, it can be a bit daunting to get it going. Simon fiddled with it so you don’t have to and posted how he got it working on his system.
SIMON WILLISON
An often overlooked aspect of software development is architecture at the module & function level. It is important to write code that is simple and easy to move from one place to another.
DAVIDVUJIC.BLOGSPOT.COM • Shared by David Vujic
This is a simple, quick guide to getting started running LLMs on your local computer. It covers the basics of the powerful libraries Ollama and LangChain for controlling these AI models.
JOSHUA COOK • Shared by Joshua Cook
This tutorial covers hash tables, collision handling, performance optimization and how it relates to the implementation of the set data structure in Python.
ABHINAV UPADHYAY
Keeping a counter across objects in Django means having to be careful about race conditions. This article outlines several approaches to the problem.
GONÇALO VALÉRIO
Brett talks about the different ways you can support the many maintainers of open source projects, and often times just saying “thanks” means a lot.
BRETT CANNON
Learn all about weak references in Python: reference counting, garbage collection, and practical uses of the weakref module
MARTIN HEINZ • Shared by Martin Heinz
In this tutorial, you’ll learn how to create, organize, and format high-quality README files for your Python projects.
REAL PYTHON
This quick TIL post from Rodrigo shows you how to get information about the terminal size from the shutil module.
RODRIGO GIRÃO SERRÃO
Learn how to use pytest fixtures for writing maintainable and isolated tests.
STANLEY ULILI
GITHUB.COM/POMPONCHIK • Shared by pomponchik
oxo: Security Scanning Orchestrator jax: Composable Transformations of Python+NumPy Programs dbt-utils: Utility Functions for DBT Projects Events Weekly Real Python Office Hours Q&A (Virtual) July 3, 2024
REALPYTHON.COM
July 4, 2024
MEETUP.COM
July 4, 2024
SYPY.ORG
July 8 to July 15, 2024
EUROPYTHON.EU
July 8 to July 14, 2024
SCIPY.ORG
July 10 to July 14, 2024
PYCON.ORG
Happy Pythoning!
This was PyCoder’s Weekly Issue #636.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]
Dima Kogan: vnlog.slurp() with non-numerical data
For a while now I'd see an annoying problem when trying to analyze data. I would be trying to import into numpy an innocuous-looking data file like this:
# image x y z temperature image1.png 1 2 5 34 image2.png 3 4 1 35As usual, I would be using vnlog.slurp() (a thin wrapper around numpy.loadtxt()) to read this in, but that doesn't work: the image filenames aren't parseable as numerical values. Up until now I would work around this by using the suprocess module to fork off a vnl-filter -p !image and then slurp that, but it's a pain and slow and has other issues. I just solved this conclusively using the numpy structured dtypes. I can now do this:
dtype = np.dtype([ ('image', 'U16'), ('x y z', int, (3,)), ('temperature', float), ]) arr = vnlog.slurp("data.vnl", dtype=dtype)This will read the image filename, the xyz points and the temperature into different sub-arrays, with different types each. Accessing the result looks like this:
print(arr['image']) ---> array(['image1.png', 'image2.png'], dtype='<U16') print(arr['x y z']) ---> array([[1, 2, 5], [3, 4, 1]]) print(arr['temperature']) ---> array([34., 35.])Notes:
- The given structured dtype defines both how to organize the data, and which data to extract. So it can be used to read in only a subset of the available columns. Here I could have omitted the temperature column, for instance
- Sub-arrays are allowed. In the example I could say either
dtype = np.dtype([ ('image', 'U16'),
('x y z', int, (3,)),
('temperature', float), ])
or
dtype = np.dtype([ ('image', 'U16'), ('x', int), ('y', int), ('z', int), ('temperature', float), ])The latter would read x, y, z into separate, individual arrays. Sometime we want this, sometimes not.
- Nested structured dtypes are not allowed. Fields inside other fields are not supported, since it's not clear how to map that to a flat vnlog legend
- If a structured dtype is given, slurp() returns the array only, since the field names are already available in the dtype
We still do not support records with any null values (-). This could probably be handled with the converters kwarg of numpy.loadtxt(), but that sounds slow. I'll look at that later.
This is available today in vnlog 1.38.
Bits from Debian: Bits from the DPL
Dear Debian community,
Statement on Daniel PocockThe Debian project has successfully taken action to secure its trademarks and interests worldwide, as detailed in our press statement. I would like to personally thank everyone in the community who was involved in this process. I would have loved for you all to have spent your volunteer time on more fruitful things.
Debian Boot team might need helpI think I've identified the issue that finally motivated me to contact our teams: for a long time, I have had the impression that Debian is driven by several "one-person teams" (to varying extents of individual influence and susceptibility to burnout). As DPL, I see it as my task to find ways to address this issue and provide support.
I received private responses from Debian Boot team members, which motivated me to kindly invite volunteers to some prominent and highly visible fields of work that you might find personally challenging. I recommend subscribing to the Debian Boot mailing list to see where you might be able to provide assistance.
/usrmergeHelmut Grohne confirmed that the last remaining packages shipping aliased files inside the package set relevant to debootstrap were uploaded. Thanks a lot for Helmut and all contributors that helped to implement DEP17.
Contacting more teamsI'd like to repeat that I've registered a BoF for DebConf24 in Busan with the following description:
This BoF is an attempt to gather as much as possible teams inside Debian to exchange experiences, discuss workflows inside teams, share their ways to attract newcomers etc.
Each participant team should prepare a short description of their work and what team roles (“openings”) they have for new contributors. Even for delegated teams (membership is less fluid), it would be good to present the team, explain what it takes to be a team member, and what steps people usually go to end up being invited to participate. Some other teams can easily absorb contributions from salsa MRs, and at some point people get commit access. Anyway, the point is that we work on the idea that the pathway to become a team member becomes more clear from an outsider point-of-view.
I'm lagging a bit behind my team contacting schedule and will not manage to contact every team before DebConf. As a (short) summary, I can draw some positive conclusions about my efforts to reach out to teams. I was able to identify some issues that were new to me and which I am now working on. Examples include limitations in Salsa and Salsa CI. I consider both essential parts of our infrastructure and will support both teams in enhancing their services.
Some teams confirmed that they are basically using some common infrastructure (Salsa team space, mailing lists, IRC channels) but that the individual members of the team work on their own problems without sharing any common work. I have also not read about convincing strategies to attract newcomers to the team, as we have established, for instance, in the Debian Med team.
DebConf attendanceThe amount of money needed to fly people to South Korea was higher than usual, so the DebConf bursary team had to make some difficult decisions about who could be reimbursed for travel expenses. I extended the budget for diversity and newcomers, which enabled us to invite some additional contributors. We hope that those who were not able to come this year can make it next year to Brest or to MiniDebConf Cambridge or Toulouse
tag2uploadOn June 12, Sean Whitton requested comments on the debian-vote list regarding a General Resolution (GR) about tag2upload. The discussion began with technical details but unfortunately, as often happens in long threads, it drifted into abrasive language, prompting the community team to address the behavior of an opponent of the GR supporters. After 560 emails covering technical details, including a detailed security review by Russ Allbery, Sean finally proposed the GR on June 27, 2024 (two weeks after requesting comments).
Firstly, I would like to thank the drivers of this GR and acknowledge the technical work behind it, including the security review. I am positively convinced that Debian can benefit from modernizing its infrastructure, particularly through stronger integration of Git into packaging workflows.
Sam Hartman provided some historical context [1], [2], [3], [4], noting that this discussion originally took place five years ago with no results from several similarly lengthy threads. My favorite summary of the entire thread was given by Gregor Herrmann, which reflects the same gut feeling I have and highlights a structural problem within Debian that hinders technical changes. Addressing this issue is definitely a matter for the Debian Project Leader, and I will try to address it during my term.
At the time of writing these bits, a proposal from ftpmaster, which is being continuously discussed, might lead to a solution. I was also asked to extend the GR discussion periods which I will do in separate mail.
Talk: Debian GNU/Linux for Scientific ResearchI was invited to have a talk in the Systems-Facing Track of University of British Columbia (who is sponsoring rack space for several Debian servers). I admit it felt a bit strange to me after working more than 20 years for establishing Debian in scientific environments to be invited to such a talk "because I'm DPL". 😉
Kind regards Andreas.
direvent @ Savannah: GNU Direvent Version 5.4
GNU direvent version 5.4 is available for download.
New in this version:
It is possible to limit number of command instances that are allowed to run simultaneously for a particular watcher. This is done using
the max-instances statement in watcher section.
In previous version, watchers waited for the handler to terminate, unless given the nowait option explicitly. It is now fixed and nowait is the default, as described in the documentation.
In some cases setting the sentinel effectively removed the original watcher. That happened if the full file name of the original watcher
and its directory part produced the same hash code.