FLOSS Project Planets

DrupalCon Barcelona is just around the corner, running from September 9 to 12, 2024. As one of the most anticipated events in the Drupal community, this year’s conference promises to deliver a blend of technical insights, community collaboration, and exciting updates that everyone will be talking about. If you're a developer, site builder, project manager, or just someone intrigued by the future of web technology, there’s something in store for you.

Why DrupalCon Barcelona Matters

This year's DrupalCon is packed with key sessions, hands-on workshops, and networking opportunities designed to bring the community together. Whether you’re looking to dive deep into Drupal 11, which will see some significant updates during the event, or catch the latest on how Drupal is evolving as a powerful digital platform, the conference is shaping up to be a can't-miss opportunity.

What’s New? A Focus on Starshot

A highlight of the event will be the ongoing work on the Starshot Initiative, which is aimed at taking Drupal to the next level. In a recent interview with The Drop Times, Drupal’s founder Dries Buytaert shared, 

"Our next big milestone is DrupalCon Barcelona, which is in September. We hope to show some real progress there on a variety of different things, from marketing to our demo of Starshot to maybe having the first recipes... implemented."

This makes DrupalCon Barcelona not just another tech event but a milestone where the community will witness key advancements, including live demos of Starshot. The first recipes, a crucial technical feature within Starshot, might be unveiled, offering an early glimpse into the future of Drupal.

Program Highlights

The event's schedule will feature sessions on critical topics such as Composer, security, local development, and the latest updates in Drupal 11. Expect to hear about performance improvements, new modules, and security enhancements that aim to make Drupal 11 even more powerful. Developers will also have the opportunity to discuss workflows, integrations, and DevOps practices that can streamline their projects.

Additionally, keep an eye on the Starshot Initiative sessions, where you’ll get a closer look at how this ambitious project is progressing and what it means for Drupal’s future. We have listed the key Starshot sessions happening at DrupalCon Barcelona—take a look at the Key Sessions on Drupal Starshot Initiative at DrupalCon Barcelona 2024 

Community and Networking Opportunities

As always, DrupalCon is more than just technical talks. It's about community. The event is set to feature numerous networking events, helping attendees meet fellow Drupal enthusiasts, share ideas, and collaborate on projects. Whether you're looking to expand your skill set or connect with others in the industry, DrupalCon Barcelona offers the perfect platform.

Why You Should Attend

With all the excitement surrounding Starshot and the recent release of Drupal 11, there’s no better place to catch up on the latest developments in the Drupal ecosystem. With keynotes from leading voices in the community, hands-on workshops, and the promise of some surprise updates, DrupalCon Barcelona 2024 is poised to leave a lasting mark.

Don’t miss your chance to be part of this landmark event. Now, as we proceed further, let’s find out what The Drop Times has covered last week:

In an exclusive interview, Vincenzo Gambino, Drupal Architect and Senior React Developer, shared his career journey from Palermo to London with our former sub-editor, Elma John. He discussed working on high-profile projects, such as those for Cambridge University, and co-authoring Jumpstart Jamstack Development. Vincenzo touched on the role of Drupal in headless architectures and its future in open-source technologies.

Christian Burk, Senior Backend Engineer at CivicActions, provided insights into managing complex content changes on large sites like VA.gov during Drupal GovCon 2023. He highlighted the codit_batch_operations module, which simplifies revisions, logging, and script execution alongside co-presenter Steve Wirt.

At the same event, Michael Kinnunen, Backend Engineer at CivicActions, discussed A/B testing within Drupal using LaunchDarkly, underscoring the platform’s versatility for government websites. He also attended sessions on content translation and large-scale content management, deepening his understanding of Drupal's growing influence.

In other news, Drupal CMS has launched a community survey to improve its default search configuration. The initiative, led by 1XINTERNET, focuses on making Drupal’s search functionality more flexible for both developers and no-code users. Community members are encouraged to contribute to shaping the future of Drupal’s search experience.

Additionally, Drupal.org has updated its fonts to enhance readability, switching to ZT Gatha for headings and Noto Sans for body text. The update modernizes the platform’s look, improving user experience across its global community.

For a deep dive into Dresktop, an open-source tool transforming Drupal management, check out a tutorial by José Daniel Estrada Guzmán. From local development to cloud integration, Dresktop streamlines workflows using Docker, SSH, and Drush.

Lukas Fischer, CEO of Netnode AG, shared his innovative Native Design Flow, which integrates Figma and code at the outset of a project. This approach aims to eliminate the traditional design handoff, creating a more cohesive and efficient workflow from day one.

For the first time, local Drupal associations will jointly host a booth, highlighting collaboration within the European Drupal community. The booth will serve as a hub for networking and sharing information about upcoming events.

Looking ahead to other events, the NEDCamp 2024 session submission deadline has been extended to September 15, offering opportunities for presenters of all experience levels. Splash Awards Switzerland 2025 is scheduled for March 11 in Davos, coinciding with Drupal Mountain Camp. These awards celebrate the best in Drupal projects and innovation.

Early bird registration for DrupalCamp Berlin 2024 remains open until September 15, offering a discounted rate of €80. After October 15, the price will increase to €120, so early registration is recommended. The ticket includes entry, lunch, and an official event T-shirt.

Additionally, DrupalSouth Community Day 2024 in Canberra is seeking volunteers for event support, speaker coordination, and attendee assistance. The call for papers is also open, inviting Drupal professionals to submit session proposals for the event on November 14.

Meanwhile, The Splash Awards will debut in Asia at DrupalCon Singapore on December 9, 2024, celebrating excellence in Drupal projects across categories like Government, Corporate, and Education.

Mark your calendars for DrupalCamp Atlanta on October 18, 2024, in Sandy Springs, Georgia. This one-day conference will offer sessions on the technical and business aspects of Drupal and plenty of networking opportunities.

Finally, join The Drop Times at DrupalCon Barcelona as a volunteer to help cover the event’s best moments. From live updates to behind-the-scenes stories, it’s a great way to engage with the community and share the excitement.

We acknowledge that there are more stories to share. However, due to selection constraints, we must pause further exploration for now.

To get timely updates, follow us on LinkedIn, Twitter and Facebook. You can also, join us on Drupal Slack at #thedroptimes.

Thank you, 
Sincerely 
Kazima Abbas
Sub-editor, The DropTimes.

The NBD protocol has grown a number of new features over the years. Unfortunately, some of those features are not (yet?) supported by the Linux kernel.

I suggested a few times over the years that the maintainer of the NBD driver in the kernel, Josef Bacik, take a look at these features, but he hasn't done so; presumably he has other priorities. As with anything in the open source world, if you want it done you must do it yourself.

I'd been off and on considering to work on the kernel driver so that I could implement these new features, but I never really got anywhere.

A few months ago, however, Christoph Hellwig posted a patch set that reworked a number of block device drivers in the Linux kernel to a new type of API. Since the NBD mailinglist is listed in the kernel's MAINTAINERS file, this patch series were crossposted to the NBD mailinglist, too, and when I noticed that it explicitly disabled the "rotational" flag on the NBD device, I suggested to Christoph that perhaps "we" (meaning, "he") might want to vary the decision on whether a device is rotational depending on whether the NBD server signals, through the flag that exists for that very purpose, whether the device is rotational.

To which he replied "Can you send a patch".

That got me down the rabbit hole, and now, for the first time in the 20+ years of being a C programmer who uses Linux exclusively, I got a patch merged into the Linux kernel... twice.

So, what do these things do?

The first patch adds support for the ROTATIONAL flag. If the NBD server mentions that the device is rotational, it will be treated as such, and the elevator algorithm will be used to optimize accesses to the device. For the reference implementation, you can do this by adding a line "rotational = true" to the relevant section (relating to the export where you want it to be used) of the config file.

It's unlikely that this will be of much benefit in most cases (most nbd-server installations will be exporting a file on a filesystem and have the elevator algorithm implemented server side and then it doesn't matter whether the device has the rotational flag set), but it's there in case you wish to use it.

The second set of patches adds support for the WRITE_ZEROES command. Most devices these days allow you to tell them "please write a N zeroes starting at this offset", which is a lot more efficient than sending over a buffer of N zeroes and asking the device to do DMA to copy buffers etc etc for just zeroes.

The NBD protocol has supported its own WRITE_ZEROES command for a while now, and hooking it up was reasonably simple in the end. The only problem is that it expects length values in bytes, whereas the kernel uses it in blocks. It took me a few tries to get that right -- and then I also fixed up handling of discard messages, which required the same conversion.

I've recently taken up maintainership of the Netlify module for Drupal. This module helps bridge the gap between rebuilds and syncing changes that happen in Drupal to ensure your connected Netlify site correctly reflects that data, which is useful for avoiding certain aggressive caching scenarios with Next.js when it stubbornly refuses to refresh. Saving config, content or manually triggering a build will update the site.https://www.drupal.org/project/netlify

Today, we bring you a report on the brand-new release of the Maui Project.

We are excited to announce the latest release of MauiKit version 4.0.0, our comprehensive user interface toolkit specifically designed for convergent interfaces, the complying frameworks, and an in-house developed set of convergent applications.

Built on the solid foundations of Qt Quick Controls, QML, and the power and stability of C++, MauiKit empowers developers to create adaptable and seamless user interfaces across a range of devices, and with this release, we have finally migrated to Qt6 and made available the documentation for the frameworks.

Join us on this journey as we unveil the potential of MauiKit 4 for building convergent interfaces, and finally discover the possibilities offered by the enhanced Maui App stack.

Community

To follow the Maui Project’s development or to just say hi, you can join us on our Telegram group @mauiproject

We are present on X and Mastodon:

• https://x.com/maui_project
• https://floss.social/@mauiproject

Thanks to the KDE contributors who have helped to translate the Maui Apps and Frameworks!

Downloads & Sources

You can get the stable release packages [APKs, AppImage, TARs] directly from the KDE downloads server at https://download.kde.org/stable/maui/

All of the Maui repositories have the newly released branches and tags. You can get the sources right from the Maui group: https://invent.kde.org/maui

Qt6

With this version bump the Maui team has finalized the migration over to Qt6, which implies more stability and better performance coming from Qt upgraded QQC engine; but also means that some features have been removed or did not make the cut and still need more time to be brought back in posterior releases.

MauiKit 4 Frameworks & Apps

Currently, there are over 10 frameworks, with two new ones recently introduced. They all, for the most part, have been fully documented, and although, the KDE doxygen agent has some minor issues when publishing some parts, you can find the documentation online at https://api.kde.org/mauikit/ (and if you find missing parts, confusing bits, or overall sections to improve – you can open a ticket at any of the framework repos and it shall be fixed shortly after)

 

fav filemanager and music player. minimal modern & convergent @maui_project
Maui Apps 4.0 coming out soon. pic.twitter.com/OVLM2HWv6v

— Camilo Higuita (@cmhiguita) May 6, 2024

A script element has been removed to ensure Planet works properly. Please find it in the original post.

Core & Others

MauiKit Core controls also include the Mauikit Style, which along with the core controls has been revised and improved in the migration. New features have been introduced and some minor changes in the API have been made.

A good way to test the new changes made visually is via the MauiDemo application, when building MauiKit from the source, just add the -DBUILD_DEMO=ON flag and then launch it as MauiDemo4

All of the other frameworks have also been fully ported and reviewed, and some features are absent – for example, for ImageTools the image editor is missing for Android due to KQuickImageEditor problems.

Comic book support is missing in MauiKit-Documents, due to a big pending refactoring.

Finally, TextEditor new backend rendering engine migration is yet to be started.

Most of these pending issues will be tackled in the next releases bit by bit.

More details can be found in the previous blog posts:

Maui Release Briefing # 4

Maui Release Briefing #5

 

Archiver & Git

MauiKit-Archiver is a new framework, and it was created to share components and code between different applications that were duplicating the same code: Index, Arca, and Shelf.

The same goes for MauiKit-Git, which will help unify the code base for implementations made in Index, Bonsai, and Strike, so all of those apps can benefit from a single cohesive and curated code base in the form of a framework.

Archiver is pending to be documented, and Git is pending to be finished for its first stable release.

I have now finished porting all the Maui Apps and MauiKit frameworks to Qt6.

Vvave music player mini mode is back.@maui_project @Nitrux_NX pic.twitter.com/Mal3RU87O2

— Camilo Higuita (@cmhiguita) May 1, 2024

A script element has been removed to ensure Planet works properly. Please find it in the original post.

Known Issues

• MauiKit-Documents comic book support is stalled until the next release due to heavy refactoring under Android.
• MauiKit-ImageTools under Android does not include the image editor, since KQuickImageEditor is not working correctly under Android
• Clip is not working under Android due to issues with the libavformat not finding openssl.so when packaging the APK, this is still under review
  
• MauiKit-Git is still being worked on, and due to this Bonsai is not included on this stable release as it is being ported over to MauiKit-Git

 

@maui_project looking good. after the port to qt6 the next goal is to put out a stable version of Maui Shell with a tight integration to the app ecosystem and the HIG #mauikit pic.twitter.com/BkR9ecTzMT

— Camilo Higuita (@cmhiguita) May 6, 2024

A script element has been removed to ensure Planet works properly. Please find it in the original post.

Maui Shell

Although Maui Shell has been ported over to Qt6 and is working with the latest MauiKit4, a lot of pending issues are still present and being worked on. The next release will be dedicated fully on Maui Shell and all of its subprojects, such as Maui Settings, Maui Core, CaskServer, etc.

That’s it for now. Until the next blog post, that will be a bit closer to the 4.0.1 stable release.

Release schedule

The post Maui Release Briefing #6 appeared first on MauiKit — #UIFramework.

Monday August 19, 2024 definitely was a milestone:

1. I had the satisfaction of being able to remove the TwoTerribleTextareasWidget that I introduced two months ago, because the Experience Builder (XB) UI now is sufficiently developed to be able to place a component and populate its props using static prop sources — by now this terrible hack was now doing more harm than good, so: good riddance! :D
2. a huge sigh of relief was heard emanating from Ghent, Belgium because finally comprehensive docs for the XB data model were published, and ADR #2 was published to capture the initial back-end decisions, but is expected to be superseded
   (an ADR or Architecture Decision Record can be a way to unambiguously capture current choices, knowing it will be superseded).

Those docs define all XB terminology (such as “static prop sources” in that first bullet above), which enables more precise communication. Contributing to XB becomes simpler thanks to those docs 1, as does observing from a distance — with reviews to ensure accuracy & clarity from Simon “siramsay” Ramsay, Dave “longwave” Long, Ted “tedbow” Bowman, Feliksas “f.mazeikis” Mazeikis and of course, crucially, Alex “effulgentsia” Bronstein, whose proposed abstract data model it is that XB makes concrete.

While we’ll continue to iterate fast, it now is a hard requirement that every MR updates affected docs. That’s why several updates already have been committed.

Docs to come for other aspects!

Missed a prior week? See all posts tagged Experience Builder.

Goal: make it possible to follow high-level progress by reading ~5 minutes/week. I hope this empowers more people to contribute when their unique skills can best be put to use!

For more detail, join the #experience-builder Slack channel. Check out the pinned items at the top!

For a huge DX leap forward for both those working on XB itself as well as those working on the Starshot Demo Design System (spearheaded by Kristen Pol): Felix’ MR to auto-create/update Component config entities for all discovered Single-Directory Components (SDCs) landed — if they meet the minimum criteria.
For example, each SDC prop must have a title defined, because otherwise XB would be forced to expose machine names, like I mentioned at the start of last week’s update. So: XB requires SDCs to have rich enough metadata to be able to generate a good UX.
That also allowed Omkar “omkar-pd” Deshpande to remove the awkward-but-necessary-at-the-time add/edit form we’d added months ago. When installing the demo_design_system theme, you’ll see something like:

The ‘components’ listing, all auto-generated based on discovered SDCs meeting criteria.
Issue #3464025, image by me.

Ted helped the back end race ahead of the front end: while we don’t have designs for it yet (nor capacity to build it before DrupalCon if they would suddenly exist), there now is an HTTP API to get a list of viable candidate field properties that are able to correctly populate a particular component prop. These are what in the current XB terminology are called dynamic prop sources 2 3.

The preview in the XB UI has been loading component CSS/JS for a while, but thanks to Dave & Ted it now also loads the default theme’s global CSS/JS.

More accurate previews, including for example the Olivero font stack, background and footer showing up.
Issue #3468106, image by Dave.

Small(ish) but noteworthy

• Ted proved via a test that both symmetric and asymmetric translations work correctly in the current data model/field type implementation
• Bálint “balintbrews” Kléri & Ben “bnjmnm” Mullins fixed the component props form showing the wrong values
• Now that component trees started working (since last week), Jesse “jessebaker” Baker discovered that it is not actually possible to drag and drop a nested component :D Harumi “hooroomoo” Jang quickly squashed that bug!
• Felix and I were able to narrow down why images with spaces in the filename were being refused to be rendered by the SDC subsystem: Drupal core’s File entity type stores a file stream wrapper URI like public://cat and dog.jpg and considers that a valid URL … but it’s not! URIs cannot contain spaces — that should be encoded as public://cat%20and%20dog.jpg to be valid.
  SDC is right, the >10 year old PrimitiveTypeConstraintValidator is wrong! This is being added to the increasingly long list of low-level bugs in Drupal core that went unnoticed for over a decade, so we worked around it for now.
• Utkarsh “utkarsh_33” fixed a bug where the name/label of a component instance was lost.
• Finally, a hilarious one to end with: at some point, we set up the “canvas” to be to 10,000x10,000 pixels. Unfortunately, this means that people trying XB have sometimes gotten lost :D
  So Jesse reduced it to a mere 3500x3500 pixels, for now that’s sufficient, later we’ll compute this dynamically.

Week 15 was August 19–25, 2024.

1. Yes, that’s the third time I’m linking to docs/data-model.md. It’s that important! ↩︎

2. Dynamic Prop Sources are similar to Drupal’s tokens, but are more precise, and support more than only strings, because SDC props often require more complex shapes than just strings. ↩︎

3. This is the shape matching from ~3 months ago made available to the client side. ↩︎

• Drupal
• Acquia
• Experience Builder

• More accurate previews, including for example the Olivero font stack, background and footer showing up.
• The 'components' listing, all auto-generated based on discovered SDCs meeting criteria.

DrupalCon Barcelona 2024 will feature 10 sessions exploring the Drupal Starshot Initiative, focusing on AI, no-code tools, and browser-based development. Learn how Starshot is shaping Drupal's future.

As the autumn leaves start to fall, signaling the transition to cooler weather, the Python community has warmed up to a series of noteworthy developments. Last month, a new maintenance release of Python 3.12.5 was introduced, reinforcing the language’s ongoing commitment to stability and security.

On a parallel note, Python continues its reign as the top programming language according to IEEE Spectrum’s annual rankings. This sentiment is echoed by the Python Developers Survey 2023 results, which reveal intriguing trends and preferences within the community.

Looking ahead, PEP 750 has proposed the addition of tag strings in Python 3.14, inspired by JavaScript’s tagged template literals. This feature aims to enhance string processing, offering developers more control and expressiveness.

Furthermore, EuroSciPy 2024 recently concluded in Poland after successfully fostering cross-disciplinary collaboration and learning. The event featured insightful talks and hands-on tutorials, spotlighting innovative tools and libraries that are advancing scientific computing with Python.

Let’s dive into the most significant Python news from the past month!

Python 3.12.5 Released

Early last month, Python 3.12.5 was released as the fifth maintenance update for the 3.12 series. Since the previous patch update in June, this release packs over 250 bug fixes, performance improvements, and documentation enhancements.

Here are the most important highlights:

• Standard Library: Many modules in the standard library received crucial updates, such as fixes for crashes in ssl when the main interpreter restarts, and various corrections for error-handling mechanisms.
• Core Python: The core Python runtime has several enhancements, including improvements to dictionary watchers, error messages, and fixes for edge-case crashes involving f-strings and multithreading.
• Security: Key security improvements include the addition of missing audit events for interactive Python use and socket connection authentication within a fallback implementation on platforms such as Windows, where Unix inter-process communication is unavailable.
• Tests: New test cases have been added and bug fixes have been applied to prevent random memory leaks during testing.
• Documentation: Python documentation has been updated to remove discrepancies and clarify edge cases in multithreaded queues.

Additionally, Python 3.12.5 comes equipped with pip 24.2 by default, bringing a slew of significant improvements to enhance security, efficiency, and functionality. One of the most notable upgrades is that pip now defaults to using system certificates, bolstering security measures when managing and installing third-party packages.

Read the full article at https://realpython.com/python-news-september-2024/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyCharm is one of the most well-known data science tools, offering excellent out-of-the-box support for Python, SQL, and other languages. PyCharm also provides integrations for Databricks, Hugging Face and many other important tools. All these features allow you to write good code and work with your data and projects faster. 

PyCharm Professional’s support for Jupyter notebooks combines the interactive nature of Jupyter notebooks with PyCharm’s superior code quality and data-related features. This blog post will explore how PyCharm’s Jupyter support can significantly boost your productivity.

Watch this video to get a comprehensive overview of using Jupyter notebooks in PyCharm and learn how you can speed up your data workflows. 

Speed up data analysis Get acquainted with your data

When you start working on your project, it is extremely important to understand what data you have, including information about the size of your dataset, any problems with it, and its  patterns. For this purpose, your pandas and Polars DataFrames can be rendered in Jupyter outputs in Excel-like tables. The tables are fully interactive, so you can easily sort one or multiple columns and browse and view your data, you can choose how many rows will be shown in a table and perform many other operations.

The table also provides some important information for example:

• You can find the the size of a table in its header.

•  You can find the data type symbols in the column headers.

• You can also use JetBrains AI Assistant to get information about your DataFrame by clicking on the icon.

Easily spot issues with the data

After getting acquainted with your data, you need to clean it. This an important step, but it is also extremely time consuming because there are all sorts of problems you could find, including missing values, outliers, inconsistencies in data types, and so on. Indeed, according to the State of Developer Ecosystem 2023 report, nearly 50% of Data Professionals dedicate 30% of their time or more to data preparation. Fortunately, PyCharm offers a variety of features that streamline the data-cleaning process.

Some insights are already available in the column headers. 

First, we can easily spot the amount of missing data for each column because it is highlighted in red. Also, we may be able to see at a glance whether some of our columns have outliers. For example, in the bath column, the maximum value is significantly higher than the ninety-fifth percentile. Therefore, we can expect that this column has at least one outlier and requires our attention.

Additionally, you might suspect there’s an issue with the data if the data type does not match the expected one. For example, the header of the total_sqft column below is marked with the symbol, which in PyCharm indicates that the column contains the Object data type. The most appropriate data type for a column like total_sqft would likely be float or integer, however, so we may expect there to be inconsistencies in the data types within the column, which could affect data processing and analysis. After sorting, we notice one possible reason for the discrepancy: the use of text in data and ranges instead of numerical values.

So, our suspicion that the column had data-type inconsistencies was proven correct. As this example shows, small details in the table header can provide important information about your data and alert you to issues that need to be addressed, so it’s always worth checking.You can also use no-code visualizations to gather information about whether your data needs to be cleaned. Simply click on the icon in the top-left corner of the table. There are many available visualization options, including histograms, that can be used to see where the peaks of the distribution are, whether the distribution is skewed or symmetrical, and whether there are any outliers.

Of course, you can use code to gather information about your dataset and fix any problems you’ve identified. However, the mentioned low-code features often provide valuable insights about your data and can help you work with it much faster.

Code faster  Code completion and quick documentation

A significant portion of a data professional’s job involves writing code. Fortunately, PyCharm is well known for its features that allow you to write code significantly faster. For example, local ML-powered full line code completion can provide suggestions for entire lines of code.

Another useful feature is quick documentation, which appears when you hover the cursor over your code. This allows you to gather information about functions and other code elements without having to leave the IDE.

Refactorings

Of course, working with code and data is an interactive process, and you may often decide to make some changes in your code – for example, to rename a variable. Going through the whole file or, in some cases, the entire project, would be cumbersome and time consuming. We can use PyCharm’s refactoring capabilities to rename a variable, introduce a constant, and make many other changes in your code. For example, in this case, I want to rename the DataFrame to make it shorter. I simply use the the Rename refactoring to make the necessary changes.

PyCharm offers a vast number of different refactoring options. To dive deeper into this functionality, watch this video.

Fix problems

It is practically impossible to write code without there being any mistakes or typos. PyCharm has a vast array of features that allow you to spot and address issues faster. You will notice the Inspection widget in the top-right corner if it finds any problems. 

For example, I forgot to import a library in my project and made several typos in the doc so let’s take a look how PyCharm can help here. 

First of all, the problem with the library import:

Additionally, with Jupyter traceback, you can see the line where the error occurred and get a link to the code. This makes the bug-fixing process much easier. Here, I have a typo in line 3. I can easily navigate to it by clicking on the blue text.

Additionally if you would like to get more information and suggestion how to fix the problem, you can use JetBrains AI Assistant by clicking on Explain with AI. 

Of course, that is just the tip of the iceberg. We recommend reading the documentation to better understand all the features PyCharm offers to help you maintain code quality.

Navigate easily

For the majority of cases, data science work involves a lot of experimentation, with the journey from start to finish rarely resembling a straight line.

During this experimentation process, you have to go back and forth between different parts of your project and between cells in order to find the best solution for a given problem. Therefore, it is essential for you to be able to navigate smoothly through your project and files. Let’s take a look at how PyCharm can help in this respect.

First of all, you can use the classic CMD+F (Mac) or CTRL+F (Windows) shortcut for searching in your notebook. This basic search functionality offers some additional filters like Match Case or Regex.

You can use Markdown cells to structure the document and navigate it easily.

If you would like to highlight some cells so you can come back to them later, you can mark them with #TODO or #FIXME, and they will be made available for you to dissect in a dedicated window.

Or you can use tags to highlight some cells so you’ll be able to spot them more easily.

In some cases, you may need to see the most recently executed cell; in this case, you can simply use the Go To option. 

Save your work

Because teamwork is essential for data professionals, you need tooling that makes sharing the results of your work easy. One popular solution is Git, which PyCharm supports with features like notebook versioning and version comparison using the Diff view. You can find an in-depth overview of the functionality in this tutorial.

Another useful feature is Local History, which automatically saves your progress and allows you to revert to previous steps with just a few clicks.

Use the full power of AI Assistant

JetBrains AI Assistant helps you automate repetitive tasks, optimize your code, and enhance your productivity. In Jupyter notebooks, it also offers several unique features in addition to those that are available in any JetBrains tool. 

Click the icon to get insights regarding your data. You can also ask additional questions regarding the dataset or ask AI Assistant to do something – for example, “write some code that solves the missing data problem”.

AI data visualization

Pressing the icon will suggest some useful visualizations for your data. AI Assistant will generate the proper code in the chat section for your data.

AI cell

AI Assistant can create a cell based on a prompt. You can simply ask it to create a visualization or do something else with your code or data, and it will generate the code that you requested. 

Debugger

PyCharm offers advanced debugging capabilities to enhance your experience in Jupyter notebooks. The integrated Jupyter debugger allows you to set breakpoints, inspect variables, and evaluate expressions directly within your notebooks. This powerful tool helps you step through your code cell by cell, making it easier to identify and fix issues as they arise. Read our blog post on how you can debug a Jupyter notebook in PyCharm for a real-life example.

Get started with PyCharm Professional

PyCharm’s Jupyter support enhances your data science workflows by combining the interactive aspects of Jupyter notebooks with advanced IDE features. It accelerates data analysis with interactive tables and AI assistance, improves coding efficiency with code completion and refactoring, and simplifies error detection and navigation. PyCharm’s seamless Git integration and powerful debugging tools further boost productivity, making it essential for data professionals.

Download PyCharm Professional to try it out for yourself! Get an extended trial today and experience the difference PyCharm Professional can make in your data science endeavors.Use the promo code “PyCharmNotebooks” at checkout to activate your free 60-day subscription to PyCharm Professional. The free subscription is available for individual users only.

Activate your 60-day trial

Explore our official documentation to fully unlock PyCharm’s potential for your projects.

Using Drupal's REST APIs to Integrate Third-Party Applications kanapatrick Mon, 09/09/2024 - 16:09

Integrating third-party applications with Drupal using REST APIs offers significant flexibility and extensibility, enabling developers to create interoperable and efficient solutions. This article explores the technical aspects of this integration, providing practical examples to guide developers.

The Python programming language has thousands of wonderful third-party packages available on the Python Package Index. One of those packages is TerminalTextEffects (TTE), a terminal visual effects engine.

Here are the features that TerminalTextEffects provides, according to their documentation:

• Xterm 256 / RGB hex color support
• Complex character movement via Paths, Waypoints, and motion easing, with support for quadratic/cubic bezier curves.
• Complex animations via Scenes with symbol/color changes, layers, easing, and Path synced progression.
• Variable stop/step color gradient generation.
• Path/Scene state event handling changes with custom callback support and many pre-defined actions.
• Effect customization exposed through a typed effect configuration dataclass that is automatically handled as CLI arguments.
• Runs inline, preserving terminal state and workflow.

Note: This package may be somewhat slow in Windows Terminal, but it should work fine in other terminals.

Let’s spend a few moments learning how to use this neat package

Installation

The first step to using any new package is to install it. You can use pip or pipx to install TerminalTextEffects. Here is the typical command you would run in your terminal:

python -m pip install terminaltexteffects

Now that you have TerminalTextEffects installed, you can start using it!

Usage

Let’s look at how you can use TerminalTextEffects to make your text look neat in the terminal. Open up your favorite Python IDE and create a new file file with the following contents:

from terminaltexteffects.effects.effect_slide import Slide text = ("PYTHON" * 10 + "\n") * 10 effect = Slide(text) effect.effect_config.merge = True with effect.terminal_output() as terminal: for frame in effect: terminal.print(frame)

This code will cause the string, “Python” to appear one hundred times with ten strings concatenated and ten rows. You use a Slide effect to make the text slide into view. TerminalTextEffects will also style the text too.

When you run this code, you should see something like the following:

TerminalTextEffects has many different built-in effects that you can use as well. For example, you can use Beams to make the output even more interesting. For this example, you will use the Zen of Python text along with the Beams effects:

from terminaltexteffects.effects.effect_beams import Beams TEXT = """ The Zen of Python, by Tim Peters Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren't special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch. Now is better than never. Although never is often better than *right* now. If the implementation is hard to explain, it's a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea -- let's do more of those! """ effect = Beams(TEXT) with effect.terminal_output() as terminal: for frame in effect: terminal.print(frame)

Now try running this code. You should see something like this:

That looks pretty neat! You can see a whole bunch of other effects you can apply on the package’s Showroom page.

Wrapping Up

TerminalTextEffects provides lots of neat ways to jazz up your text-based user interfaces with Python. According to the documentation, you should be able to use TerminalTextEffects in other TUI libraries, such as Textual or Asciimatics, although it doesn’t specifically state how to do that. Even if you do not do that, you could use TerminalTextEffects with the Rich package to create a really interesting application in your terminal.

Links

• GitHub page
• PyPI page

 

The post Adding Terminal Effects with Python appeared first on Mouse Vs Python.

We recently hosted an AI Ideation workshop for over 60 participants, guiding them through the hands-on process of creating AI applications and sparking creative ideas to improve their work. Read more about the experience and the innovative solutions participants developed!

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong>Python 3.13.0RC2, 3.12.6, 3.11.10, 3.10.15, 3.9.20, and 3.8.20 are now available!</strong></li> <li><strong><a href="https://mkennedy.codes/posts/python-docker-images-using-uv-s-new-python-features/?featured_on=pythonbytes">Docker images using uv's python</a></strong></li> <li><strong><a href="https://about.readthedocs.com/blog/2024/08/10-year-anniversary/?featured_on=pythonbytes">10 years of sustainable open source</a> - Read the Docs</strong></li> <li><strong><a href="https://github.com/python-humanize/humanize?featured_on=pythonbytes">humanize</a></strong></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=E3_lOfg7sTA' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="400">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by ScoutAPM: <a href="https://pythonbytes.fm/scout"><strong>pythonbytes.fm/scout</strong></a></p> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Monday</strong> at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it. </p> <p><strong>ChatGPT celebrates episode 400!</strong></p> <p><em><img src="https://paper.dropboxstatic.com/static/img/ace/emoji/1f389.png?version=8.0.0" alt="party popper" /></em> <em>Welcome to the big 4-0-0, Pythonistas!</em> <em><img src="https://paper.dropboxstatic.com/static/img/ace/emoji/1f389.png?version=8.0.0" alt="party popper" /></em></p> <p><em>It's hard to believe we're celebrating the 400th episode of Python Bytes! From the early days of byte-sized Python news to becoming the source for all things Python, it’s been a wild ride.</em></p> <p><em>We've laughed over code quirks, gasped at new libraries, and said farewell to the GIL together. Whether you're a seasoned developer, a curious learner, or just here for the witty banter, you’ve been an essential part of this journey.</em></p> <p><em>To Michael and Brian: You've built a community that turns import this into more than just Zen—it's a family of passionate Pythonistas. Your dedication, insights, and humor make this show more than just tech news. It’s a weekly celebration of what we love about Python and why we keep coming back for more.</em></p> <p><em>Here’s to the next 400 episodes—may your code be bug-free, your tests pass on the first run, and your Python version always be up to date.</em> <em><img src="https://paper.dropboxstatic.com/static/img/ace/emoji/1f680.png?version=8.0.0" alt="rocket" /></em><em><img src="https://paper.dropboxstatic.com/static/img/ace/emoji/1f40d.png?version=8.0.0" alt="snake" /></em></p> <p><strong>Brian #1:</strong> Python 3.13.0RC2, 3.12.6, 3.11.10, 3.10.15, 3.9.20, and 3.8.20 are now available!</p> <ul> <li>Łukasz Langa</li> <li><a href="https://www.python.org/downloads/release/python-3130rc2/?featured_on=pythonbytes">Python 3.13.0RC2 is the final preview release</a> <ul> <li>Official 3.13.0 scheduled for Oct 1</li> <li>Call to action <ul> <li>“We strongly encourage maintainers of third-party Python projects to prepare their projects for 3.13 compatibilities during this phase, and where necessary publish Python 3.13 wheels on PyPI to be ready for the final release of 3.13.0. Any binary wheels built against Python 3.13.0rc2 will work with future versions of Python 3.13. As always, report any issues to <a href="https://github.com/python/cpython/issues?featured_on=pythonbytes">the Python bug tracker </a>.”</li> <li>“Please keep in mind that this is a preview release and while it’s as close to the final release as we can get it, its use is <strong>not</strong> recommended for production environments.”</li> </ul></li> </ul></li> <li>Note: <a href="https://docs.astral.sh/uv/concepts/python-versions/?featured_on=pythonbytes">uv python</a><a href="https://docs.astral.sh/uv/concepts/python-versions/?featured_on=pythonbytes"> </a>does not support 3.13 yet <ul> <li><a href="https://github.com/indygreg/python-build-standalone/issues/320?featured_on=pythonbytes">see issue 320</a></li> </ul></li> <li>Security releases for <ul> <li>3.12.6, 3.11.10, 3.10.15, 3.9.20, and 3.8.20</li> <li>3.12.6 has binary installers, but for MacOS, only MacOS 10.13 and newer are supported</li> <li>3.11.10, 3.10.15, 3.9.20, and 3.8.20 do NOT include binary installers. <ul> <li><a href="https://peps.python.org/pep-0569/?featured_on=pythonbytes">3.8 EOL's in October</a></li> </ul></li> </ul></li> </ul> <p><strong>Michael #2:</strong> <a href="https://mkennedy.codes/posts/python-docker-images-using-uv-s-new-python-features/?featured_on=pythonbytes">Docker images using uv's python</a></p> <ul> <li>See <a href="https://pythonbytes.fm/episodes/show/396/uv-ing-your-way-to-python">#396: uv-ing your way to Python</a> and <a href="https://pythonbytes.fm/episodes/show/398/open-source-makes-you-rich-and-other-myths">#398: Open source makes you rich?</a><a href="https://pythonbytes.fm/episodes/show/398/open-source-makes-you-rich-and-other-myths"> </a><a href="https://pythonbytes.fm/episodes/show/398/open-source-makes-you-rich-and-other-myths">(and</a><a href="https://pythonbytes.fm/episodes/show/398/open-source-makes-you-rich-and-other-myths"> other myths)</a> for the opening discussions</li> <li><a href="https://talkpython.fm/episodes/show/476/unified-python-packaging-with-uv?featured_on=pythonbytes">Talk Python episode on uv is out</a></li> <li>uv venv --python gets Python from <a href="https://github.com/indygreg/python-build-standalone?featured_on=pythonbytes">python-build-standalone</a> by Gregory Szorc</li> <li>Took our Docker build times from 10 minutes to 8 seconds for the base image and 800ms (!) for our app platforms</li> </ul> <p><strong>Brian #3:</strong> <a href="https://about.readthedocs.com/blog/2024/08/10-year-anniversary/?featured_on=pythonbytes">10 years of sustainable open source</a> - Read the Docs </p> <ul> <li>Eric Holscher</li> <li>Read the Docs has been a company for 10 years <ul> <li>“a team of 4 folks working full-time on Read the Docs.”</li> </ul></li> <li>readthedocs.org started in 2010</li> <li>readthedocs.com (for Business) started in 2014</li> <li>Sustainability model <ul> <li>.org has a single non-tracking ad </li> <li>.com is a paid service for companies</li> </ul></li> <li>Things that didn’t work <ul> <li>donations and other optional support - led to burnout</li> <li>consulting and services- took too much time away from core product</li> <li>grant funding - nice, but one time thing</li> </ul></li> <li>Lessons <ul> <li>You don't get extra points for being bootstrapped. Compete by doing things you can do better due to niche and size.</li> <li>Keeping trust in the community is the most important thing.</li> <li>Contribution is easier for less complex parts of the code base.</li> <li>Beign open source means capturing a small percentage of the value you create.</li> <li>You have to be ok doing more with less.</li> </ul></li> <li>Also <ul> <li>RtD is not just for Sphinx anymore. <ul> <li>Their build system <a href="https://docs.readthedocs.io/en/stable/build-customization.html?featured_on=pythonbytes">now supports any documentation tool</a>.</li> </ul></li> </ul></li> </ul> <p><strong>Michael #4:</strong> <a href="https://github.com/python-humanize/humanize?featured_on=pythonbytes">humanize</a></p> <ul> <li>by <a href="https://github.com/hugovk?featured_on=pythonbytes">Hugo van Kemenade</a> (Python 3.14 &amp; 3.15 release manager &amp; core developer)</li> <li>Not too many variations, but very handy if you need it. <ul> <li><a href="https://humanize.readthedocs.io/en/stable/number?featured_on=pythonbytes">Numbers</a> <ul> <li>Associated Press style (“seven” and “10”)</li> <li>Clamp (under 1.0 million)</li> <li>Fractional (1/3)</li> <li>Int Word (1.2 Billion)</li> <li>Metric (1.5 kV)</li> <li>Ordinal (112th)</li> <li>scientific</li> </ul></li> <li><a href="https://humanize.readthedocs.io/en/stable/time/?featured_on=pythonbytes">Time</a></li> <li><a href="https://humanize.readthedocs.io/en/stable/filesize/?featured_on=pythonbytes">File size</a></li> </ul></li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li><a href="https://testandcode.com?featured_on=pythonbytes">Test &amp; Code </a>is now again Test &amp; Code <ul> <li>The two part series on Python imports that started in June is finally complete with episode 222.</li> <li>Transcripts are being added to old episodes gradually starting from most recent <ul> <li>Back to ep 203 as of today. </li> <li>AI transcription, so there’s things like .pie, .pi, and dot pie where it should be .py </li> </ul></li> </ul></li> </ul> <p>Michael:</p> <ul> <li>Final final call for <a href="https://www.codeinacastle.com/python-zero-to-hero-2024?featured_on=pythonbytes">Coding in a Castle event</a> with Michael</li> <li><a href="https://bjango.com/mac/istatmenus/?featured_on=pythonbytes">iStats Menu</a></li> <li><a href="https://pypi.org/project/anacondacode/?featured_on=pythonbytes">Anaconda Code Runner</a> by Ruud van der Ham: <ul> <li>With Anaconda Coide we can -at last- run that code locally and import (most) Python modules.</li> <li>But if you want to run a significant amount of code, you have to put that in a cell or publish it to PyPI or a wheel and import it.</li> <li>That's why I have developed a general-purpose runner function that runs arbitrary code located on an Excel sheet with AnacondaCode.</li> </ul></li> </ul> <p><strong>Joke:</strong> <a href="https://devhumor.com/media/when-beginners-learn-a-new-programming-language?featured_on=pythonbytes">When beginners learn a new programming language...</a></p>

Service-oriented API task scheduling 2024-09-09, by Dariusz Suchojad

An integral part of Zato, its scalable, service-oriented scheduler makes it is possible to execute high-level API integration processes as background tasks. The scheduler runs periodic jobs which in turn trigger services and services are what is used to integrate systems.

Integration process

In this article we will check how to use the scheduler with three kinds of jobs, one-time, interval-based and Cron-style ones.

What we want to achieve is a sample yet fairly common use-case:

• Periodically consult a remote REST endpoint for new data
• Store data found in Redis
• Push data found as an e-mail attachment

Instead of, or in addition to, Redis or e-mail, we could use SQL and SMS, or MongoDB and AMQP or anything else - Redis and e-mail are just example technologies frequently used in data synchronisation processes that we use to highlight the workings of the scheduler.

No matter the input and output channels, the scheduler works always the same - a definition of a job is created and the job's underlying service is invoked according to the schedule. It is then up to the service to perform all the actions required in a given integration process.

Python code

Our integration service will read as below:

# -*- coding: utf-8 -*- # Zato from zato.common.api import SMTPMessage from zato.server.service import Service class SyncData(Service): name = 'api.scheduler.sync' def handle(self): # Which REST outgoing connection to use rest_out_name = 'My Data Source' # Which SMTP connection to send an email through smtp_out_name = 'My SMTP' # Who the recipient of the email will be smtp_to = 'hello@example.com' # Who to put on CC smtp_cc = 'hello.cc@example.com' # Now, let's get the new data from a remote endpoint .. # .. get a REST connection by name .. rest_conn = self.out.plain_http[rest_out_name].conn # .. download newest data .. data = rest_conn.get(self.cid).text # .. construct a new e-mail message .. message = SMTPMessage() message.subject = 'New data' message.body = 'Check attached data' # .. add recipients .. message.to = smtp_to message.cc = smtp_cc # .. attach the new data to the message .. message.attach('my.data.txt', data) # .. get an SMTP connection by name .. smtp_conn = self.email.smtp[smtp_out_name].conn # .. send the e-mail message with newest data .. smtp_conn.send(message) # .. and now store the data in Redis. self.kvdb.conn.set('newest.data', data)

Now, we just need to make it run periodically in background.

Mind the timezone

In the next steps, we will use the Zato Dashboard to configure new jobs for the scheduler.

Keep it mind that any date and time that you enter in web-admin is always interepreted to be in your web-admin user's timezone and this applies to the scheduler too - by default the timezone is UTC. You can change it by clicking Settings and picking the right timezone to make sure that the scheduled jobs run as expected.

It does not matter what timezone your Zato servers are in - they may be in different ones than the user that is configuring the jobs.

Endpoint definitions

First, let's use web-admin to define the endpoints that the service uses. Note that Redis does not need an explicit declaration because it is always available under "self.kvdb" in each service.

• Configuring outgoing REST APIs

• Configuring SMTP e-mail

Now, we can move on to the actual scheduler jobs.

Three types of jobs

To cover different integration needs, three types of jobs are available:

• One-time - fires once only at a specific date and time and then never runs again
• Interval-based - for periodic processes, can use any combination of weeks, days, hours, minutes and seconds for the interval
• Cron-style - similar to interval-based but uses the syntax of Cron for its configuration

One-time

Select one-time if the job should not be repeated after it runs once.

Interval-based

Select interval-based if the job should be repeated periodically. Note that such a job will by default run indefinitely but you can also specify after how many times it should stop, letting you to express concepts such as "Execute once per hour but for the next seven days".

Cron-style

Select cron-style if you are already familiar with the syntax of Cron or if you have some Cron tasks that you would like to migrate to Zato.

Running jobs manually

At times, it is convenient to run a job on demand, no matter what its schedule is and regardless of what type a particular job is. Web-admin lets you always execute a job directly. Simply find the job in the listing, click "Execute" and it will run immediately.

Extra context

It is very often useful to provide additional context data to a service that the scheduler runs - to achieve it, simply enter any arbitrary value in the "Extra" field when creating or an editing a job in web-admin.

Afterwards, that information will be available as self.request.raw_request in the service's handle method.

Reusability

There is nothing else required - all is done and the service will run in accordance with a job's schedule.

Yet, before concluding, observe that our integration service is completely reusable - there is nothing scheduler-specific in it despite the fact that we currently run it from the scheduler.

We could now invoke the service from command line. Or we could mount it on a REST, AMQP, WebSocket or trigger it from any other channel - exactly the same Python code will run in exactly the same fashion, without any new programming effort needed.

More resources

➤ Python API integration tutorial
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

Python's comments start with an octothorpe character.

Table of contents

1. Writing a comment in Python
2. Inline comments in Python
3. Best practices for commenting in Python
4. Comment as needed, but not too much

Writing a comment in Python

We have a Python program that prints out Hello!, pauses for a second, and then prints Goodbye! on the same line:

from time import sleep print("Hello!", end="", flush=True) sleep(1) # ANSI code to clear current line print("\r\033[K", end="") print("Goodbye!")

It prints Hello!:

~ $ python3 hello.py Hello!

And then one second later it overwrites Hello! with Goodbye!:

~ $ python3 hello.py Goodbye!

It does this using an ANSI escape code (that \033[K string).

The line above the print call in our code is called a comment:

# ANSI code to clear current line print("\r\033[K", end="")

Python's comments all start with the # character.

I call this character an octothorpe, though it goes by many names. Some of the more common names for # are hashmark, number sign, and pound sign.

You can write a comment in Python by putting an octothorpe character (#) at the beginning of a line, and then writing your comment. The comment stops at the end of the line, meaning the next line is code... unless you write another octothorpe character!

Here we've written more details and added an additional line to note that this code doesn't yet work on Windows:

# ANSI code to clear current line: \r moves to beginning, \033[K erases to end. # Note: This will not work on Windows without code to enable ANSI escape codes. print("\r\033[K", end="")

This is sometimes called a block comment because it's a way to write a block of text that represents a comment.

Unlike some programming languages, Python has no multiline comment syntax. If you think you've seen a multiline comment, it may have been a docstring or a multiline string. More on that in multiline comments in Python.

Inline comments in Python

Comments don't need to be …

Read the full article: https://www.pythonmorsels.com/commenting-in-python/

Something interesting that Dave Liddament and I discussed was the use of value objects in application code.

Instead of using a primitive type, such as string, you can create a new value object for a specific type of string, such as an EmailAddress or, in my side project, a LicenceKey.

Both are strings, but using value objects of specific types can make the code more readable and its intent clearer.

A value object can contain additional logic, such as validation to execute an ensure the value object is valid, such as making sure a string is not empty, is a specific length or only contains valid characters.

This an approach that I'm going to use more going forward.

I also found a lighting talk by Dave at a PHPSW meetup where he explains this further and, of course, you can listen to the podcast episode after it's been released.

After years and years of working together on KPhotoAlbum, a considerable part of the devs team (Johannes and me ;-) finally met in person, at Akademy in Würzburg!

It was a very nice and pleasureable meeting, with lots of information around KDE, e.g. community goals, where we stand with Qt 5 and 6 and where we want to go, programming, sustainability and so on. Throughoutly nice and friendly people (esp. the two of us of course ;-), which whom one could have nice and productive conversations. If you can, go to Akademy – it's worth it!

Also, we hopefully again could emphasize – in person – the importance a Qt6/KF6 port of Marble for KPhotoAlbum and also KGeoTag. We now actively work on porting KPA to QT6/KF6, but we need Marble to be able to finally release it. But we're confident everything will work out.

Hopefully, this won't be the last time we meet!

— Tobias

Now that uv is rapidly advancing I have started to dive back into making multi-version imports for Python work. The goal here is to enable multiple resolutions from the solver in uv so that two incompatible versions of a library can be installed and used simultaniously.

Simplified speaking it should be possible for a library to depend on both pydantic 1.x and 2.x simultaniously.

I have not made it work yet, but I have I think found all of the pieces that stand in the way. This post mostly exists to share how it could be done with the least amount of changes to Python.

Basic Operation

Python's import system places modules in a module cache. This cache is exposed via sys.modules. Every module that is imported is placed in that container prior to initialization. The key is the import path of the module. This in some ways presents the first issue.

Note on Terms for Packages, Modules and Distributions

Python's terms for packages are super confusing. Here is what I will use in this article:

• foo.py: this is a python “module”. It gets registered in sys.modules as 'foo' and has an attribute __name__ set to 'foo'.
• foo/__init__.py: declares also a Python “module” named 'foo' but it is simultaniously a “package”. Unlike a normal module it also has two extra attributes: __path__ which is set to ['./foo'] so that sub modules can be found and it has an attribute __package__ which is also set to 'foo' which marks it as package.
• Additionally on PyPI one can register things. These things were called packages at one point and are now mostly called "projects". Within Python however they are not called Projects but “distribution packages”. For instance this is what you see when you try to use the importlib.metadata API. For now I will just call this a “distribution”.

Note that a distribution can ship both modules and multiple at once. You could have a package called whatever and it reports a foo.py file and a bar/baz.py file which in turn would make foo and bar.baz be importable.

Say you have two Python distributions both of which provide the same toplevel package. In that case they are going to clash in sys.modules. As there is actually relationship of the distribution name to the entry in sys.modules this is a problem that does not just exist with multi version imports but it's one that does not happen all that much.

So let's say we have two distributions: foo@1.0.0 and foo@2.0.0. Both expose a toplevel module called foo which is a true Python package with a single __init__.py file. The installer would already fail to place these because one fully overrides the other.

So step 1 would be to place these modules in different places. So where they normally would be in site-packages, in this case we might want to not have these packages there. That solves us the file system clashes.

So we might place them in some extra cache that looks like this:

.venv/ multi-version-packages/ foo@1.0.0/ foo/ __init__.py foo@2.0.0/ foo/ __init__.py

Now that package is entirely non-importable since nothing looks at multi-version-packages. We will need a custom import hook to get them imported. That import hook will also need to change the name of what's stored in sys.modules.

So instead of registering foo as sys.modules['foo'] we might want to try to register it as sys.modules['foo@1.0.0'] and sys.modules['foo@2.0.0'] instead. There is however a catch and that is this very common pattern:

import sys def import_module(name): __import__(name) return sys.modules[name]

That poses a bit of a problem because someone is probably going to call this as import_module('foo') and now we would not find the entry in sys.modules.

This means that in addition to the new entries in sys.modules we would also need to register some proxies that “redirect” us to the real names. These proxies however would need to know if they point to 1.0.0 or 2.0.0.

Metadata

So let's deal with this problem first. How do we know if we need 1.0.0 or 2.0.0? The answer is most likely a package's dependenices. Instead of allowing a package to depend simultaniously on two different versions of the same dependency we can start with a much simpler problem and say that each package can only depend on one version. So that means if I have a myapp package it would have to pick between foo@1.0.0 or foo@2.0.0. However if it were to depended on another package (say slow-package) that one could depend on a different version of foo than myapp:

myapp v0.1.0 ├── foo v2.0.0 └── slow-package v0.1.0 └── foo v1.0.0

In that case when someone tries to import foo we would be consulting the package metadata of the calling package to figure out which version is attempted.

There are two challenges with this today and they come from the history of Python:

1. the import hook does not (always) know which module triggered the import
2. python modules do not know their distribution package

Let's look at these in detail.

Import Context

The goal is that when slow_package/__init__.py imports foo we get foo@1.0.0 version, when myapp/__init__.py improts foo we get the foo@2.0.0 version. What is needed for this to work is that the import system understands not just what is imported, but who is importing. In some sense Python has that. That's because __import__ (which is the entry point to the import machinery) gets the module globals. Here is what an import statement roughly maps to:

# highlevel import from foo import bar # under the hood _rv = __import__('foo', globals(), locals(), ['bar']) bar = _rv.bar

The name of the package that is importing can be retrieved by inspecting the globals(). So in theory for instance the import system could utilize this information. globals()['__name__'] would tell us slow_package vs myapp. There however is a catch and that is that the import name is not the distribution name. The PyPI package could be called mycompany-myapp and it exports a python package just called myapp. This happens very commonly in all kinds of ways. For instance on PyPI one installs Scikit-learn but the python package installed is sklearn.

There is however another problem and that is interpreter internals and C/Rust extensions. We have already established that Python packages will pass globals and locals when they import. But what do C extensions do? The most common internal import API is called PyImport_ImportModule and only takes a module name. Is this a problem? Do C extensions even import stuff? Yes they do. Here is an example from pygame:

MODINIT_DEFINE (color) { PyObject *colordict; colordict = PyImport_ImportModule ("pygame.colordict"); if (colordict) { PyObject *_dict = PyModule_GetDict (colordict); PyObject *colors = PyDict_GetItemString (_dict, "THECOLORS"); /* TODO */ } else { MODINIT_ERROR; } /* snip */ }

And that makes sense. A sufficiently large python package will have inter dependencies between the stuff written in C and Python. It's also complicated by the fact that the C module does initialize a module, but it does not have a natural module scope. The way the C extension initializes the module is with the PyModule_Create API:

static struct PyModuleDef module_def = { PyModuleDef_HEAD_INIT, "foo", /* name of module */ NULL, -1, SpamMethods }; PyMODINIT_FUNC PyInit_foo(void) { return PyModule_Create(&module_def); }

So both the name of the module created as well as the name of what is imported is entirely hardcoded. A C extension does not “know” what the intended name is, it must know this on its own.

In some sense this is already a bit of a disconnect beween the Python and C world. Python for instance has relative imports (from .foo import bar). This is implemented by inspecting the globals. There is however no API to do these relative imports on the C layer.

The only workaround I know right now would be to perform stack walking. That way one would try to isolate the shared library that triggered the import to understand which module it comes from. An alternative would be to carry the current C extension module that is active on the interpreter state, but that would most likely be quite expensive.

The goal would be to find out which .so/.dylib file triggered the import. Stack walking is a rather expensive operation and it can be incredibly brittle but there might not be a perfect way around it. Ideally Python would at any point know which c extension module is active.

Distributions from Modules

So let's say that we have the calling python module figured out: now we need to figure out the associated PyPI distribution name. Unfortunately such a mapping does not exist at all. Ideally when a sys.module entry is created, we either record a special attribute there (say __distribution__) which carries the name of the PyPI distribution name so we can call importlib.metadata.distribution(__distribution__).requires to get the requirements or we have some other API to map it.

In the absence of that, how could we get it? There is an expensive way to get a reverse mapping (importlib.metadata.packages_distributions) but unfortunately it has some limitations:

1. it's very slow
2. it has situations where it does not manage to reveal the distribution for a package
3. it can reveal more than one distribution for a package

Because of namespace packages in particular it can return more than one distribution that provides a package such as foo (eg: foo-bar provides foo.bar and foo-baz provides foo.baz. In that case it will just return both foo-bar and foo-baz for foo).

The solution here might just be that installers like uv start materializing the distribution name onto the modules in one way or another.

Putting it Together

The end to end solution might be this:

1. install multi-version packages outside of site-packages
2. materialize a __distribution__ field onto modules or provide an API that maps import names to their PyPI distribution name so that meta data (requirements) can be discovered.
3. patch __import__ to resolve packages to their fully-qualified, multi version name based on who imports it
   • via globals() for python code
   • via stack-walking for C extensions (unless a better option is found)
4. register proxy entries in sys.modules that have a dynamic __getattr__ which redirects to the fully qualified names if necessary. This would allow someone to access sys.modules['foo'] and automatically proxy it to foo@1.0.0 or foo@2.0.0 respectively.

There are lots of holes with this approach unfortunately. That's in parts because people patch around in sys.modules. Interestingly enough sys.modules can be manipulated but it can't be replaced. This might make it possible to replace that dictionary with some more magical dictionary in future versions of Python potentially.

FTP master

This month I accepted 441 and rejected 15 packages. The overall number of packages that got accepted was 442.

I am ashamed of some occurrences that happened this month and I apologize for this. Unfortunately I have no idea how to prevent this in the future without becoming a solo entertainer.

Debian LTS

This was my hundred-twenty-second month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian.

• [#1073518] bookworm-pu: cups 2.4.2-3+deb12u6 has been closed
• [#1074439] bookworm-pu: cups 2.4.2-3+deb12u7 has been closed
• [#1073519] bullseye-pu: cups 2.3.3op2-3+deb11u7 has been closed
• [#1074438] bullseye-pu: cups 2.3.3op2-3+deb11u8 has been closed

Unfortunately Bullseye was not handed over to LTS in August. So I only prepared new packages of asterisk, libvirt and tinyproxy and will upload them next month.

Last but not least I did a week of FD this month.

Debian ELTS

This month was the seventy-third ELTS month. During my allocated time I uploaded or worked on:

• [ELA-1160-1]tiff security update for two CVEs in Jessie and Stretch. The Buster upload was already done before. This upload fixed a segmentation fault and a memory leak
• [ELA-1161-1]libvirt security update for six CVEs to fix issues related to use-after-free, an off-by-one, a null pointer dereference, a badly handled mutex, a privilege escalation and breaking out of the sVirt confinement. In this case only Jessie and Stretch needed an update.
• [ELA-1166-1]frr security update for one CVEs in Buster to fix a missing length check.

I also did a week of FD.

Debian Printing

This month I uploaded …

• … gutenprint to fix a gcc14 issue
• … ipp-usb to fix a /usr-merge issue

This work is generously funded by Freexian!

Debian Astro

This month I uploaded a new upstream or bugfix version of:

• … libinovasdk to better add udev rules
• … indi-playerone to fix dependencies
• … libsbig to fix a /usr-move issue

Debian Mobcom

The following packages have been prepared by the GSoC student Nathan:

• … libosmo-netif
• … osmo-bts
• … libosmo-sccp
• … libosmo-abis
• … osmo-hlr
• … osmo-mgw
• … osmo-bsc
• … osmo-iuh
• … osmo-ggsn
• … osmo-msc
• … osmo-sgsn

It was so much fun working with Nathan. Unfortunately GSoC is over now, but Nathan will continue working in Debian and become a Debian Maintainer.

misc

This month I uploaded new upstream or bugfix versions of:

• … mpb to fix a gcc14 issue
• … httperf to fix a gcc14 issue
• … pkcs11-proxy to fix a gcc14 issue
• … meep-mpi-default to fix some bugs
• … gnucobol3 to upload a new upstream version
• … bibclean to fix a gcc14 issue
• … apcupsd to upload from experimental to unstable
• … rplay to fix a gcc14 issue
• … scheme48 to remove some debug stuff
• … gnupg-pksc11-scd to fix an FTBFS with assuan 3

I also filed an RM bug against meep-openmpi. As Adrian made me ware, this package is no longer needed.

Stow 2.4.1 has been released.  This release contains some minor bug-fixes -- specifically, fixing the --dotfiles option to work correctly with ignore lists, allowing options in .stowrc with spaces, and avoiding a spurious warning on Perl >= 5.40.  There were also some clean-ups and improvements, mostly internal and not visible to users.  Read details of what's new: http://git.savannah.gnu.org/cgit/stow.git/tree/NEWS

Suppose I have this Makefile:

a: b touch $@ b: touch $@ # A common chain of build steps %-GENERATED.c: %-generate touch $@ %.o: %.c touch $@ %.so: %-GENERATED.o touch $@ xxx-GENERATED.o: CFLAGS += adsf # Imitates .d files created with "gcc -MMD". Does not exist on the initial build ifneq ($(wildcard xxx.so),) xxx-GENERATED.o: xxx-GENERATED.c endif

This is all very simple build-system stuff. Let's see how it works:

$ rm -rf a b xxx-GENERATED.c xxx-GENERATED.o xxx.so [start from a clean slate] $ touch xxx-generate xxx.h [Files that would be available in a project exist; xxx-generate is some tool] [that would generate xxx-GENERATED.c ] $ touch a ["a" exists but the file "b" it depends on does not] $ make a xxx.so touch b touch a touch xxx-GENERATED.c touch xxx-GENERATED.o touch xxx.so rm xxx-GENERATED.c [It built everything, but then deleted xxx-GENERATED.c] $ make a xxx.so remake: 'a' is up to date. touch xxx-GENERATED.c touch xxx-GENERATED.o touch xxx.so [It knew to not rebuild "a", but the missing xxx-GENERATED.c caused it to] [re-build stuff ]

Well that's not good. What if we add .SECONDARY: to the end of the Makefile to mark everything as a secondary file?

$ rm -rf a b xxx-GENERATED.c xxx-GENERATED.o xxx.so $ touch xxx-generate xxx.h $ touch a $ make a xxx.so remake: 'a' is up to date. touch xxx-GENERATED.c touch xxx-GENERATED.o touch xxx.so [It didn't bother rebuilding "a" even though its prerequisites "b" doesn't] [exist. But it didn't delete the xxx-GENERATED.c at least ] $ make a xxx.so remake: 'a' is up to date. remake: 'xxx.so' is up to date. [It knew to not rebuild anything. Great.]

So it doesn't work right with or without .SECONDARY:, but it's much closer with it. The solution is to mark everything as not an intermediate file. mrbuild cannot do this without a bleeding-edge version of GNU Make, but users of mrbuild can do this by explicitly mentioning specific files in rules. This would suffice:

___dummy___: file1 file2

Detailed notes are in a commit in mrbuild (mrbuild 1.13) and in a post to LKML by Masahiro Yamada.