Feeds

What changes are happening under the hood in the latest versions of Python? How are these updates laying the groundwork for a faster Python in the coming years? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Learn how to turn Drupal into a scalable publishing platform that developers, marketers & editors love while meeting your regulatory & compliance requirements.

The diffoscope maintainers are pleased to announce the release of diffoscope version 282. This version includes the following changes:

[ Chris Lamb ] * Ignore errors when listing .ar archives. (Closes: #1085257) * Update copyright years.

You find out more by visiting the project homepage.

After reading about Jonathan McDowell feed reader install and the back to blogging initiative, I decided to install a feed reader to follow all those nice blog posts. With a feed reader you can compose your own feed of news based on blog posts, websites, mastodon toots. And then you are independant from ad oriented ranking algorithms of social networks.

Since Jonathan used FreshRSS as a feed reader, I started with the same software. On a quick glance on its github page, it sounded like a good project:

• active contributions
• different channels for stable and latest version of the software
• container images pointing to the stable release
• support multiple databases for storage, including PostgreSQL
• correct documentation mentioning security caveats

I prefer to do the container image installation using podman since:

• upgrades from FreshRSS are easy to do and can be done separately from operating system upgrades
• I do not mess my based operating system with php (subjective) and in case of a compromized freshrss, the freshrss/apache install would be still restrained to its own Linux namespaces, separated from the rest of the system.

Podman is image compatible with Docker as they both implement the OCI runtime specification, and have a nearly identical command line interface. This installation will be done on a Debian server, but should work too on any Linux distribution.

Initial setup

• start a container image based on the start command provided by the FreshRSS project. The podman command line is nearly identical to the docker command line, excepts that podman expects the fully qualified domain name associated with the container image, and I chose to run the freshrss container on the localhost interface only. I also use a defined version tag, because using the latest tag makes it complicated to track which exact ersion I have installed.

# podman pull docker.io/freshrss/freshrss:1.20.1 # podman run --detach --restart unless-stopped --log-opt max-size=10m \ --publish 127.0.0.1:8081:80 \ --env TZ=Europe/Paris \ --env 'CRON_MIN=1,31' \ --volume freshrss_data:/var/www/FreshRSS/data \ --volume freshrss_extensions:/var/www/FreshRSS/extensions \ --name freshrss \ docker.io/freshrss/freshrss:1.20.1

• verify where the podman volumes have been created. This is where the user data of freshrss will be stored.

# podman volume ls # podman volume inspect freshrss_data

• now that freshrss is installed, you can start its configuration wizard at localhost:8081. You should keep the default sqlite choice
• finally after running the wizard, you can login again and add some feeds
• verify that your config has been stored outside the container, and inside the volume (so that it will not be erased in case of upgrages)

# ls -l /var/lib/containers/storage/volumes/freshrss_data/_data/users/

• verify the state of sqlite database

echo '.tables'| sqlite3 /var/lib/containers/storage/volumes/freshrss_data/_data/users/<your freshrss user>/db.sqlite category entry entrytag entrytmp feed tag Going with FreshRSS in Production

Podman has this very nice feature that it can generate a systemd unit from a running container, and use systemd to start a container on boot. This is in contrary to docker where the docker daemon does the stop/start of containers on boot. I prefer the systemd approach as it treats containers the same way as other system services.

Once the freshrss container is running we can generate a systemd unit of it with:

# podman generate systemd --new --name freshrss | tee /etc/systemd/system/container-freshrss.service

Let’s stop the container we started previously, and use systemd to manage it:

# podman stop freshrss # systemctl enable --now container-freshrss.service

We can verify that we have a listening socket on the localhost interface, on the source port 8081

# systemctl status container-freshrss.service ... # ss --listening --numeric --process '( sport = 8081 )' Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process tcp LISTEN 0 4096 127.0.0.1:8081 0.0.0.0:* users:(("conmon",pid=4464,fd=5))

Nota Bene: conmon (8) is the process managing the network namespace in which fresh-rss is running, hence it is displayed as the process owning the listening socket

Exposing FreshRSS to the external world

We have now a running service, but we need to make it reachable from the internet. The simplest, classical way, is to create a subdomain and a VirtualHost configured as a reverse proxy to access the service at 127.0.0.1:8081. Fortunately the FreshRSS authors have documented this setup in https://github.com/FreshRSS/FreshRSS/tree/edge/Docker#alternative-reverse-proxy-using-apache and those steps are no different from a standard application behind a web reverse proxy.

Upgrading freshrss container to a newer version

A documentation showing how to install a piece of software is nothing when it does not show how to upgrade that said software. Installing is easy, upgrading is where the challenge is. Fortunately to the good stateless design of freshrss (everything is in the sqlite database, which is backed by a non-epheremal volume in our setup), switchting versions is a peace of cake.

# podman pull docker.io/freshrss/freshrss:1.20.2 # systemctl stop container-freshrss.service # sed -i 's,docker.io/freshrss/freshrss:1.20.1,docker.io/freshrss/freshrss:1.20.2,' /etc/systemd/system/container-freshrss.service # systemctl daemon-reload # systemctl start container-freshrss.service

If you need to rollback, you just need to revert version numbers in the instruction above.

Enjoy your own reader feed !

I will add the following feeds of blogs I like, let us see if I follow them better with a feed reader !

An effective Request for Proposals (RFP) or Call for Proposals (CFP) not only outlines the goals and expectations of your project but also defines the framework within which potential vendors must operate. It goes beyond simply finding the right vendor to build your website or deliver a content management system (CMS) tailored to your needs—it's an opportunity to establish a partnership, support open source software, and contribute to a vibrant community ecosystem.

For many organizations, choosing open source software isn’t just a preference—it’s a strategic imperative. The advantages of free and open source software (FOSS) include cost savings, solutions tailored precisely to your organization’s needs, and robust security, strengthened by a vigilant community.

In this blog post, we’ll guide you through crafting an RFP that prioritizes open source solutions while tapping into the expertise of Drupal Certified Partners. We also offer a free, downloadable RFP template to help streamline the process, ensuring your project specifications attract top-tier vendors dedicated to innovation and contributing to the Drupal community.

The advantages of open source software

• Cost savings: Open source eliminates hefty licensing fees, allowing organizations to allocate resources more efficiently. While there may be costs associated with customization and maintenance, the overall financial burden is often significantly lower.
• Flexibility and extensibility: Open source platforms can be tailored to meet specific organizational needs. With access to the source code, developers can modify and extend functionalities without waiting for vendor updates or feature requests.
• Enhanced security: Open source communities actively monitor and address security vulnerabilities. The collaborative nature ensures that security patches and updates are promptly developed and deployed. 
• Alignment with organizational values: Open source promotes transparency, collaboration, and community-driven development. Organizations that prioritize these values find open source solutions to be a natural fit.
• Case study: Swiss Government's open source mandate

A notable example of strategic open source adoption is the Swiss government's recent decision to prioritize open source solutions in public sector projects. This mandate not only underscores the benefits of open source but also sets a precedent for other governmental bodies. By embracing open source, the Swiss government aims to enhance transparency, reduce costs, and foster innovation within its digital infrastructure.

Finding the ideal service provider

Finding the right service provider that aligns with your vision is crucial to the success of your project. The right partner not only brings the necessary technical expertise but also understands your long-term goals, ensures smooth collaboration, and shares your commitment to quality and innovation. A well-aligned service provider becomes a trusted partner, invested in both your immediate needs and your future growth.

Here's why partnering with Drupal Certified Partners makes a significant difference:

• Rigorous certification process: The Drupal Association evaluates potential partners based on their contributions to Drupal core, contributed modules, and themes. This ensures that only the most dedicated and skilled agencies receive certification.
• Proven track record: Certified Partners have a history of successful Drupal implementations, showcasing their ability to handle complex projects with efficiency and expertise.
• Commitment to the community: These partners actively contribute to the Drupal project through code contributions, module development, and participation and sponsorship in Drupal events and initiatives.
• Verifiable capabilities: The Drupal Association provides verified letters of recommendation for Certified Partners to include in RFP responses, giving procurement teams trusted verification of their skills and commitment to the Drupal ecosystem.

Certified partners are contributing experts

When drafting your Request for Proposals (RFP) or tender, specifying a preference for officially certified implementation partners — such as Drupal Certified Partners — can dramatically elevate the quality of vendor responses. Drupal Certified Partners are distinguished not only by their expertise in deploying Drupal solutions but also by their active contributions to the Drupal project itself. This dual commitment ensures that these partners are intimately familiar with the latest developments in Drupal, enabling them to deliver solutions that are both innovative and sustainable. Moreover, by requiring a Drupal Certified Partner, organizations directly support vendor involvement with the open source community, fostering a collaborative ecosystem that drives continuous improvement and long-term success.

The flywheel effect: How partner contributions benefit everyone

Choosing a Drupal Certified Partner also supports the broader Drupal project by empowering top contributors to maintain and enhance the platform that underpins your organization's digital presence. These partners often invest more resources into contributing to Drupal core, contributed modules, and themes than they do into traditional marketing efforts. This investment creates a "flywheel" effect: as partners develop new features or improvements to meet your specific needs, these enhancements are reintegrated into the Drupal community, benefiting all users and ensuring the platform remains cutting-edge and secure. You benefit as well, though, as the community jumps on board to test, extend, maintain, and update the code that you (through your partner) contributed. This makes your code better in the long run at no additional cost to you.

About the Drupal Certified Partner program

The Drupal Association, a nonprofit organization dedicated to promoting and sustaining the Drupal project, plays a crucial role in identifying and certifying these top-tier partners. Through evaluation of their contributions to Drupal core, contributed modules, and themes, the Drupal Association designates certain agencies as Drupal Certified Partners. This certification not only recognizes their technical prowess and commitment to the Drupal ecosystem but also provides procurers with verified attestations of their capabilities, simplifying the vendor selection process.

Testimonials and success stories

Organizations that have partnered with Drupal Certified Partners consistently report higher satisfaction levels, smoother project executions, and more robust and scalable solutions. These partners bring not only technical expertise but also a collaborative spirit that aligns with the open source philosophy, ensuring that projects are both innovative and sustainable.

Crafting your RFP for success

An effective RFP not only clearly defines your requirements and expectations, it also sets the boundaries within which potential vendors must operate. For example, specifying the need for mobile-responsive design ensures all proposals meet modern accessibility standards, while outlining strict data security requirements guarantees vendors prioritize protecting sensitive information. Additionally, specifying a preference for open source software like Drupal can impact your project's flexibility, cost, and alignment with organizational values.

Here's how to structure your RFP to prioritize open source solutions and Drupal Certified Partners:

1. Define project goals and objectives

   • Clearly outline what you aim to achieve with your website redesign or CMS selection.

   • Include specific functionalities, design preferences, and performance metrics.

2. Specify open source requirements

   • Highlight the importance of using open source software.

   • Explain how open source aligns with your organization’s values and strategy.

3. Mandate Drupal Certified Partner certification

   • State that only proposals from Drupal Certified Partners will be considered.

   • Provide information about the certification and its significance.

4. Outline evaluation criteria

   • Detail how proposals will be assessed, focusing on contributions to Drupal.

   • Include criteria such as technical expertise, project management skills, and community involvement.

5. Provide a clear timeline and budget

   • Offer realistic deadlines and budget ranges.

   • Allow flexibility for high-quality vendors to propose innovative solutions.

6. Include legal and compliance requirements

   • Address legal considerations such as data protection and accessibility standards.

7. Offer resources and support

   • Provide access to your organization’s content, branding guidelines, and technical documentation.

   • Encourage collaboration and ongoing communication.

Jumpstart your RFP with a ready-made template

The Drupal Association is proud to offer a downloadable RFP template tailored for open source website design and CMS selection projects. This template includes all the essential sections outlined above, along with customizable fields to suit your organization's unique needs. The template is also applicable to Request for Quotation (RFQ), Invitation to Bid (ITB), Request for Information (RFI), and Request for Tender (RFT) procurement processes. 

Download the open source RFP template

Many thanks to Vardot, a Drupal Certified Partner, for providing the inspiration for this post and the initial version of the template!

Strategies for evaluating vendor proposals

Evaluating vendor proposals can be daunting, especially when faced with lengthy submissions or a high volume of responses. A common approach is to use a weighted scoring system to compare proposals based on key criteria while ensuring your priorities and values are accounted for. Keep in mind that the best fit may not meet every criterion perfectly, but a vendor who aligns with your organization’s values and fully understands your vision can offer the greatest long-term success. 

Use these strategies to ensure a thorough assessment:

• Alignment with goals: Make sure the proposal clearly aligns with your project’s goals and objectives.
• Technical expertise: Assess the vendor's technical capabilities and experience with Drupal. Have they successfully delivered projects for clients similar to yours in size and industry? Looks for published case studies to verify their claims.
• Community contributions: Check the vendor's contributions to the Drupal project. Their involvement can demonstrate both commitment and expertise. From the vendor's page on Drupal.org, you can see if they have contributed to or maintained modules that may be essential to your project.
• References and case studies: Review client testimonials and case studies to gauge the vendor's reliability and quality of work. Drupal.org publishes case studies for Drupal Certified Partners to showcase their success stories.
• Long-term support: elect vendors who offer ongoing support and maintenance to keep your website secure, up-to-date, and adaptable to future needs.

Get the results you want with a targeted RFP

A well-crafted RFP is the foundation of a successful website redesign or CMS selection project. By prioritizing open source solutions and requiring Drupal Certified Partner certification, you ensure that your project is handled by capable vendors committed to both your success and the open source community. This approach not only enhances the quality and sustainability of your project but also supports the broader Drupal community, fostering an environment of continuous improvement and innovation.

Ready to create an effective RFP that attracts top-tier Drupal Certified Partners? Download our comprehensive RFP template today and take the first step towards a successful, sustainable, and community-driven project.

Start your project with confidence with this RFP Template!

As I said on linked in, this week my brain was fried and also buzzing while working on getting a beta release of the LocalGov Live Preview module.

In this article, I'll show you how to import data from CSV files into Drupal. This is often a step in creating a new website, and the solutions shown here will come in handy when transferring data from an old system to a new one. I’ll also demonstrate how to handle such a situation using the Feeds module and its Feeds Tamper extension and walk you through importing data using three types of data as examples: products, users, and blog posts.

In this quiz, you’ll test your understanding of The Python Standard REPL: Try Out Code and Ideas Quickly.

The Python REPL allows you to run Python code interactively, which is useful for testing new ideas, exploring libraries, refactoring and debugging code, and trying out examples.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Implementing Pagination in Drupal admin Thu, 10/24/2024 - 14:43

Changing your mind about dependency injection

When I start writing a class that has a dependency injection, I have a clear idea about which services it needs. I generate it -- the plugin, form, controller, or service -- and specify those services.

Then nearly always, unless it's something really very simple, I find that no matter how much I thought about it and planned it, I need to add more services. Maybe remove some too.

Fortunately, because Module Builder saves the configuration of the module code you've generated, it's easy to go back to it and edit it to add more services:

1. Edit your module in Module Builder
2. Add to the injected services for your component
3. Ensure your code file is committed to version control
4. Generate the code, and write the updated version of the code file
5. Add and commit the new DI code, while discarding the changes that remove your code. (I find it helps to use a git GUI for things like this, though git add -p works too.)

But I tend to find that I make this mistake several times as the class developers, and so I adopt the approach of using the \Drupal::service() function to get my services, and only when I'm fairly confident I'm not going to need to make any more changes to DI, I update the injected services in one go, converting all the service calls to use the service properties.

I was asked yesterday at Drupal Drinks about how to do that, and it occurred to me that there's a way of doing this so after you've updated the dependency injection with Module Builder, it's a simple find and replace to update your code.

If you write your code like this whenever you need a service:

$service_entityTypeManager = \Drupal::service('entity_type.manager'); $stuff = $service_entityTypeManager->doSomething();

Then you need to do only two find and replace operations to convert this to DI:

1. Replace '^.+Drupal::service.+\n' with ''. This removes all the lines where you get the service from the Drupal class.
2. Replace '\$service_(\w+)' with '$this->$1'. This replaces all the service variables with the class property.

Up until now I'd been calling the service variables something like $entityTypeManager so that I could easily change that to $this->entityTypeManager manually, but prefixing the variable name with a camel case 'service_' gives you something to find with a regular expression.

If you want to be really fancy, you can use a regular expression like '(?<=::service..)[\w.]+' (using dots to avoid having to escape the open bracket and the quote mark) to find all the services that you need to add to the class's dependency injection.

Something like this:

$ ag -G MyClass.php '(?<=::service..)[\w.]+' -o --nonumbers --nofilename | sort | uniq | tr "\n" ", "

will give you a list of service names that you can copy-paste into the Module Builder form. This is probably overkill for something you can do pretty quickly with the search in a text editor or IDE, but it's a nice illustration of the power of unix tools: ag has options to output just the found text, then sort and uniq eliminate duplicates, and finally tr turns it into a comma-separated list.

joachim Thu, 10/24/2024 - 11:25 Tags

• dependency injection
• drupal code builder
• module builder

Drupal 7: Navigating the Maintenance Maze bhavinhjoshi Thu, 10/24/2024 - 15:52

As the technology advances, and where change is the only constant, the question arises: why remain anchored to Drupal 7? This familiar platform has served many well, but as digital ecosystems grow, the costs of inaction multiply. The world of web development is never static, and adhering to outdated technology can impede your progress and competitive edge.

Let's dive deeper into the specific challenges and opportunities for those still using Drupal 7. One crucial aspect is the complexity of customisations made out of necessity during its heyday. While these custom modules and tweaks may have added significant value at the time, they now serve as barriers to progress—highlighting the risks of deviating from best practices.

 

 

Custom modules often become the lifeblood of a Drupal 7 site, but they also present hurdles. Each line of bespoke code demands scrutiny. This isn't merely a case of asking whether the customisations still function, but whether they are still fit for purpose in a more modern context. As we look towards Drupal 10 and beyond, it's worth questioning whether there exists a current core or contributed module that could replace complex custom work. This requires an expert evaluation, which isn’t optional but essential for a smooth migration. Understanding the intent and function of each customisation empowers you to streamline and possibly automate future updates.

Contributed modules add another layer of complexity. While it's tempting to wait for updates, proactive engagement within the Drupal community can unlock solutions, paving the way for an easier migration. The Drupal community is one of the platform's strongest assets—by getting involved, you can influence the direction of module development and even participate in updating modules critical to your website. This active participation ensures that you’re not only a passive consumer of someone else's labour but an active contributor to your digital landscape.

Security remains a paramount concern. Maintaining Drupal 7 involves more than just patching security holes. It’s about optimising performance, refining user experience, and bolstering defences. Old modules can introduce vulnerabilities, and outdated practices might not comply with current security standards. Here, the role of a dedicated hosting expert becomes invaluable. Collaborating with hosting partners who specialise in Drupal can ease the administrative burden, ensuring your site remains secure and efficient without derailing your internal resources.

The decision to stay with Drupal 7 should be informed by strategic foresight, not stagnation. It’s about recognising the potential for transformation rather than clinging to what's comfortable. Embracing newer versions does more than just upgrade your tech stack—it revitalises your business processes and user interactions.

An upgrade to Drupal 10 isn't just a technical necessity; it's a strategic advantage. It positions your organisation to leverage the latest innovations, enhance security, and improve scalability. The transition is not merely a swap of versions but an opportunity to realign with the evolving digital landscape.

Migration may seem daunting, but with a clear strategy, it becomes a journey of growth. Start by assessing your existing architecture, identify key stakeholders, and set clear objectives. Assemble a team of experts who can guide this transition, ensuring that each step aligns with your business goals.

In the end, Drupal 7's dilemma isn't just about the platform itself. It's a reflection of how we choose to adapt to change. Commit to evolution and unlock the full potential of what modern technology can offer. The future awaits those willing to embrace it with open arms and strategic intent.

Drupal 7 Drupal Drupal Planet Add new comment

 

The PSF is pleased to announce its second batch of PSF Fellows for 2024! Let us welcome the new PSF Fellows for Q2! The following people continue to do amazing things for the Python community:

Leonard Richardson

Blog

Winnie Ke 

Facebook, LinkedIn

Thank you for your continued contributions. We have added you to our Fellow roster.

The above members help support the Python ecosystem by being phenomenal leaders, sustaining the growth of the Python scientific community, maintaining virtual Python communities, maintaining Python libraries, creating educational material, organizing Python events and conferences, starting Python communities in local regions, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.

Let's continue recognizing Pythonistas all over the world for their impact on our community. The criteria for Fellow members is available online: https://www.python.org/psf/fellows/. If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. Quarter 3 nominations are currently in review. We are accepting nominations for Quarter 4 through November 20th, 2024.

Are you a PSF Fellow and want to help the Work Group review nominations? Contact us at psf-fellow at python.org.

Do you struggle to make sure your code is always correct before you check it in? What about your team members' code? That one person who never wants to run the linter? Tired of dealing with tons of conflicts and spurious git changes? You need git pre-commit hooks. We're lucky to have Stefanie Molin on this episode who has done a bunch of writing and teaching of git hooks.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/bluehost'>Bluehost</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Stefanie Molin</b>: <a href="https://stefaniemolin.com/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <br/> <b>Talk Python Blog</b>: <a href="https://talkpython.fm/blog/" target="_blank" >talkpython.fm/blog</a><br/> <br/> <b>How to Set Up Pre-Commit Hooks</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/setup-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Common Pre-Commit Errors and How to Solve Them</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/troubleshooting-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>A Behind-the-Scenes Look at How Pre-Commit Works</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/behind-the-scenes/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Pre-Commit Hook Creation Guide</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/hook-creation-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>(Pre-)Commit to Better Code Workshop</b>: <a href="https://stefaniemolin.com/workshops/pre-commit-workshop/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>exif-stripper</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/exif-stripper/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>exif-stripper on GitHub</b>: <a href="https://github.com/stefmolin/exif-stripper?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>docstring-validation-using-pre-commit-hook</b>: <a href="https://numpydoc.readthedocs.io/en/latest/validation.html#docstring-validation-using-pre-commit-hook" target="_blank" >numpydoc.readthedocs.io</a><br/> <b>Data Morph: Moving Beyond the Datasaurus Dozen</b>: <a href="https://stefaniemolin.com/articles/data-science/introducing-data-morph/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Data Morph on GitHub</b>: <a href="https://github.com/stefmolin/data-morph?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=EzlzX1OL92w" target="_blank" >youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/482/pre-commit-hooks-for-python-devs" target="_blank" >talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

NOTE: pacman v7 is currently in [libre-testing]; but it will be promoted to libre soon

from arch:

With the release of [version 7.0.0] pacman has added support for downloading packages as a separate user with dropped privileges.

For users with local repos however this might imply that the download user does not have access to the files in question, which can be fixed by assigning the files and folder to the alpm group and ensuring the executable bit (+x) is set on the folders in question.

$ chown :alpm -R /path/to/local/repo

Remember to [merge the .pacnew] files to apply the new default.

Pacman also introduced [a change] to improve checksum stability for git repos that utilize .gitattributes files. This might require a one-time checksum change for PKGBUILDs that use git sources.

What does “pushing code” mean for your business? Read on to discover how easy it is to get your code from development to production without headaches.

Posted on October 24, 2024
Tags: madeof:atoms, madeof:bits, craft:zine

I have no idea either.

Happy Maladay1 to those who celebrate it, I guess.

If you care about the how, it started as china ink on tracing paper, with the help of a template (and a correction sheet for one page where I used the wrong line on the template).

A rubber stamp was carved with the author’s signature and stamped on white paper because the ink from the pad wasn’t working well on tracing paper.

Then everything was scanned (with the correction on top of the wrong page) asemic_zine_scans.tar.

Imported in Inkscape and traced asemic_zine_svg.tar.

Printed, cut in half, folded and stapled. The magenta lines weren’t by design, but are there because my printer is currently2 cursed.

And finally, asemic_zine.pdf was created, joining the pages together with pdfjam, for convenience in case somebody wants to download the full thing.

All the .tar and .pdf downloads from this page are released under the WTFPL, or All Rites Reversed..

1. it’s still technically Maladay when I write this, even if by the time you’ll get this it’s probably the 6th of The Aftermath.↩︎

2. I mean, all printers are always cursed, but at different times they can be cursed in different and novel ways.↩︎

Learn to develop an effective content strategy for your Drupal site with expert advice. Drive traffic, engage users, and boost conversions with strategic content planning.

Introduction

A content strategy is essential for any modern Drupal site as it aligns content with business goals, optimizes user experience, ensures quality and consistency, and leverages Drupal's advanced content management features for efficient governance and workflow management. Your organization’s content strategy will incorporate SEO best practices to amplify online visibility and prepare the organization for future digital trends and multi-channel distribution. 

Ultimately, a well-crafted content strategy is a linchpin in transforming a Drupal CMS into a dynamic asset that supports engagement, lead conversion, and customer loyalty, thus driving an organization's digital success.

I’m going to change up the Calamares release process a little. It’s been slow going as a community-maintained project – which isn’t to say that that is a bad thing. Just slow. I’ve decided to make releases marginally more predictable than “when [ade] has a relaxed kind of Tuesday” and have marked a couple of issues with the Calamares 3.3 milestone. When the milestone is empty again, then there will be a release. After the next release, I’ll put a couple more issues on the milestone, and the recipe can be repeated.

Many years ago I was involved in software-quality research – the SQO-OSS project and things like that. That work begat the code-quality checking scripts that we in the KDE community called “the EBN”, or EnglishBreakfastNetwork. I was a tea-drinker then. The EBN stuff has been surpassed by Klazy and many other software-quality-checking tools. But the EBN domain carries on. Although I haven’t got anything to put on it I just renewed the domain again for two years – just in case.

As is probably apparent from the sequence of blog posts about the topic in the last year, I have been thinking about and working on integer optimizations in the JIT compiler a lot. This work was mainly motivated by Pydrofoil, where integer operations matter a lot more than for your typical Python program.

In this post I'll describe my most recent change, which is a new small domain specific language that I implemented to specify peephole optimizations on integer operations in the JIT. It uses pattern matching to specify how (sequences of) integer operations should be simplified and optimized. The rules are then compiled to RPython code that then becomes part of the JIT's optimization passes.

To make it less likely to introduce incorrect optimizations into the JIT, the rules are automatically proven correct with Z3 as part of the build process (for a more hands-on intro to how that works you can look at the knownbits post). In this blog post I want to motivate why I introduced the DSL and give an introduction to how it works.

Motivation

This summer, after I wrote my scripts to mine JIT traces for missed optimization opportunities, I started implementing a few of the integer peephole rewrite that the script identified. Unfortunately, doing so led to the problem that the way we express these rewrites up to now is very imperative and verbose. Here's a snippet of RPython code that shows some rewrites for integer multiplication (look at the comments to see what the different parts actually do). You don't need to understand the code in detail, but basically it's in very imperative style and there's quite a lot of boilerplate.

def optimize_INT_MUL(self, op): arg0 = get_box_replacement(op.getarg(0)) b0 = self.getintbound(arg0) arg1 = get_box_replacement(op.getarg(1)) b1 = self.getintbound(arg1) if b0.known_eq_const(1): # 1 * x == x self.make_equal_to(op, arg1) elif b1.known_eq_const(1): # x * 1 == x self.make_equal_to(op, arg0) elif b0.known_eq_const(0) or b1.known_eq_const(0): # 0 * x == x * 0 == 0 self.make_constant_int(op, 0) else: for lhs, rhs in [(arg0, arg1), (arg1, arg0)]: lh_info = self.getintbound(lhs) if lh_info.is_constant(): x = lh_info.get_constant_int() if x & (x - 1) == 0: # x * (2 ** c) == x << c new_rhs = ConstInt(highest_bit(lh_info.get_constant_int())) op = self.replace_op_with(op, rop.INT_LSHIFT, args=[rhs, new_rhs]) self.optimizer.send_extra_operation(op) return elif x == -1: # x * -1 == -x op = self.replace_op_with(op, rop.INT_NEG, args=[rhs]) self.optimizer.send_extra_operation(op) return else: # x * (1 << y) == x << y shiftop = self.optimizer.as_operation(get_box_replacement(lhs), rop.INT_LSHIFT) if shiftop is None: continue if not shiftop.getarg(0).is_constant() or shiftop.getarg(0).getint() != 1: continue shiftvar = get_box_replacement(shiftop.getarg(1)) shiftbound = self.getintbound(shiftvar) if shiftbound.known_nonnegative() and shiftbound.known_lt_const(LONG_BIT): op = self.replace_op_with( op, rop.INT_LSHIFT, args=[rhs, shiftvar]) self.optimizer.send_extra_operation(op) return return self.emit(op)

Adding more rules to these functions is very tedious and gets super confusing when the functions get bigger. In addition I am always worried about making mistakes when writing this kind of code, and there is no feedback at all about which of these rules are actually applied a lot in real programs.

Therefore I decided to write a small domain specific language with the goal of expressing these rules in a more declarative way. In the rest of the post I'll describe the DSL (most of that description is adapted from the documentation about it that I wrote).

The Peephole Rule DSL Simple transformation rules

The rules in the DSL specify how integer operation can be transformed into cheaper other integer operations. A rule always consists of a name, a pattern, and a target. Here's a simple rule:

add_zero: int_add(x, 0) => x

The name of the rule is add_zero. It matches operations in the trace of the form int_add(x, 0), where x will match anything and 0 will match only the constant zero. After the => arrow is the target of the rewrite, i.e. what the operation is rewritten to, in this case x.

The rule language has a list of which of the operations are commutative, so add_zero will also optimize int_add(0, x) to x.

Variables in the pattern can repeat:

sub_x_x: int_sub(x, x) => 0

This rule matches against int_sub operations where the two arguments are the same (either the same box, or the same constant).

Here's a rule with a more complicated pattern:

sub_add: int_sub(int_add(x, y), y) => x

This pattern matches int_sub operations, where the first argument was produced by an int_add operation. In addition, one of the arguments of the addition has to be the same as the second argument of the subtraction.

The constants MININT, MAXINT and LONG_BIT (which is either 32 or 64, depending on which platform the JIT is built for) can be used in rules, they behave like writing numbers but allow bit-width-independent formulations:

is_true_and_minint: int_is_true(int_and(x, MININT)) => int_lt(x, 0)

It is also possible to have a pattern where some arguments needs to be a constant, without specifying which constant. Those patterns look like this:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete # more goes here => int_sub(x, C)

Variables in the pattern that start with a C match against constants only. However, in this current form the rule is incomplete, because the variable C that is being used in the target operation is not defined anywhere. We will see how to compute it in the next section.

Computing constants and other intermediate results

Sometimes it is necessary to compute intermediate results that are used in the target operation. To do that, there can be extra assignments between the rule head and the rule target.:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete C = C1 + C1 => int_sub(x, C)

The right hand side of such an assignment is a subset of Python syntax, supporting arithmetic using +, -, *, and certain helper functions. However, the syntax allows you to be explicit about unsignedness for some operations. E.g. >>u exists for unsigned right shifts (and I plan to add >u, >=u, <u, <=u for comparisons).

Here's an example of a rule that uses >>u:

urshift_lshift_x_c_c: uint_rshift(int_lshift(x, C), C) mask = (-1 << C) >>u C => int_and(x, mask) Checks

Some rewrites are only true under certain conditions. For example, int_eq(x, 1) can be rewritten to x, if x is known to store a boolean value. This can be expressed with checks:

eq_one: int_eq(x, 1) check x.is_bool() => x

A check is followed by a boolean expression. The variables from the pattern can be used as IntBound instances in checks (and also in assignments) to find out what the abstract interpretation of the JIT knows about the value of a trace variable (IntBound is the name of the abstract domain that the JIT uses for integers, despite the fact that it also stores knownbits information nowadays).

Here's another example:

mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

It expresses that x * (1 << y) can be rewritten to x << y but checks that y is known to be between 0 and LONG_BIT.

Checks and assignments can be repeated and combined with each other:

mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift)

In addition to calling methods on IntBound instances, it's also possible to access their attributes, like in this rule:

and_x_c_in_range: int_and(x, C) check x.lower >= 0 and x.upper <= C & ~(C + 1) => x Rule Ordering and Liveness

The generated optimizer code will give preference to applying rules that produce a constant or a variable as a rewrite result. Only if none of those match do rules that produce new result operations get applied. For example, the rules sub_x_x and sub_add are tried before trying sub_add_consts, because the former two rules optimize to a constant and a variable respectively, while the latter produces a new operation as the result.

The rule sub_add_consts has a possible problem, which is that if the intermediate result of the int_add operation in the rule head is used by some other operations, then the sub_add_consts rule does not actually reduce the number of operations (and might actually make things slightly worse due to increased register pressure). However, currently it would be extremely hard to take that kind of information into account in the optimization pass of the JIT, so we optimistically apply the rules anyway.

Checking rule coverage

Every rewrite rule should have at least one unit test where it triggers. To ensure this, the unit test file that mainly checks integer optimizations in the JIT has an assert at the end of a test run, that every rule fired at least once.

Printing rule statistics

The JIT can print statistics about which rule fired how often in the jit-intbounds-stats logging category, using the PYPYLOG mechanism. For example, to print the category to stdout at the end of program execution, run PyPy like this:

PYPYLOG=jit-intbounds-stats:- pypy ...

The output of that will look something like this:

int_add add_reassoc_consts 2514 add_zero 107008 int_sub sub_zero 31519 sub_from_zero 523 sub_x_x 3153 sub_add_consts 159 sub_add 55 sub_sub_x_c_c 1752 sub_sub_c_x_c 0 sub_xor_x_y_y 0 sub_or_x_y_y 0 int_mul mul_zero 0 mul_one 110 mul_minus_one 0 mul_pow2_const 1456 mul_lshift 0 ... Termination and Confluence

Right now there are unfortunately no checks that the rules actually rewrite operations towards "simpler" forms. There is no cost model, and also nothing that prevents you from writing a rule like this:

neg_complication: int_neg(x) # leads to infinite rewrites => int_mul(-1, x)

Doing this would lead to endless rewrites if there is also another rule that turns multiplication with -1 into negation.

There is also no checking for confluence (yet?), i.e. the property that all rewrites starting from the same input trace always lead to the same output trace, no matter in which order the rules are applied.

Proofs

It is very easy to write a peephole rule that is not correct in all corner cases. Therefore all the rules are proven correct with Z3 before compiled into actual JIT code, by default. When the proof fails, a (hopefully minimal) counterexample is printed. The counterexample consists of values for all the inputs that fulfil the checks, values for the intermediate expressions, and then two different values for the source and the target operations.

E.g. if we try to add the incorrect rule:

mul_is_add: int_mul(a, b) => int_add(a, b)

We get the following counterexample as output:

Could not prove correctness of rule 'mul_is_add' in line 1 counterexample given by Z3: counterexample values: a: 0 b: 1 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 0 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 1

If we add conditions, they are taken into account and the counterexample will fulfil the conditions:

mul_is_add: int_mul(a, b) check a.known_gt_const(1) and b.known_gt_const(2) => int_add(a, b)

This leads to the following counterexample:

Could not prove correctness of rule 'mul_is_add' in line 46 counterexample given by Z3: counterexample values: a: 2 b: 3 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 6 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 5

Some IntBound methods cannot be used in Z3 proofs because their control flow is too complex. If that is the case, they can have Z3-equivalent formulations defined (in every case this is done, it's a potential proof hole if the Z3 friendly reformulation and the real implementation differ from each other, therefore extra care is required to make very sure they are equivalent).

It's possible to skip the proof of individual rules entirely by adding SORRY_Z3 to its body (but we should try not to do that too often):

eq_different_knownbits: int_eq(x, y) SORRY_Z3 check x.known_ne(y) => 0 Checking for satisfiability

In addition to checking whether the rule yields a correct optimization, we also check whether the rule can ever apply. This ensures that there are some runtime values that would fulfil all the checks in a rule. Here's an example of a rule violating this:

never_applies: int_is_true(x) check x.known_lt_const(0) and x.known_gt_const(0) # impossible condition, always False => x

Right now the error messages if this goes wrong are not completely easy to understand. I hope to be able to improve this later:

Rule 'never_applies' cannot ever apply in line 1 Z3 did not manage to find values for variables x such that the following condition becomes True: And(x <= x_upper, x_lower <= x, If(x_upper < 0, x_lower > 0, x_upper < 0)) Implementation Notes

The implementation of the DSL is done in a relatively ad-hoc manner. It is parsed using rply, there's a small type checker that tries to find common problems in how the rules are written. Z3 is used via the Python API, like in the previous blog posts that are using it. The pattern matching RPython code is generated using an approach inspired by Luc Maranget's paper Compiling Pattern Matching to Good Decision Trees. See this blog post for an approachable introduction.

Conclusion

Now that I've described the DSL, here are the rules that are equivalent to the imperative code in the motivation section:

mul_zero: int_mul(x, 0) => 0 mul_one: int_mul(x, 1) => x mul_minus_one: int_mul(x, -1) => int_neg(x) mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift) mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

The current status of the DSL is that it got merged to PyPy's main branch. I rewrote a part of the integer rewrites into the DSL, but some are still in the old imperative style (mostly for complicated reasons, the easily ported ones are all done). Since I've only been porting optimizations that had existed prior to the existence of the DSL, performance numbers of benchmarks didn't change.

There are a number of features that are still missing and some possible extensions that I plan to work on in the future:

• All the integer operations that the DSL handles so far are the variants that do not check for overflow (or where overflow was proven to be impossible to happen). In regular Python code the overflow-checking variants int_add_ovf etc are much more common, but the DSL doesn't support them yet. I plan to fix this, but don't completely understand how the correctness proofs for them should be done correctly.

• A related problem is that I don't understand what it means for a rewrite to be correct if some of the operations are only defined for a subset of the input values. E.g. division isn't defined if the divisor is zero. In theory, a division operation in the trace should always be preceded by a check that the divisor isn't zero. But sometimes other optimization move the check around and the connection to the division gets lost or muddled. What optimizations can we still safely perform on the division? There's lots of prior work on this question, but I still don't understand what the correct approach in our context would be.

• Ordering comparisons like int_lt, int_le and their unsigned variants are not ported to the DSL yet. Comparisons are an area where the JIT is not super good yet at optimizing away operations. This is a pretty big topic and I've started a project with Nico Rittinghaus to try to improve the situation a bit more generally.

• A more advanced direction of work would be to implement a simplified form of e-graphs (or ae-graphs). The JIT has like half of an e-graph data structure already, and we probably can't afford a full one in terms of compile time costs, but maybe we can have two thirds or something?

Acknowledgements

Thank you to Max Bernstein and Martin Berger for super helpful feedback on drafts of the post!