FLOSS Project Planets

Welcome to this tutorial where we will explore Python map() and List Comprehension best practices and share some cool coding tips. These techniques, when mastered, can make your code cleaner, more concise, and efficient. In this guide, we’ll explore these concepts from the ground up, using simple and practical examples. Let’s begin mastering the map […]

The post Python’s Map() and List Comprehension: Tips and Best Practices appeared first on TechBeamers.

Hey, welcome! Today, we’re talking about practical Python data analysis tips and tricks. With Pandas and NumPy, we’ll tidy up data, spot trends, and do some smart analysis. It’s all about making your data work easier and getting cool insights. Let’s dive in to optimize your data analysis tasks. Practical Python Tips for Solid Data […]

The post 20 Practical Python Data Analysis Tips and Tricks appeared first on TechBeamers.

Upgrading your site from Drupal 9 to 10 Perry Aryee Developer 08.02.2024

With Drupal 9 having reached its end of life (EOL) on November 1, it’s time to start planning for an upgrade.

For those already operating on Drupal 9, upgrading to Drupal 10 is not as daunting as earlier upgrades and promises to be easy, reflecting the software’s overall trend towards smaller, more incremental upgrades and faster iterations. According to Drupal, you should "completely update your Drupal 9 site to the most recent version of the modules and theme(s), before updating to Drupal 10." 

The deprecated code will be based on Drupal 9 moving into Drupal 10, but there are ways to search your system and update the specified code block.

The first thing to do is to install the  drupal/upgrade_status module in your project and enable it. composer require drupal/upgrade_status && drush en upgrade_status, if you have drush installed else go to admin > modules  and search for upgrade status and install it. After the module is enabled, go to Admin > Reports > Upgrade Status. This page should contain all the upgrades and code changes necessary before your site can be upgraded to Drupal 10.

Steps to upgrade Drupal 9 to Drupal 10

Migrating from Drupal 9 to Drupal 10 can be easy or can be difficult depending on the project you are involved in. Yes, because every site is different and may present its own unique challenges.

These are the following steps to migrate from Drupal 9 to Drupal 10.

1. Keeping your custom modules up to date with new standards and removing deprecated code will result in small changes before you have to do a major core upgrade. If the site is well maintained the changes are mostly with the core_version_requirement. You will need to update from core_version_requirement: ^9 to core_version_requirement: ^9 || ^10 . As soon as we are sure that our custom code is compatible with Drupal 10, we can move to check the contrib modules. And this is where things might get tricky

2. Upgrade contrib modules to versions which support Drupal 9 and 10. If the new version only supports D10, then you can add it to composer as an alternative version, for example: 'drupal/module_name': '^1 || ^2', and you can have version 1 still installed while on Drupal 9. Once you install Drupal 10, version 2 of the module will be installed by composer.

3. Uninstall obsolete modules and themes like Color, RDF, Seven etc. You can’t remove core modules or themes, but if you are not sure if these are depended upon, it can be moved from core to contrib especially classy and stable, which you can keep as contrib modules. You may need to set your Admin theme to Claro and re-export your config on this step. Use drush theme:uninstall theme(s) command (drush thun for short) to uninstall themes.

4. Remove orphaned permissions which are still assigned to user roles. Export your config before starting this step. The upgrade_status module will tell you which permissions need to go from which roles. Simply edit the user.role.[role].yml config ymls and remove the relevant lines from the permissions array. 

5. Upgrade Drupal core to latest 9.5, in order to upgrade to 10.

composer require drupal/core-recommended:^9.5 drupal/core-composer-scaffold:^9.5  --update-with-all-dependencies

Now let's look at the tricky part. Drupal 10 has some specific things we have to do to allow us to upgrade to drupal 10.

1. Drupal 10 runs with PHP 8.2 or later so make sure you have it to be able to upgrade. 

2. Not all modules are Drupal 10 ready so you will have to use drupal lenient.  You will have to get a patch for Drupal 9 only modules. Install mglaman/composer-drupal-lenient before attempting to upgrade. Otherwise these modules will create a dependency problem. After the installation, you need to declare the Drupal 9 modules which should be treated as Drupal 10 modules. For example, to allow drupal/token to be installed run:

composer require mglaman/composer-drupal-lenient

composer config --merge --json extra.drupal-lenient.allowed-list '["drupal/token"]'

3. If you have Drush installed make sure the version is ^11 || ^12, we use this because some module may not be version 12 compatible.

4. Drupal Console (drupal/console) is not Drupal 10 compatible and as such we have to remove it. 

composer remove drupal/console --no-update

5. Upgrade CKEditor 4 to CKEditor 5,if you have custom CKEditor 4 plugins which don’t have an upgrade path to CKEditor 5, then you can keep using CKEditor 4 but as a contrib module. You may have to use it although it is deprecated, to avoid errors when you deploy the Drupal 10 upgrade, when the config import tries to uninstall CKEditor.

6. If you use Entity Embed, you need to manually replace CKEditor Embed buttons with SVG icons as Drupal 10 is not using png.

7. Check custom code for core/jquery.once if it does exist update it to use core/once because jquery.once is removed from Drupal 10

8. If you encounter dependency issues, and composer just won’t let you upgrade to D10, but you are sure that all the dependencies are in order, then the quickest thing to do is to delete the composer.lock file(although its not a best practice), delete vendor folder then run composer install which will install from the composer.json file and create a new .lock file.

9. Note that you can ignore some false positives such as an empty custom profile, which phpstan can’t scan, and as such upgrade status would complain, the trick would be to create an empty .php file to get the issue fix

After all these are done and ready for upgrade, you can run this command

composer require 'drupal/core-recommended:^10' 'drupal/core-composer-scaffold:^10' 'drupal/core-project-message:^10' --no-update

If   drupal/core happens to be part of the composer.json file, remove it. This dependency is included in drupal/core-recommended and may cause problems. If you have drupal/core-dev installed, you can run this

composer require 'drupal/core-dev:^10' --dev --no-update

Now,let us test perform the update with the --dry-run option: this allows us to see if the update will runs smoothly or might encounter some errors.

composer update --dry-run

If you encounter any errors, walk through the process to resolve them. Resume the process when the errors are resolved, run the update with dependencies to update any transitive module.

composer update -W

Don’t forget to run drush updb and drush cex after the upgrade. This means you should run the upgrade on top of an installed and functioning D9 database.

If you have a custom theme that is  based on classy,seven or stable then don’t forget to install them as  contrib themes - composer require 'drupal/[module_name]'. You should uninstall and remove upgrade status after the upgrade process.

Pathauto has an issue with some config files so, if you have Pathauto installed then you need to update some configs by hand. Namely pathauto.pattern.[name] config files have had their selection_criteria plugins updated. For example node_type becomes entity_bundle:node. or  try the following command

drush eval 'include_once(DRUPAL_ROOT . "/modules/contrib/pathauto/pathauto.install"); pathauto_update_8108()'

In summary the whole update process for a site can be very difficult and ranges from project to project. It could have different modules installed, with some even locked to specific dev versions, as well as heavily patched old major versions of modules which needed to be upgraded. You walk through these steps and iterate through if need be to get your upgrade completed.

Discover how the Automated Testing Kit for Drupal, led by André Angelantoni, revolutionizes end-to-end testing with its versatile features, impactful vision, and seamless integration of Cypress.io and Playwright frameworks.

Our Security Developer-in-Residence, Seth Larson, has been working to improve the management of vulnerabilities for Python users. Seth has championed progress on this goal in a variety of areas:

• Authorizing the Python Software Foundation as a CVE Numbering Authority (CNA) to publish CVE IDs and records
• Revitalizing the security advisory mailing list (security-announce@python.org)
• Migrating all historical vulnerabilities to the Open Source Vulnerability format (OSV) and having the records indexed into the global OSV database

With the release of CPython 3.12.2, the next step of the Python Software Foundation’s vulnerability management strategy is now available in the form of Software Bill-of-Materials (SBOM) documents for CPython source releases. The documents are available for download in their own column labeled “SBOM” in the “Files” table on the release page. User documentation and a getting started guide for CPython SBOMs is available on python.org.

These documents are relatively new but have been tested with multiple tools that accept SPDX SBOM documents. Please report any feedback on the SBOM to the CPython issue tracker.

What is a Software Bill-of-Materials (SBOM)?

Software Bill-of-Materials are machine-readable documents using an ecosystem-independent format like SPDX or CycloneDX to describe what a piece of software is made of and how each component within the software relates to other components. There are multiple use-cases for SBOMs, but for CPython we primarily focused on software supply chain and vulnerability management.

Many vulnerability scanning tools support passing an SBOM document as input to provide a comprehensive scan for software vulnerabilities without needing to rely on fallible software discovery. This means there’s less chances for vulnerabilities to be missed by scanners.

There are existing tools for automatically creating SBOMs for software, but SBOMs which aren’t accurate are sometimes more dangerous than having no SBOM due to causing a false sense of security. This is especially true for complex pieces of software or projects which exist outside of package ecosystems, both of which apply to CPython and make generating an SBOM difficult. For this reason the content of CPython SBOMs is curated by hand on first pass to ensure accuracy and completeness and then automated to track updates as the software changes.

SBOM documents are becoming a requirement for compliance in multiple areas and industries. In order to meet those requirements we are providing a comprehensive and accurate SBOM for CPython that will provide assurance for Python users.

What is included in CPython SBOMs?

CPython SBOMs use the SPDX SBOM standard. SBOM documents include a description of the contained software, including all of its dependencies. Information in CPython SBOMs includes:

• Names and versions of all software components
• Software identifiers (like CPE and Package URLs)
• Download URLs for source code with checksums
• File names and content checksums
• Dependency relationships between each component

CPython SBOMs satisfy the requirements listed in the NTIA Minimum Elements for a Software Bill of Materials. Software identifiers can be used for correlating software in use to vulnerability databases like the CVE database and Open Source Vulnerability database, typically done automatically using vulnerability scanning tools.

What isn’t included in CPython SBOMs?

Keep in mind that software libraries that you supply yourself to compile CPython, such as OpenSSL and zlib, are not included in the SBOMs for source artifacts.

This is due to these libraries not being included in source artifacts, so CPython users have a choice of which version and sources to use for these third-party libraries. Folks who are compiling CPython from source are responsible for tracking their own dependencies either in a separate SBOM document or by appending new entries to your local CPython SBOM.

CPython’s SBOMs don’t include licensing information for dependencies. See the CPython licensing page for licensing information.

What is coming next for CPython SBOMs?

This is only the beginning for CPython SBOMs, as mentioned above there are only SBOM documents published for source releases today. The CPython release managers also publish binary installers for Windows and macOS on a variety of distribution channels. These artifacts will need their own SBOM documents as they are compiled with software that’s typically not available on those platforms (e.g. OpenSSL).

There’s also more infrastructure needed to reduce noise and churn for Python users and Python Security Response Team members alike. Vulnerability EXchange (VEX) statements are a set of standards which allows software producers to signal to user tooling whether a piece of software in use is affected by a vulnerability, even for vulnerabilities affecting dependencies. This is an area of active development and is being explored alongside the OpenSSF Security Tooling Working Group.

The Security Developer-in-Residence role and this work is funded by a substantial investment from the OpenSSF Alpha-Omega Project. Thanks to Alpha-Omega for their support in improving the security posture of the entire Python ecosystem.The OpenSSF is a non-profit cross-industry collaboration that unifies security initiatives and brings together leaders to improve the security of open source software by building a broader community, targeted initiatives, and best practices.

Write it down before I forget about it again:

for x in $(gh api graphql --paginate -f query='query($endCursor:String) { organization(login:"myorg") { repositories(first: 100, after: $endCursor, isArchived:false) { pageInfo { hasNextPage endCursor } nodes { name } } } }' --jq '.data.organization.repositories.nodes[].name'); do secrets=$(gh secret list --json name --jq '.[].name' -R "myorg/${x}" | tr '\n' ',') if ! [ -z "${secrets}" ]; then echo "${x},${secrets}" fi done

Requests a list of all not archived repositories in a GitHub org and queries repository secrets. If we find some we output the repo name and the secrets in a comma separated list. Not real CSV, but good enough for further processing. I've to admit it's kinda beautiful what you can do with the gh cli by now. Sadly it seems the secrets are not yet available via GraphQL (or I missed it in the docs), so I just use the gh cli to do the REST calls.

So you've created a Python-based open source project and it's started to take off. You're getting contributors, lots of buzz in the podcast space, and more. But you have that day job working on Java. How do you make the transition from popular hobby project to full time job? After all, you are giving away your open source project for free, right? Well, on this episode, I have put together an amazing panel of guests who all have done exactly this: Turned their project into full time work and even companies in some cases. We have Samuel Colvin, Gina Häußge, Sebastián Ramírez, Charlie Marsh, Will McGugan and Eric Holscher on to share their stories.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/basedash'>Basedash</a><br> <a href='https://talkpython.fm/sentry-monorepo'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Will McGugan</b>: <a href="https://twitter.com/willmcgugan" target="_blank" rel="noopener">@willmcgugan</a><br/> <b>Charlie Marsh</b>: <a href="https://hachyderm.io/@charliermarsh" target="_blank" rel="noopener">@charliermarsh@hachyderm</a><br/> <b>Sebastián Ramírez</b>: <a href="https://twitter.com/tiangolo" target="_blank" rel="noopener">@tiangolo</a><br/> <b>Samuel Colvin</b>: <a href="https://twitter.com/samuel_colvin" target="_blank" rel="noopener">@samuel_colvin</a><br/> <b>Gina on Mastodon</b>: <a href="https://chaos.social/@foosel" target="_blank" rel="noopener">chaos.social/@foosel</a><br/> <b>Eric Holscher</b>: <a href="https://twitter.com/ericholscher" target="_blank" rel="noopener">@ericholscher</a><br/> <br/> <b>Pydantic</b>: <a href="https://pydantic.dev" target="_blank" rel="noopener">pydantic.dev</a><br/> <b>Astral (makes of Ruff)</b>: <a href="https://astral.sh" target="_blank" rel="noopener">astral.sh</a><br/> <b>Octoprint</b>: <a href="https://octoprint.org" target="_blank" rel="noopener">octoprint.org</a><br/> <b>Read the Docs</b>: <a href="https://about.readthedocs.com/" target="_blank" rel="noopener">readthedocs.com</a><br/> <b>FastAPI</b>: <a href="https://fastapi.tiangolo.com" target="_blank" rel="noopener">fastapi.tiangolo.com</a><br/> <b>Textual (makes of Rich)</b>: <a href="https://www.textualize.io" target="_blank" rel="noopener">textualize.io</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=HV1LKitAr44" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/448/full-time-open-source-devs-panel" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

Earlier this month the DebConf team announced the DebConf24 Logo Contest asking aspiring artists, designers, and contributors to submit an image that would represent the host city of Busan, the host nation of South Korea, and promote the next Debian Developer Conference.

The logo contest for DebConf24 received 10 submissions and garnered 354 responses with 3 proposals in particular getting very close to first place. The winning logo received 88 votes, the 2nd favored logo received 87 votes, and the 3rd most favored received 86 votes.

Thank you to Woohee Yang and Junsang Moon for sharing their artistic visions.

A very special Thank You to everyone who took the time to vote for our beautiful new logo!

The DebConf24 Team is proud to share for preview only the winning logo for the 24th Debian Developer Conference:

'sun-seagull-sea' by Woohee Yang

This is a preview copy, other revisions will occur for sizing, print, and media... but we had to share it with you all now. :).

Looking forward to seeing you all at #debconf24 in #Busan, South Korea 2024!

In this episode, we did work to get the Stripe checkout session going. We set up Stripe Product and Price objects to get the subscription plan ready and got the Stripe checkout session working mostly end-to-end

CPython 3.12.2 is SBOM-ified! About • Blog • Newsletter • Links CPython 3.12.2 is SBOM-ified!

Published 2024-02-08 by Seth Larson
Reading time: minutes

This critical role would not be possible without funding from the OpenSSF Alpha-Omega project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem! SBOM for CPython source artifacts

CPython 3.12.2 is the first release that has SBOMs for source artifacts 🥳 There's an announcement for the PSF blog, so go read that first!

Work items for SBOMs that got worked on this week:

• SBOM tooling was backported to the 3.12 branch
• Merged SBOM building to the CPython release process
• Both of the above items together mean 3.12.2 could be the first release with SBOMs for source artifacts.
• Published user documentation and a getting started guide for CPython SBOMs to python.org/download/sbom
• Received more feedback from CPython downstream distributors and pip maintainers and have opened an issue for further automating pip's SBOM data discovery.
• Chilled with release managers on Discord during 3.12.2 release in case anything went wrong.
• Wrote the blog post announcement.

After consulting with legal help regarding the licensing questions that came up during SBOM development I opted to change all licenseConcluded fields for dependencies in the SBOM to NOASSERTION as the primary use-case for CPython's SBOMs was for supply chain and vulnerability management.

Next steps for this project include investigating SBOMs for Windows installers and continuing to learn about Vulnerability Exchange (VEX) and how it can be applied to CPython SBOMs.

Other items

• My work on the libwebp vulnerability was referenced at a FOSDEM talk by Henrik Plate of Endor Labs.
• Got access to CDN logs for python.org to track downloads of SBOMs and Sigstore artifacts.
• Got up to speed with the effort to switch mail protocols (SMTP, IMAP, and POP3) and FTP in the standard library to verify certificates by default.
• Had both of my PyCon US 2024 talk proposals declined, but will be attending regardless (look forward to a security-focused open space!)
• Attended the Alpha-Omega monthly community meeting.
• Triaged reports to the Python Security Response Team.

That's all for this week! 👋 If you're interested in more you can read last week's report.

Thanks for reading! ♡ Did you find this article helpful and want more content like it? Get notified of new posts by subscribing to the RSS feed or the email newsletter.

This work is licensed under CC BY-SA 4.0

Core Reproducible Builds developer Holger Levsen presented at the main track at FOSDEM on Saturday 3rd February this year in Brussels, Belgium. Titled Reproducible Builds: The First Ten Years…

In this talk Holger ‘h01ger’ Levsen will give an overview about Reproducible Builds: How it started with a small BoF at DebConf13 (and before), then grew from being a Debian effort to something many projects work on together, until in 2021 it was mentioned in an Executive Order of the President of the United States. And of course, the talk will not end there, but rather outline where we are today and where we still need to be going, until Debian stable (and other distros!) will be 100% reproducible, verified by many.

h01ger has been involved in reproducible builds since 2014 and so far has set up automated reproducibility testing for Debian, Fedora, Arch Linux, FreeBSD, NetBSD and coreboot.

More information can be found on FOSDEM’s own page for the talk.

Separate from Holger’s talk, however, there were a number of other talks about reproducible builds at FOSDEM this year:

• Reproducible builds for confidential computing: Why remote attestation is worthless without it by Malte Poll and Paul Meyer.
• RISC-V Bootstrapping in Guix and Live-Bootstrap by Ekaitz.
• rix: an R package for reproducible dev environments with Nix by Bruno Rodrigues.
• Making reproducible and publishable large-scale HPC experiments by Philippe Swartvagher.
• Documenting and Fixing Non-Reproducible Builds due to Configuration Options by RANDRIANAINA Georges Aaron.

… and there was even an entire track on Software Bill of Materials.

Welcome to the January 2024 report from the Reproducible Builds project. In these reports we outline the most important things that we have been up to over the past month. If you are interested in contributing to the project, please visit our Contribute page on our website.

“How we executed a critical supply chain attack on PyTorch”

John Stawinski and Adnan Khan published a lengthy blog post detailing how they executed a supply-chain attack against PyTorch, a popular machine learning platform “used by titans like Google, Meta, Boeing, and Lockheed Martin”:

Our exploit path resulted in the ability to upload malicious PyTorch releases to GitHub, upload releases to [Amazon Web Services], potentially add code to the main repository branch, backdoor PyTorch dependencies – the list goes on. In short, it was bad. Quite bad.

The attack pivoted on PyTorch’s use of “self-hosted runners” as well as submitting a pull request to address a trivial typo in the project’s README file to gain access to repository secrets and API keys that could subsequently be used for malicious purposes.

New Arch Linux forensic filesystem tool

On our mailing list this month, long-time Reproducible Builds developer kpcyrd announced a new tool designed to forensically analyse Arch Linux filesystem images.

Called archlinux-userland-fs-cmp, the tool is “supposed to be used from a rescue image (any Linux) with an Arch install mounted to, [for example], /mnt.” Crucially, however, “at no point is any file from the mounted filesystem eval’d or otherwise executed. Parsers are written in a memory safe language.”

More information about the tool can be found on their announcement message, as well as on the tool’s homepage. A GIF of the tool in action is also available.

Issues with our SOURCE_DATE_EPOCH code?

Chris Lamb started a thread on our mailing list summarising some potential problems with the source code snippet the Reproducible Builds project has been using to parse the SOURCE_DATE_EPOCH environment variable:

I’m not 100% sure who originally wrote this code, but it was probably sometime in the ~2015 era, and it must be in a huge number of codebases by now.

Anyway, Alejandro Colomar was working on the shadow security tool and pinged me regarding some potential issues with the code. You can see this conversation here.

Chris ends his message with a request that those with intimate or low-level knowledge of time_t, C types, overflows and the various parsing libraries in the C standard library (etc.) contribute with further info.

Distribution updates

In Debian this month, Roland Clobus posted another detailed update of the status of reproducible ISO images on our mailing list. In particular, Roland helpfully summarised that “all major desktops build reproducibly with bullseye, bookworm, trixie and sid provided they are built for a second time within the same DAK run (i.e. [within] 6 hours)”. Additionally 7 of the 8 bookworm images from the official download link build reproducibly at any later time.

In addition to this, three reviews of Debian packages were added, 17 were updated and 15 were removed this month adding to our knowledge about identified issues.

Elsewhere, Bernhard posted another monthly update for his work elsewhere in openSUSE.

Community updates

There were made a number of improvements to our website, including Bernhard M. Wiedemann fixing a number of typos of the term ‘nondeterministic’. […] and Jan Zerebecki adding a substantial and highly welcome section to our page about SOURCE_DATE_EPOCH to document its interaction with distribution rebuilds. […].

diffoscope is our in-depth and content-aware diff utility that can locate and diagnose reproducibility issues. This month, Chris Lamb made a number of changes such as uploading versions 245 and 255 to Debian but focusing on triaging and/or merging code from other contributors. This included adding support for comparing eXtensible ARchive’ (.XAR/.PKG) files courtesy of Seth Michael Larson […][…], as well considerable work from Vekhir in order to fix compatibility between various and subtle incompatible versions of the progressbar libraries in Python […][…][…][…]. Thanks!

Reproducibility testing framework

The Reproducible Builds project operates a comprehensive testing framework (available at tests.reproducible-builds.org) in order to check packages and other artifacts for reproducibility. In January, a number of changes were made by Holger Levsen:

• Debian-related changes:

  • Reduce the number of arm64 architecture workers from 24 to 16. […]
  • Use diffoscope from the Debian release being tested again. […]
  • Improve the handling when killing unwanted processes […][…][…] and be more verbose about it, too […].
  • Don’t mark a job as ‘failed’ if process marked as ‘to-be-killed’ is already gone. […]
  • Display the architecture of builds that have been running for more than 48 hours. […]
  • Reboot arm64 nodes when they hit an OOM (out of memory) state. […]

• Package rescheduling changes:

  • Reduce IRC notifications to ‘1’ when rescheduling due to package status changes. […]
  • Correctly set SUDO_USER when rescheduling packages. […]
  • Automatically reschedule packages regressing to FTBFS (build failure) or FTBR (build success, but unreproducible). […]

• OpenWrt-related changes:

  • Install the python3-dev and python3-pyelftools packages as they are now needed for the sunxi target. […][…]
  • Also install the libpam0g-dev which is needed by some OpenWrt hardware targets. […]

• Misc:

  • As it’s January, set the real_year variable to 2024 […] and bump various copyright years as well […].
  • Fix a large (!) number of spelling mistakes in various scripts. […][…][…]
  • Prevent Squid and Systemd processes from being killed by the kernel’s OOM killer. […]
  • Install the iptables tool everywhere, else our custom rc.local script fails. […]
  • Cleanup the /srv/workspace/pbuilder directory on boot. […]
  • Automatically restart Squid if it fails. […]
  • Limit the execution of chroot-installation jobs to a maximum of 4 concurrent runs. […][…]

Significant amounts of node maintenance was performed by Holger Levsen (eg. […][…][…][…][…][…][…] etc.) and Vagrant Cascadian (eg. […][…][…][…][…][…][…][…]). Indeed, Vagrant Cascadian handled an extended power outage for the network running the Debian armhf architecture test infrastructure. This provided the incentive to replace the UPS batteries and consolidate infrastructure to reduce future UPS load. […]

Elsewhere in our infrastructure, however, Holger Levsen also adjusted the email configuration for @reproducible-builds.org to deal with a new SMTP email attack. […]

Upstream patches

The Reproducible Builds project tries to detects, dissects and fix as many (currently) unreproducible packages as possible. We endeavour to send all of our patches upstream where appropriate. This month, we wrote a large number of such patches, including:

• Bernhard M. Wiedemann:

  • cython (nondeterminstic path issue)
  • deluge (issue with modification time of .egg file)
  • gap-ferret, gap-semigroups & gap-simpcomp (nondeterministic config.log file)
  • grpc (filesystem ordering issue )
  • hub (random)
  • kubernetes1.22 & kubernetes1.23 (sort-related issue)
  • kubernetes1.24 & kubernetes1.25 (go -trimpath vs random issue)
  • libjcat (drop test files with random bytes)
  • luajit (Use new d option for deterministic bytecode output)
  • meson […][…] (sort the results from Python filesystem call)
  • python-rjsmin (drop GCC instrumentation artifacts)
  • qt6-virtualkeyboard+others (bug parallelism/race)
  • SoapySDR (parallelism-related issue)
  • systemd (sorting problem)
  • warewulf (CPIO modification time issue, etc.)

• Chris Lamb:

  • #1060254 filed against mumble.

• James Addison:

  • guake (‘Schroedinger’ file due to race condition)
  • qhelpgenerator-qt5 (timezone localization; fix also merged upstream for QT6)
  • sphinx (search index doctitle sorting)

Separate to this, Vagrant Cascadian followed up with the relevant maintainers when reproducibility fixes were not included in newly-uploaded versions of the mm-common package in Debian — this was quickly fixed, however. […]

If you are interested in contributing to the Reproducible Builds project, please visit our Contribute page on our website. However, you can get in touch with us via:

• IRC: #reproducible-builds on irc.oftc.net.

• Mailing list: rb-general@lists.reproducible-builds.org

• Mastodon: @reproducible_builds

• Twitter: @ReproBuilds

Take advantage of Symfony Mailer’s first-class integration with Symfony Messenger brought to Drupal via the SM project, allowing your site to send emails asynchronously.

by daniel.phin / 8 February 2024

This post is part 6 in a series about Symfony Messenger.

1. Introducing Symfony Messenger integrations with Drupal
2. Symfony Messenger’ message and message handlers, and comparison with @QueueWorker
3. Real-time: Symfony Messenger’ Consume command and prioritised messages
4. Automatic message scheduling and replacing hook_cron
5. Adding real-time processing to QueueWorker plugins
6. Making Symfony Mailer asynchronous: integration with Symfony Messenger
7. Displaying notifications when Symfony Messenger messages are processed
8. Future of Symfony Messenger in Drupal

Since Swift Mailer and its Drupal contrib integration were recently deprecated, many projects have naturally switched to its replacement: Symfony Mailer, either via Drupal Symfony Mailer or Drupal Symfony Mailer Lite.

This post outlines how you can take advantage of Symfony Mailer’s first class integration with Symfony Messenger brought to Drupal via the SM project. This integration allows for dispatching emails off-thread, potentially improving performance of the dispatching (usually web-) thread by offloading email-related tasks to dedicated Symfony Messenger workers. This setup can be considered an alternative to using Queue Mail.

Setup

As of writing, of the two Symfony Mailer implementations in contrib, Drupal Symfony Mailer Lite has built in support for Symfony Messenger. Drupal Symfony Mailer does not yet support it, an issue and merge request exist to add it. Apply a patch until the changes are merged.

Symfony Messenger itself does not require any special configuration, other than installing SM.

To run asynchronously, the \Symfony\Component\Mailer\Messenger\SendEmailMessage message must have routing configuration to a transport. Or at least the fallback transport must be configured. Without transport configuration, Emails will still be dispatched through Messenger, however they will be executed synchronously in the same thread they were dispatched.

Opting out

If you happen to have both Symfony Mailer and Symfony Messenger installed but do not want emails to be sent asynchronously, you can configure routing for the \Symfony\Component\Mailer\Messenger\SendEmailMessage message to instead use the synchronous transport.

If you’re using the SM Config submodule:

Sending emails and dispatching emails

Emails may be dispatched using the usual Drupal mechanism, or you can dispatch using Symfony Mailer directly by constructing an email object:

$email = (new \Symfony\Component\Mime\Email()) ->to('jane@example.com') ->from('john@example.com') ->subject('Hello world!') ->text('Some sample text.') ->html('<p>some <strong>sample</strong> text.</p>'); /** @var \Symfony\Component\Mailer\MailerInterface $mailer */ $mailer = \Drupal::service(\Symfony\Component\Mailer\MailerInterface::class); $mailer->send($email);

After the send method is executed, Mailer checks Messenger is available, creates a new SendEmailMessage message to wrap the \Symfony\Component\Mime\Email object. Then dispatches SendEmailMessage to the messenger bus.

As is typical with Symfony Messenger, email messages must be serialisable. Avoid including any Drupal entities or service references in an email object, and render email contents before sending it.

Processing emails

To process email messages, run the worker with sm messenger:consume. This command will either listen or poll for messages and execute them in a dedicated thread, ensuring quick processing after they are dispatched. For more information on the worker, please refer to post 3 of this series.

In the next post, we’ll explore how to add a user interface to notify users when relevant tasks have been processed.

Tagged Symfony, Symfony Messenger, Symfony Mailer, Email

I got a new work laptop this year: A Thinkpad X1 Carbon (Gen 11). It wasn't the one I wanted: I'd ordered an X1 Nano, which had a footprint very reminiscent to me of my beloved x40.

Never mind! The Carbon is lovely. Despite ostensibly the same size as the T470s it's replacing, it's significantly more portable, and more capable. The two USB-A ports, as well as the full-size HDMI port, are welcome and useful (over the Nano).

I used to keep notes on setting up Linux on different types of hardware, but I haven't really bothered now for years. Things Just Work. That's good!

My old machine naming schemes are stretched beyond breaking point (and I've re-used my favourite hostname, qusp, one too many times) so I went for something new this time: Riffing on Carbon, I settled (for now) on carbyne, a carbon allotrope which is of interest to nanotechnologists (Seems appropriate)

Episode 27 of The Python Show Podcast welcomes Lukasz Langa as our guest.

Lukasz is a CPython Developer in Residence, which means he works full-time on the core CPython language. He is also the creator of Black, a super popular Python code formatter.

In this episode, we talked about the following topics:

• Core python development
• The origin of Black
• YAPF and Ruff
• Python, Python, Python
• and more!

Links

You can follow or connect with ?ukasz at the following sites:

• X / Twitter
• Personal website

The post Episode 27 – Python Formatters with Lukasz Langa appeared first on Mouse Vs Python.

Instead of showing the original author of a node, show the last person to edit it.

PEP 8, sometimes spelled PEP8 or PEP-8, is a document that provides guidelines and best practices on how to write Python code. It was written in 2001 by Guido van Rossum, Barry Warsaw, and Alyssa Coghlan. The primary focus of PEP 8 is to improve the readability and consistency of Python code.

By the end of this tutorial, you’ll be able to:

• Write Python code that conforms to PEP 8
• Understand the reasoning behind the guidelines laid out in PEP 8
• Set up your development environment so that you can start writing PEP 8 compliant Python code

PEP stands for Python Enhancement Proposal, and there are many PEPs. These documents primarily describe new features proposed for the Python language, but some PEPs also focus on design and style and aim to serve as a resource for the community. PEP 8 is one of these style-focused PEPs.

In this tutorial, you’ll cover the key guidelines laid out in PEP 8. You’ll explore beginner to intermediate programming topics. You can learn about more advanced topics by reading the full PEP 8 documentation.

Get Your Code Click here to download the free sample code that shows you how to write PEP 8 compliant code.

Why We Need PEP 8

“Readability counts.”

— The Zen of Python

PEP 8 exists to improve the readability of Python code. But why is readability so important? Why is writing readable code one of the guiding principles of the Python language, according to the Zen of Python?

Note: You may encounter the term Pythonic when the Python community refers to code that follows the idiomatic writing style specific to Python. Pythonic code adheres to Python’s design principles and philosophy and emphasizes readability, simplicity, and clarity.

As Guido van Rossum said, “Code is read much more often than it’s written.” You may spend a few minutes, or a whole day, writing a piece of code to process user authentication. Once you’ve written it, you’re never going to write it again.

But you’ll definitely have to read it again. That piece of code might remain part of a project you’re working on. Every time you go back to that file, you’ll have to remember what that code does and why you wrote it, so readability matters.

It can be difficult to remember what a piece of code does a few days, or weeks, after you wrote it.

If you follow PEP 8, you can be sure that you’ve named your variables well. You’ll know that you’ve added enough whitespace so it’s easier to follow logical steps in your code. You’ll also have commented your code well. All of this will mean your code is more readable and easier to come back to. If you’re a beginner, following the rules of PEP 8 can make learning Python a much more pleasant task.

Note: Following PEP 8 is particularly important if you’re looking for a development job. Writing clear, readable code shows professionalism. It’ll tell an employer that you understand how to structure your code well.

If you have more experience writing Python code, then you may need to collaborate with others. Writing readable code here is crucial. Other people, who may have never met you or seen your coding style before, will have to read and understand your code. Having guidelines that you follow and recognize will make it easier for others to read your code.

Naming Conventions

“Explicit is better than implicit.”

— The Zen of Python

When you write Python code, you have to name a lot of things: variables, functions, classes, packages, and so on. Choosing sensible names will save you time and energy later. You’ll be able to figure out, from the name, what a certain variable, function, or class represents. You’ll also avoid using potentially confusing names that might result in errors that are difficult to debug.

One suggestion is to never use l, O, or I single letter names as these can be mistaken for 1 and 0, depending on what typeface a programmer uses.

For example, consider the code below, where you assign the value 2 to the single letter O:

Python O = 2 # ❌ Not recommended Copied!

Doing this may look like you’re trying to reassign 2 to zero. While making such a reassignment isn’t possible in Python and will cause a syntax error, using an ambigious variable name such as O can make your code more confusing and harder to read and reason about.

Naming Styles

The table below outlines some of the common naming styles in Python code and when you should use them:

Read the full article at https://realpython.com/python-pep8/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

We are happy to announce the release of Qt Creator 12.0.2!

Introduction

As a Python developer I want to stay up2date with trends and useful tips & tricks.

Of course there are great newsletters like Pycoders, but those are already hitting my inbox.

Let’s look at Planet Python in this article, an aggregation site/ service that indexes a lot of good Python blog feeds. Keeping an eye on that resource will be useful.

In this article we’ll build a tool to parse this resource and get a daily email with new articles.

We will use Sendgrid for the emailing and GitHub Actions to run it automatically.

Planet Python parse and email script

Here is the script I came up with:

from datetime import datetime, timedelta, UTC from typing import NamedTuple import feedparser from dateutil.parser import parse from decouple import config from sendgrid import SendGridAPIClient from sendgrid.helpers.mail import Mail SENDGRID_API_KEY = config("SENDGRID_API_KEY") FROM_EMAIL = config("FROM_EMAIL") TO_EMAIL = config("TO_EMAIL") PLANET_PYTHON_FEED = "https://planetpython.org/rss20.xml" ONE_DAY = 1 class Article(NamedTuple): title: str link: str publish_date: str def fetch_articles() -> list[Article]: feed = feedparser.parse(PLANET_PYTHON_FEED) return [ Article(entry.title, entry.link, entry.published) for entry in feed.entries ] def filter_recent_articles( articles: list[Article], days: int = ONE_DAY ) -> list[Article]: recent_articles = [] now = datetime.now(UTC) for article in articles: publish_date = parse(article.publish_date) if now - publish_date <= timedelta(days): recent_articles.append(article) return recent_articles def send_email( from_email: str, to_email: str, subject: str, content: str ) -> None: message = Mail( from_email=from_email, to_emails=to_email, subject=subject, html_content=content, ) try: sg = SendGridAPIClient(SENDGRID_API_KEY) response = sg.send(message) print(f"Email sent with status code: {response.status_code}") except Exception as e: print(e) def main() -> None: articles = fetch_articles() recent_articles = filter_recent_articles(articles) if len(recent_articles) == 0: print("No new articles found") return subject = "New Planet Python articles" def _create_link(article: Article) -> str: return f"<a href='{article.link}'>{article.title}</a>" body = "<br>".join( [_create_link(article) for article in recent_articles] ) send_email(FROM_EMAIL, TO_EMAIL, subject, body) if __name__ == "__main__": main()

Explanation:

Environment Configuration: I use decouple (python-decouple package, related article here) to securely manage environment variables, including the SendGrid API key and email addresses for both sender and receiver.

RSS Feed Parsing: I use feedparser to fetch and parse the RSS feed from Planet Python, extracting articles’ titles, links, and published dates.

Article Data Structure: I use a typed NamedTuple, Article, to store essential information about each article, including its title, link, and publish date.

Fetching Recent Articles: I created a function fetch_articles that retrieves all articles from the RSS feed and returns a list of Article instances.

Filtering by Publish Date: The filter_recent_articles function filters articles to include only those published within the last day (or a specified number of days), using dateutil.parser to handle date parsing and datetime for date comparison. Note that I later learned that you can use datetime.UTC instead of ZoneInfo, saving the import.

Email Preparation and Sending: Leverages the SendGrid API through sendgrid library to compose and send an HTML email. The email contains links to recent articles, formatted using a helper function _create_link.

Dynamic Email Content: I create the email body dynamically by converting the list of recent articles into HTML links.

By the way, my first go was text only, but that looked ugly:

Error Handling in Email Sending: Includes try-except blocks around the email sending process to catch and print any errors, enhancing reliability and debugging ease.

While our script employs a broad Exception catch for simplicity and broad coverage, understanding and implementing more granular exception handling can significantly enhance your application’s robustness and debuggability. In Python, different exceptions can be raised for various reasons, such as ValueError for an invalid parameter, or IOError for issues accessing a file. By catching exceptions more specifically, you can provide more detailed error messages, recover gracefully from certain errors, or even retry failed operations under specific conditions. For those interested in diving deeper into this topic, the Python documentation on Built-in Exceptions offers a comprehensive overview of the different exception types available. You can also check out our 7 Tips to Improve Your Error Handling in Python article.

Main Workflow Execution: The main function orchestrates the script’s workflow: fetching articles, filtering for recency, composing, and sending the email notification if new articles are found.

Conditional Execution Check: Before sending an email, checks if there are any new articles. If none are found, it prints a message and exits, avoiding unnecessary emails (while the printed message will still end up in the GitHub Action logs).

Hooking it up with a GitHub Action

Here is the GitHub workflow I created in .github/workflows (the required target directory structure for GitHub Actions):

name: Send Email Notification on: schedule: - cron: '0 7 * * *' # 9 AM CET workflow_dispatch: # also enable on button click jobs: send_email: runs-on: ubuntu-latest steps: - name: Checkout code uses: actions/checkout@v2 - name: Set up Python uses: actions/setup-python@v2 with: python-version: '3.11' - name: Install dependencies run: | python -m pip install --upgrade pip pip install -r requirements.txt - name: Send email env: SENDGRID_API_KEY: ${{ secrets.SENDGRID_API_KEY }} FROM_EMAIL: ${{ secrets.FROM_EMAIL }} TO_EMAIL: ${{ secrets.TO_EMAIL }} run: python script.py

Explanation:

Scheduled Trigger: The workflow is configured to run automatically at a specific time—7:00 AM UTC (which corresponds to 9:00 AM CET)—every day, using the cron syntax in the schedule event. This ensures the email notifications are sent out consistently without manual intervention.

Manual Trigger Option: Besides the scheduled trigger, the workflow can also be manually triggered using the workflow_dispatch event. This flexibility allows users to run the workflow at any time, independent of the scheduled time, by clicking a button in the GitHub Actions interface.

Environment Setup: The workflow runs on the latest Ubuntu runner provided by GitHub Actions (ubuntu-latest). It begins by checking out the repository code using actions/checkout@v2 and then sets up the specified version of Python (3.11) with actions/setup-python@v2, preparing the environment for script execution.

Dependency Installation: To ensure all required Python libraries are available, the workflow includes a step to upgrade pip and install dependencies from the requirements.txt file. This step is crucial for the Python script to run successfully, as it relies on external libraries such as feedparser, python-decouple, and sendgrid.

Email Sending Execution: The final step of the workflow executes the Python script (script.py) that sends out the email notifications.

This step securely accesses the SendGrid API key and email addresses (for both sender and receiver) from GitHub Secrets (secrets.SENDGRID_API_KEY, secrets.FROM_EMAIL, and secrets.TO_EMAIL), ensuring sensitive information is kept secure and not exposed in the repository.

See GitHub’s documentation how to work with secrets. So locally I work with a hidden .env file, remotely I use secrets.

Result

And voilà: this was the email I received this morning with the Planet Python entries of the last 24 hours.

Conclusion

In this article I’ve showed you a way to parse Planet Python entries, filter on the latest ones (last 24 hours), and email the results, hooking it up with a GitHub Actions.

Not only will this action run once a day via its cronjob feature, thanks to workflow_dispatch you can also run it manually from the Action tab on the GitHub repo’s page.

Through this process we’ve learned about the feedparser, python-decouple, and sendgrid Python libraries and how to manage environment config variables, both locally and on GitHub using its Secrets feature.

I hope this helps you automate more tasks and leverage GitHub Actions.

Next steps

Check out the repo here. Feel free to contribute by opening issues and/or submitting code.

I am also happy to hear your thoughts in our community.

I had an interesting question from a reader of my PyQt6 book, about how to handle dragging and dropping of widgets in a container showing the dragged widget as it is moved.

I'm interested in managing movement of a QWidget with mouse in a container. I've implemented the application with drag & drop, exchanging the position of buttons, but I want to show the motion of QPushButton, like what you see in Qt Designer. Dragging a widget should show the widget itself, not just the mouse pointer.

First, we'll implement the simple case which drags widgets without showing anything extra. Then we can extend it to answer the question. By the end of this quick tutorial we'll have a generic drag drop implementation which looks like the following.

Table of Contents

• Drag & Drop Widgets
• Visual Drag & Drop
• Generic Drag & Drop Container
• Adding a Visual Drop Target

Drag & Drop Widgets

We'll start with a simple application which creates a window using QWidget and places a series of QPushButton widgets into it.

You can substitute QPushButton for any other widget you like, e.g. QLabel. Any widget can have drag behavior implemented on it, although some input widgets will not work well as we capture the mouse events for the drag.

python from PyQt6.QtWidgets import QApplication, QHBoxLayout, QPushButton, QWidget class Window(QWidget): def __init__(self): super().__init__() self.blayout = QHBoxLayout() for l in ["A", "B", "C", "D"]: btn = QPushButton(l) self.blayout.addWidget(btn) self.setLayout(self.blayout) app = QApplication([]) w = Window() w.show() app.exec()

If you run this you should see something like this.

The series of QPushButton widgets in a horizontal layout.

Here we're creating a window, but the Window widget is subclassed from QWidget, meaning you can add this widget to any other layout. See later for an example of a generic object sorting widget.

QPushButton objects aren't usually draggable, so to handle the mouse movements and initiate a drag we need to implement a subclass. We can add the following to the top of the file.

python from PyQt6.QtCore import QMimeData, Qt from PyQt6.QtGui import QDrag from PyQt6.QtWidgets import QApplication, QHBoxLayout, QPushButton, QWidget class DragButton(QPushButton): def mouseMoveEvent(self, e): if e.buttons() == Qt.MouseButton.LeftButton: drag = QDrag(self) mime = QMimeData() drag.setMimeData(mime) drag.exec(Qt.DropAction.MoveAction)

We implement a mouseMoveEvent which accepts the single e parameter of the event. We check to see if the left mouse button is pressed on this event -- as it would be when dragging -- and then initiate a drag. To start a drag, we create a QDrag object, passing in self to give us access later to the widget that was dragged. We also must pass in mime data. This is used for including information about what is dragged, particularly for passing data between applications. However, as here, it is fine to leave this empty.

Finally, we initiate a drag by calling drag.exec_(Qt.MoveAction). As with dialogs exec_() starts a new event loop, blocking the main loop until the drag is complete. The parameter Qt.MoveAction tells the drag handler what type of operation is happening, so it can show the appropriate icon tip to the user.

You can update the main window code to use our new DragButton class as follows.

python class Window(QWidget): def __init__(self): super().__init__() self.blayout = QHBoxLayout() for l in ["A", "B", "C", "D"]: btn = DragButton(l) self.blayout.addWidget(btn) self.setLayout(self.blayout)

If you run the code now, you can drag the buttons, but you'll notice the drag is forbidden.

Dragging of the widget starts but is forbidden.

What's happening? The mouse movement is being detected by our DragButton object and the drag started, but the main window does not accept drag & drop.

To fix this we need to enable drops on the window and implement dragEnterEvent to actually accept them.

python class Window(QWidget): def __init__(self): super().__init__() self.setAcceptDrops(True) self.blayout = QHBoxLayout() for l in ["A", "B", "C", "D"]: btn = DragButton(l) self.blayout.addWidget(btn) self.setLayout(self.blayout) def dragEnterEvent(self, e): e.accept()

If you run this now, you'll see the drag is now accepted and you see the move icon. This indicates that the drag has started and been accepted by the window we're dragging onto. The icon shown is determined by the action we pass when calling drag.exec_().

Dragging of the widget starts and is accepted, showing a move icon.

Releasing the mouse button during a drag drop operation triggers a dropEvent on the widget you're currently hovering the mouse over (if it is configured to accept drops). In our case that's the window. To handle the move we need to implement the code to do this in our dropEvent method.

The drop event contains the position the mouse was at when the button was released & the drop triggered. We can use this to determine where to move the widget to.

To determine where to place the widget, we iterate over all the widgets in the layout, until we find one who's x position is greater than that of the mouse pointer. If so then when insert the widget directly to the left of this widget and exit the loop.

If we get to the end of the loop without finding a match, we must be dropping past the end of the existing items, so we increment n one further (in the else: block below).

python def dropEvent(self, e): pos = e.position() widget = e.source() self.blayout.removeWidget(widget) for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if pos.x() < w.x(): # We didn't drag past this widget. # insert to the left of it. break else: # We aren't on the left hand side of any widget, # so we're at the end. Increment 1 to insert after. n += 1 self.blayout.insertWidget(n, widget) e.accept()

The effect of this is that if you drag 1 pixel past the start of another widget the drop will happen to the right of it, which is a bit confusing. To fix this we can adjust the cut off to use the middle of the widget using if pos.x() < w.x() + w.size().width() // 2: -- that is x + half of the width.

python def dropEvent(self, e): pos = e.position() widget = e.source() self.blayout.removeWidget(widget) for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if pos.x() < w.x() + w.size().width() // 2: # We didn't drag past this widget. # insert to the left of it. break else: # We aren't on the left hand side of any widget, # so we're at the end. Increment 1 to insert after. n += 1 self.blayout.insertWidget(n, widget) e.accept()

The complete working drag-drop code is shown below.

python from PyQt6.QtCore import QMimeData, Qt from PyQt6.QtGui import QDrag from PyQt6.QtWidgets import QApplication, QHBoxLayout, QPushButton, QWidget class DragButton(QPushButton): def mouseMoveEvent(self, e): if e.buttons() == Qt.MouseButton.LeftButton: drag = QDrag(self) mime = QMimeData() drag.setMimeData(mime) drag.exec(Qt.DropAction.MoveAction) class Window(QWidget): def __init__(self): super().__init__() self.setAcceptDrops(True) self.blayout = QHBoxLayout() for l in ["A", "B", "C", "D"]: btn = DragButton(l) self.blayout.addWidget(btn) self.setLayout(self.blayout) def dragEnterEvent(self, e): e.accept() def dropEvent(self, e): pos = e.position() widget = e.source() self.blayout.removeWidget(widget) for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if pos.x() < w.x() + w.size().width() // 2: # We didn't drag past this widget. # insert to the left of it. break else: # We aren't on the left hand side of any widget, # so we're at the end. Increment 1 to insert after. n += 1 self.blayout.insertWidget(n, widget) e.accept() app = QApplication([]) w = Window() w.show() app.exec() Visual Drag & Drop

We now have a working drag & drop implementation. Next we'll move onto improving the UX by showing the drag visually. First we'll add support for showing the button being dragged next to the mouse point as it is dragged. That way the user knows exactly what it is they are dragging.

Qt's QDrag handler natively provides a mechanism for showing dragged objects which we can use. We can update our DragButton class to pass a pixmap image to QDrag and this will be displayed under the mouse pointer as the drag occurs. To show the widget, we just need to get a QPixmap of the widget we're dragging.

python from PyQt6.QtCore import QMimeData, Qt from PyQt6.QtGui import QDrag, QPixmap from PyQt6.QtWidgets import QApplication, QHBoxLayout, QPushButton, QWidget class DragButton(QPushButton): def mouseMoveEvent(self, e): if e.buttons() == Qt.MouseButton.LeftButton: drag = QDrag(self) mime = QMimeData() drag.setMimeData(mime) pixmap = QPixmap(self.size()) self.render(pixmap) drag.setPixmap(pixmap) drag.exec(Qt.DropAction.MoveAction)

To create the pixmap we create a QPixmap object passing in the size of the widget this event is fired on with self.size(). This creates an empty pixmap which we can then pass into self.render to render -- or draw -- the current widget onto it. That's it. Then we set the resulting pixmap on the drag object.

If you run the code with this modification you'll see something like the following --

Dragging of the widget showing the dragged widget.

Generic Drag & Drop Container

We now have a working drag and drop behavior implemented on our window. We can take this a step further and implement a generic drag drop widget which allows us to sort arbitrary objects. In the code below we've created a new widget DragWidget which can be added to any window.

You can add items -- instances of DragItem -- which you want to be sorted, as well as setting data on them. When items are re-ordered the new order is emitted as a signal orderChanged.

python from PyQt6.QtCore import QMimeData, Qt, pyqtSignal from PyQt6.QtGui import QDrag, QPixmap from PyQt6.QtWidgets import ( QApplication, QHBoxLayout, QLabel, QMainWindow, QVBoxLayout, QWidget, ) class DragItem(QLabel): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setContentsMargins(25, 5, 25, 5) self.setAlignment(Qt.AlignmentFlag.AlignCenter) self.setStyleSheet("border: 1px solid black;") # Store data separately from display label, but use label for default. self.data = self.text() def set_data(self, data): self.data = data def mouseMoveEvent(self, e): if e.buttons() == Qt.MouseButton.LeftButton: drag = QDrag(self) mime = QMimeData() drag.setMimeData(mime) pixmap = QPixmap(self.size()) self.render(pixmap) drag.setPixmap(pixmap) drag.exec(Qt.DropAction.MoveAction) class DragWidget(QWidget): """ Generic list sorting handler. """ orderChanged = pyqtSignal(list) def __init__(self, *args, orientation=Qt.Orientation.Vertical, **kwargs): super().__init__() self.setAcceptDrops(True) # Store the orientation for drag checks later. self.orientation = orientation if self.orientation == Qt.Orientation.Vertical: self.blayout = QVBoxLayout() else: self.blayout = QHBoxLayout() self.setLayout(self.blayout) def dragEnterEvent(self, e): e.accept() def dropEvent(self, e): pos = e.position() widget = e.source() self.blayout.removeWidget(widget) for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if self.orientation == Qt.Orientation.Vertical: # Drag drop vertically. drop_here = pos.y() < w.y() + w.size().height() // 2 else: # Drag drop horizontally. drop_here = pos.x() < w.x() + w.size().width() // 2 if drop_here: break else: # We aren't on the left hand/upper side of any widget, # so we're at the end. Increment 1 to insert after. n += 1 self.blayout.insertWidget(n, widget) self.orderChanged.emit(self.get_item_data()) e.accept() def add_item(self, item): self.blayout.addWidget(item) def get_item_data(self): data = [] for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() data.append(w.data) return data class MainWindow(QMainWindow): def __init__(self): super().__init__() self.drag = DragWidget(orientation=Qt.Orientation.Vertical) for n, l in enumerate(["A", "B", "C", "D"]): item = DragItem(l) item.set_data(n) # Store the data. self.drag.add_item(item) # Print out the changed order. self.drag.orderChanged.connect(print) container = QWidget() layout = QVBoxLayout() layout.addStretch(1) layout.addWidget(self.drag) layout.addStretch(1) container.setLayout(layout) self.setCentralWidget(container) app = QApplication([]) w = MainWindow() w.show() app.exec()

Generic drag-drop sorting in horizontal orientation.

You'll notice that when creating the item, you can set the label by passing it in as a parameter (just like for a normal QLabel which we've subclassed from). But you can also set a data value, which is the internal value of this item -- this is what will be emitted when the order changes, or if you call get_item_data yourself. This separates the visual representation from what is actually being sorted, meaning you can use this to sort anything not just strings.

In the example above we're passing in the enumerated index as the data, so dragging will output (via the print connected to orderChanged) something like:

python [1, 0, 2, 3] [1, 2, 0, 3] [1, 0, 2, 3] [1, 2, 0, 3]

If you remove the item.set_data(n) you'll see the labels emitted on changes.

python ['B', 'A', 'C', 'D'] ['B', 'C', 'A', 'D']

We've also implemented orientation onto the DragWidget using the Qt built in flags Qt.Orientation.Vertical or Qt.Orientation.Horizontal. This setting this allows you sort items either vertically or horizontally -- the calculations are handled for both directions.

Generic drag-drop sorting in vertical orientation.

Adding a Visual Drop Target

If you experiment with the drag-drop tool above you'll notice that it doesn't feel completely intuitive. When dragging you don't know where an item will be inserted until you drop it. If it ends up in the wrong place, you'll then need to pick it up and re-drop it again, using guesswork to get it right.

With a bit of practice you can get the hang of it, but it would be nicer to make the behavior immediately obvious for users. Many drag-drop interfaces solve this problem by showing a preview of where the item will be dropped while dragging -- either by showing the item in the place where it will be dropped, or showing some kind of placeholder.

In this final section we'll implement this type of drag and drop preview indicator.

The first step is to define our target indicator. This is just another label, which in our example is empty, with custom styles applied to make it have a solid "shadow" like background. This makes it obviously different to the items in the list, so it stands out as something distinct.

python from PyQt6.QtCore import QMimeData, Qt, pyqtSignal from PyQt6.QtGui import QDrag, QPixmap from PyQt6.QtWidgets import ( QApplication, QHBoxLayout, QLabel, QMainWindow, QVBoxLayout, QWidget, ) class DragTargetIndicator(QLabel): def __init__(self, parent=None): super().__init__(parent) self.setContentsMargins(25, 5, 25, 5) self.setStyleSheet( "QLabel { background-color: #ccc; border: 1px solid black; }" )

We've copied the contents margins from the items in the list. If you change your list items, remember to also update the indicator dimensions to match.

The drag item is unchanged, but we need to implement some additional behavior on our DragWidget to add the target, control showing and moving it.

First we'll add the drag target indicator to the layout on our DragWidget. This is hidden to begin with, but will be shown during the drag.

python class DragWidget(QWidget): """ Generic list sorting handler. """ orderChanged = pyqtSignal(list) def __init__(self, *args, orientation=Qt.Orientation.Vertical, **kwargs): super().__init__() self.setAcceptDrops(True) # Store the orientation for drag checks later. self.orientation = orientation if self.orientation == Qt.Orientation.Vertical: self.blayout = QVBoxLayout() else: self.blayout = QHBoxLayout() # Add the drag target indicator. This is invisible by default, # we show it and move it around while the drag is active. self._drag_target_indicator = DragTargetIndicator() self.blayout.addWidget(self._drag_target_indicator) self._drag_target_indicator.hide() self.setLayout(self.blayout)

Next we modify the DragWidget.dragMoveEvent to show the drag target indicator. We show it by inserting it into the layout and then calling .show -- inserting a widget which is already in a layout will move it. We also hide the original item which is being dragged.

In the earlier examples we determined the position on drop by removing the widget being dragged, and then iterating over what is left. Because we now need to calculate the drop location before the drop, we take a different approach.

If we wanted to do it the same way, we'd need to remove the item on drag start, hold onto it and implement re-inserting at it's old position on drag fail. That's a lot of work.

Instead, the dragged item is left in place and hidden during move.

python def dragMoveEvent(self, e): # Find the correct location of the drop target, so we can move it there. index = self._find_drop_location(e) if index is not None: # Inserting moves the item if its alreaady in the layout. self.blayout.insertWidget(index, self._drag_target_indicator) # Hide the item being dragged. e.source().hide() # Show the target. self._drag_target_indicator.show() e.accept()

The method self._find_drop_location finds the index where the drag target will be shown (or the item dropped when the mouse released). We'll implement that next.

The calculation of the drop location follows the same pattern as before. We iterate over the items in the layout and calculate whether our mouse drop location is to the left of each widget. If it isn't to the left of any widget, we drop on the far right.

python def _find_drop_location(self, e): pos = e.position() spacing = self.blayout.spacing() / 2 for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if self.orientation == Qt.Orientation.Vertical: # Drag drop vertically. drop_here = ( pos.y() >= w.y() - spacing and pos.y() <= w.y() + w.size().height() + spacing ) else: # Drag drop horizontally. drop_here = ( pos.x() >= w.x() - spacing and pos.x() <= w.x() + w.size().width() + spacing ) if drop_here: # Drop over this target. break return n

The drop location n is returned for use in the dragMoveEvent to place the drop target indicator.

Next wee need to update the get_item_data handler to ignore the drop target indicator. To do this we check w against self._drag_target_indicator and skip if it is the same. With this change the method will work as expected.

python def get_item_data(self): data = [] for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if w != self._drag_target_indicator: # The target indicator has no data. data.append(w.data) return data

If you run the code a this point the drag behavior will work as expected. But if you drag the widget outside of the window and drop you'll notice a problem: the target indicator will stay in place, but dropping the item won't drop the item in that position (the drop will be cancelled).

To fix that we need to implement a dragLeaveEvent which hides the indicator.

python def dragLeaveEvent(self, e): self._drag_target_indicator.hide() e.accept()

With those changes, the drag-drop behavior should be working as intended. The complete code is shown below.

python from PyQt6.QtCore import QMimeData, Qt, pyqtSignal from PyQt6.QtGui import QDrag, QPixmap from PyQt6.QtWidgets import ( QApplication, QHBoxLayout, QLabel, QMainWindow, QVBoxLayout, QWidget, ) class DragTargetIndicator(QLabel): def __init__(self, parent=None): super().__init__(parent) self.setContentsMargins(25, 5, 25, 5) self.setStyleSheet( "QLabel { background-color: #ccc; border: 1px solid black; }" ) class DragItem(QLabel): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setContentsMargins(25, 5, 25, 5) self.setAlignment(Qt.AlignmentFlag.AlignCenter) self.setStyleSheet("border: 1px solid black;") # Store data separately from display label, but use label for default. self.data = self.text() def set_data(self, data): self.data = data def mouseMoveEvent(self, e): if e.buttons() == Qt.MouseButton.LeftButton: drag = QDrag(self) mime = QMimeData() drag.setMimeData(mime) pixmap = QPixmap(self.size()) self.render(pixmap) drag.setPixmap(pixmap) drag.exec(Qt.DropAction.MoveAction) class DragWidget(QWidget): """ Generic list sorting handler. """ orderChanged = pyqtSignal(list) def __init__(self, *args, orientation=Qt.Orientation.Vertical, **kwargs): super().__init__() self.setAcceptDrops(True) # Store the orientation for drag checks later. self.orientation = orientation if self.orientation == Qt.Orientation.Vertical: self.blayout = QVBoxLayout() else: self.blayout = QHBoxLayout() # Add the drag target indicator. This is invisible by default, # we show it and move it around while the drag is active. self._drag_target_indicator = DragTargetIndicator() self.blayout.addWidget(self._drag_target_indicator) self._drag_target_indicator.hide() self.setLayout(self.blayout) def dragEnterEvent(self, e): e.accept() def dragLeaveEvent(self, e): self._drag_target_indicator.hide() e.accept() def dragMoveEvent(self, e): # Find the correct location of the drop target, so we can move it there. index = self._find_drop_location(e) if index is not None: # Inserting moves the item if its alreaady in the layout. self.blayout.insertWidget(index, self._drag_target_indicator) # Hide the item being dragged. e.source().hide() # Show the target. self._drag_target_indicator.show() e.accept() def dropEvent(self, e): widget = e.source() # Use drop target location for destination, then remove it. self._drag_target_indicator.hide() index = self.blayout.indexOf(self._drag_target_indicator) if index is not None: self.blayout.insertWidget(index, widget) self.orderChanged.emit(self.get_item_data()) widget.show() self.blayout.activate() e.accept() def _find_drop_location(self, e): pos = e.position() spacing = self.blayout.spacing() / 2 for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if self.orientation == Qt.Orientation.Vertical: # Drag drop vertically. drop_here = ( pos.y() >= w.y() - spacing and pos.y() <= w.y() + w.size().height() + spacing ) else: # Drag drop horizontally. drop_here = ( pos.x() >= w.x() - spacing and pos.x() <= w.x() + w.size().width() + spacing ) if drop_here: # Drop over this target. break return n def add_item(self, item): self.blayout.addWidget(item) def get_item_data(self): data = [] for n in range(self.blayout.count()): # Get the widget at each index in turn. w = self.blayout.itemAt(n).widget() if w != self._drag_target_indicator: # The target indicator has no data. data.append(w.data) return data class MainWindow(QMainWindow): def __init__(self): super().__init__() self.drag = DragWidget(orientation=Qt.Orientation.Vertical) for n, l in enumerate(["A", "B", "C", "D"]): item = DragItem(l) item.set_data(n) # Store the data. self.drag.add_item(item) # Print out the changed order. self.drag.orderChanged.connect(print) container = QWidget() layout = QVBoxLayout() layout.addStretch(1) layout.addWidget(self.drag) layout.addStretch(1) container.setLayout(layout) self.setCentralWidget(container) app = QApplication([]) w = MainWindow() w.show() app.exec()

If you run this example on macOS you may notice that the widget drag preview (the QPixmap created on DragItem) is a bit blurry. On high-resolution screens you need to set the device pixel ratio and scale up the pixmap when you create it. Below is a modified DragItem class which does this.

Update DragItem to support high resolution screens.

python class DragItem(QLabel): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setContentsMargins(25, 5, 25, 5) self.setAlignment(Qt.AlignmentFlag.AlignCenter) self.setStyleSheet("border: 1px solid black;") # Store data separately from display label, but use label for default. self.data = self.text() def set_data(self, data): self.data = data def mouseMoveEvent(self, e): if e.buttons() == Qt.MouseButton.LeftButton: drag = QDrag(self) mime = QMimeData() drag.setMimeData(mime) # Render at x2 pixel ratio to avoid blur on Retina screens. pixmap = QPixmap(self.size().width() * 2, self.size().height() * 2) pixmap.setDevicePixelRatio(2) self.render(pixmap) drag.setPixmap(pixmap) drag.exec(Qt.DropAction.MoveAction)

That's it! We've created a generic drag-drop handled which can be added to any projects where you need to be able to reposition items within a list. You should feel free to experiment with the styling of the drag items and targets as this won't affect the behavior.