Feeds

"When we choose inaccessibility, we are disabling people," says AmyJune Hineline. She was speaking to TDT over an email interview with Alethia Braganza prior to the upcoming Florida DrupalCamp for which, she is one of the lead organizers. Read on to know more.

Please join the Drupal Association in welcoming our new board member:

Lynne Capozzi

When accepting her position, Lynne shared: “I’m excited to join the Drupal Association Board, and I’m hopeful that my past experience at Acquia and my nonprofit and marketing experience can benefit the community and help the community to grow!”

About Lynne
Most recently, Lynne was the Chief Marketing Officer (CMO) at Acquia in Boston. Lynne was at Acquia for eight years in this role.

As Acquia’s CMO, Lynne Capozzi oversaw all global marketing functions, including digital marketing, demand generation, operations, regional and field marketing, customer and partner marketing, events, vertical strategy, analyst relations, content, and corporate communications.
Lynne is one of Acquia’s boomerang stories, first serving as Acquia’s CMO in 2009. Lynne left Acquia in 2011 to pursue her nonprofit work full-time and then returned to Acquia in late 2016 to lead the marketing organization into its next growth stage.

Prior to her experience at Acquia, Lynne held various marketing leadership roles in the technology space. She served as CMO at JackBe, an enterprise mashup software company for real-time intelligence applications that Software AG acquired. Before that, Lynne was CMO at Systinet, which Mercury Interactive acquired. Lynne was also a VP at Lotus Development, which IBM later acquired.

Lynne is on the board of directors at the Boston Children’s Hospital Trust and runs a nonprofit through the hospital. She is also on the Advisory Board of Family Services of Merrimack Valley and the Board chair of the West Parish Garden Cemetery and Chapel in Andover, Mass.

We are thrilled to have Lynne on the Drupal Association Board!

In this post I will give a quick tutorial on how to set up fast Debian package builds using sbuild with mmdebstrap and apt-cacher-ng.

1. Background
2. Setting up apt-cacher-ng
3. Setting up mmdebstrap
4. Setting up sbuild
5. Finishing touches

Background

The usual tool for building Debian packages is dpkg-buildpackage, or a user-friendly wrapper like debuild. These are geat tools, however if you want to upload something to the Debian archive they lack the required separation from the system they are run on to ensure that your packaging also works on a different system. The usual candidate here is sbuild. But setting up a schroot is tedious and performance tuning can be annoying. There is an alternative backend for sbuild that promises to make everything simpler: unshare. In this tutorial I will show you how to set up sbuild with this backend.

Additionally to the normal performance tweaking, caching downloaded packages can be a huge performance increase when rebuilding packages. I do rebuilds quite often, mostly when a new dependency got introduced I didn’t specify in debian/control yet or lintian notices a something I can easily fix. So let’s begin with setting up this caching.

Setting up apt-cacher-ng

Install apt-cacher-ng:

sudo apt install apt-cacher-ng

A pop-up will appear, if you are unsure how to answer it select no, we don’t need it for this use-case.

To enable apt-cacher-ng on your system, create /etc/apt/apt.conf.d/02proxy and insert:

Acquire::http::proxy "http://127.0.0.1:3142"; Acquire::https::proxy "DIRECT";

In /etc/apt-cacher-ng/acng.conf you can increase the value of ExThreshold to hold packages for a shorter or longer duration. The length depends on your specific use case and resources. A longer threshold takes more disk space, a short threshold like one day effecitvely only reduces the build time for rebuilds.

If you encounter weird issues on apt update at some point the future, you can try to clean the cache from apt-cacher-ng. You can use this script:

Setting up mmdebstrap

Install mmdebstrap:

sudo apt install mmdebstrap

We will create a small helper script to ease creating a chroot. Open ~/.local/bin/mmupdate and insert:

#!/bin/sh mmdebstrap \ --variant=buildd \ --aptopt='Acquire::http::proxy "http://127.0.0.1:3142";' \ --arch=amd64 \ --components=main,contrib,non-free \ unstable \ ~/.cache/sbuild/unstable-amd64.tar.xz \ http://deb.debian.org/debian

Notes:

• aptopt enables apt-cacher-ng inside the chroot.
• --arch sets the CPU architecture (see Debian Wiki).
• --components sets the archive components, if you don’t want non-free pacakges you might want to remove some entries here.
• unstable sets the Debian release, you can also set for example bookworm-backports here.
• unstable-amd64.tar.xz is the output tarball containing the chroot, change accordingly to your pick of the CPU architecture and Debian release.
• http://deb.debian.org/debian is the Debian mirror, you should set this to the same one you use in your /etc.apt/sources.list.

Make mmupdate executable and run it once:

chmod +x ~/.local/bin/mmupdate mkdir -p ~/.cache/sbuild ~/.local/bin/mmupdate

If you execute mmupdate again you can see that the downloading stage is much faster thanks to apt-cacher-ng. For me the difference is from about 115s to about 95s. Your results may vary, this depends on the speed of your internet, Debian mirror and disk.

If you have used the schroot backend and sbuild-update before, you probably notice that creating a new chroot with mmdebstrap is slower. It would be a bit annoying to do this manually before we start a new Debian packaging session, so let’s create a systemd service that does this for us.

First create a folder for user services:

mkdir -p ~/.config/systemd/user

Create ~/.config/systemd/user/mmupdate.service and add:

[Unit] Description=Run mmupdate Wants=network-online.target [Service] Type=oneshot ExecStart=%h/.local/bin/mmupdate

Start the service and test that it works:

systemctl --user daemon-reload systemctl --user start mmupdate systemctl --user status mmupdate

Create ~/.config/systemd/user/mmupdate.timer:

[Unit] Description=Run mmupdate daily [Timer] OnCalendar=daily Persistent=true [Install] WantedBy=timers.target

Enable the timer:

systemctl --user enable mmupdate.timer

Now every day mmupdte will be run automatically. You can adjust the period if you think daily rebuilds are a bit excessive.

A neat advantage of period rebuilds is that they the base files in your apt-cacher-ng cache warm every time they run.

Setting up sbuild:

Install sbuild and (optionally) autopkgtest:

sudo apt install --no-install-recommends sbuild autopkgtest

Create ~/.sbuildrc and insert:

# backend for using mmdebstrap chroots $chroot_mode = 'unshare'; # build in tmpfs $unshare_tmpdir_template = '/dev/shm/tmp.sbuild.XXXXXXXXXX'; # upgrade before starting build $apt_update = 1; $apt_upgrade = 1; # build everything including source for source-only uploads $build_arch_all = 1; $build_arch_any = 1; $build_source = 1; $source_only_changes = 1; # go to shell on failure instead of exiting $external_commands = { "build-failed-commands" => [ [ '%SBUILD_SHELL' ] ] }; # always clean build dir, even on failure $purge_build_directory = "always"; # run lintian $run_lintian = 1; $lintian_opts = [ '-i', '-I', '-E', '--pedantic' ]; # do not run piuparts $run_piuparts = 0; # run autopkgtest $run_autopkgtest = 1; $autopkgtest_root_args = ''; $autopkgtest_opts = [ '--apt-upgrade', '--', 'unshare', '--release', '%r', '--arch', '%a' ]; # set uploader for correct signing $uploader_name = 'Stephan Lachnit <stephanlachnit@debian.org>';

You should adjust uploader_name. If you don’t want to run autopkgtest or lintian by default you can also disable it here. Note that for packages that need a lot of space for building, you might want to comment the unshare_tmpdir_template line to prevent a OOM build failure.

You can now build your Debian packages with the sbuild command :)

Finishing touches

You can add these variables to your ~/.bashrc as bonus (with adjusted name / email):

export DEBFULLNAME="<your_name>" export DEBEMAIL="<your_email>" export DEB_BUILD_OPTIONS="parallel=<threads>"

In particular adjust the value of parallel to ensure parallel builds.

If you are new to signing / uploading your package, first install the required tools:

sudo apt install devscripts dput-ng

Create ~/.devscripts and insert:

DEBSIGN_KEYID=<your_gpg_fingerpring> USCAN_SYMLINK=rename

You can now sign the .changes file with:

debsign ../<pkgname_version_arch>.changes

And for source-only uploads with:

debsign -S ../<pkgname_version_arch>_source.changes

If you don’t introduce a new binary package, you always want to go with source-only changes.

You can now upload the package to Debian with

dput ../<filename>.changes Resources for further reading:

https://wiki.debian.org/sbuild
https://www.unix-ag.uni-kl.de/~bloch/acng/html/index.html
https://wiki.ubuntu.com/SimpleSbuild
https://wiki.archlinux.org/title/Systemd/Timers
Thanks for reading!

FTP master

This month I accepted 419 and rejected 46 packages. The overall number of packages that got accepted was 429. Looking at these numbers and comparing them to the previous month, one can see: the freeze is near. Everybody wants to get some packages into the archive and I hope nobody is disappointed.

Debian LTS

This was my hundred-third month that I did some work for the Debian LTS initiative, started by Raphael Hertzog at Freexian. 

This month my all in all workload has been 14h.

During that time I uploaded:

• [DLA 3272-1] sudo (embargoed) security update for one CVE
• [DLA 3286-1] tor security update for one CVE
• [DLA 3290-1] libzen security update for one CVE
• [libzen Bullseye] debdiff sent to maintainer
• [DLA 3294-1] libarchive security update for one CVE

I also attended the monthly LTS meeting and did some days of frontdesk duties.

Debian ELTS

This month was the fifty fourth ELTS month.

• [ELA-772-1] sudo security update of Jessie and Stretch for one CVE
• [ELA-781-1] libzen security update of Stretch for one CVE
• [ELA-782-1] xorg-server security update of Jessie and Stretch for six CVEs
• [ELA-790-1] libarchive security update of Jessie and Stretch for one CVEs

Last but not least I did some days of frontdesk duties.

Debian Astro

This month I uploaded improved packages or new versions of:

• … indi-asi
• … indi-aagcloudwatcher-ng
• … indi-asi
• … indi-playerone
• … indi-eqmod

I also uploaded new packages:

• … libdfu-ahp
• … libahp-gt
• … libdfu-ahp
• … liburjtag
• … libahp-xc

Debian IoT

This month I uploaded improved packages of:

• … openzwave-controlpanel

Debian Printing

This month I uploaded new versions or improved packages of:

• … foomatic-db
• … hannah-foo2zjs
• … hplip

I also uploaded new packages:

• … cpdb-libs
• … cpdb-backend-file
• … cpdb-backend-cups

These new packages are already preparations for the new CUPS architecture. This architecture will be introduced after the release of Bookworm and shall be ready at least in Trixie. Parts of this work is generously funded by Freexian!

Other stuff

This month I uploaded improved packages of:

• … golang-github-mitch000001-go-hbci
• … golang-github-crewjam-saml
• … golang-github-jaksi-sshutils

The server that normally serves this page, all my email, and many more services was unavailable for about 24 hours. This post explains how and why.

What happened?

Starting February 2nd, I started seeing intermittent packet loss on the network. Every hour or so, the link would go down for one or two minutes, then come back up.

At first, I didn't think much of it because I was away and could blame the crappy wifi or the uplink I using. But when I came in the office on Monday, the service was indeed seriously degraded. I could barely do videoconferencing calls as they would cut out after about half an hour.

I opened a ticket with my uplink, Oricom. They replied that it was an issue they couldn't fix on their end and would need someone on site to fix.

So, the next day (Tuesday, at around 10EST) I called Oricom again, and they made me do a full modem reset, which involves plugging a pin in a hole for 15 seconds on the Technicolor TC4400 cable modem. Then the link went down, and it didn't come back up at all.

Boom.

Oricom then escalated this to their upstream (Oricom is a reseller of Videotron, who has basically the monopoly on cable in Québec) which dispatched a tech. This tech, in turn, arrived some time after lunch and said the link worked fine and it was a hardware issue.

At this point, Oricom put a new modem in the mail and I started mitigation.

Mitigation Website

The first thing I did, weirdly, was trying to rebuild this blog. I figured it should be pretty simple: install ikiwiki and hit rebuild. I knew I had some patches on ikiwiki to deploy, but surely those are not a deal breaker, right?

Nope. Turns out I wrote many plugins and those still don't ship with ikiwiki, despite having been sent upstream a while back, some years ago.

So I deployed the plugins inside the .ikiwiki directory of the site in the hope of making things a little more "standalone". Unfortunately, that didn't work either because the theme must be shipped in the system-wide location: I couldn't figure out how to put it to have it bundled with the main repository. At that point I mostly gave up because I had spent too much time on this and I had to do something about email otherwise it would start to bounce.

Email

So I made a new VM at Linode (thanks 2.5admins for the credits) to build a new mail server.

This wasn't the best idea, in retrospect, because it was really overkill: I started rebuilding the whole mail server from scratch.

Ideally, this would be in Puppet and I would just deploy the right profile and the server would be rebuilt. Unfortunately, that part of my infrastructure is not Puppetized and even if it would, well the Puppet server was also down so I would have had to bring that up first.

At first, I figured I would just make a secondary mail exchanger (MX), to spool mail for longer so that I wouldn't lose it. But I decided against that: I thought it was too hard to make a "proper" MX as it needs to also filter mail while avoiding backscatter. Might as well just build a whole new server! I had a copy of my full mail spool on my laptop, so I figured that was possible.

I mostly got this right: added a DKIM key, installed Postfix, Dovecot, OpenDKIM, OpenDMARC, glue it all together, and voilà, I had a mail server. Oh, and spampd. Oh, and I need the training data, oh, and this and... I wasn't done and it was time to sleep.

The mail server went online this morning, and started accepting mail. I tried syncing my laptop mail spool against it, but that failed because Dovecot generated new UIDs for the emails, and isync correctly failed to sync. I tried to copy the UIDs from the server in the office (which I had still access to locally), but that somehow didn't work either.

But at least the mail was getting delivered and stored properly. I even had the Sieve rules setup so it would get sorted properly too. Unfortunately, I didn't hook that up properly, so those didn't actually get sorted. Thankfully, Dovecot can re-filter emails with the sieve-filter command, so that was fixed later.

At this point, I started looking for other things to fix.

Web, again

I figured I was almost done with the website, might as well publish it. So I installed the Nginx Debian package, got a cert with certbot, and added the certs to the default configuration. I rsync'd my build in /var/www/html and boom, I had a website. The Goatcounter analytics were timing out, but that was easy to turn off.

Resolution

Almost at that exact moment, a bang on the door told me mail was here and I had the modem. I plugged it in and a few minutes later, marcos was back online.

So this was a lot (a lot!) of work for basically nothing. I could have just taken the day off and wait for the package to be delivered. It would definitely have been better to make a simpler mail exchanger to spool the mail to avoid losing it. And in fact, that's what I eventually ended up doing: I converted the linode server in a mail relay to continue accepting mail with DNS propagates, but without having to sort the mail out of there...

Right now I have about 200 mails in a mailbox that I need to move back into marcos. Normally, this would just be a simple rsync, but because both servers have accepted mail simultaneously, it's going to be simpler to just move those exact mails on there. Because dovecot helpfully names delivered files with the hostname it's running on, it's easy to find those files and transfer them, basically:

rsync -v -n --files-from=<(ssh colette.anarc.at find Maildir -name '*colette*' ) colette.anarc.at: colette/ rsync -v -n --files-from=<(ssh colette.anarc.at find Maildir -name '*colette*' ) colette/ marcos.anarc.at:

Overall, the outage lasted about 24 hours, from 11:00EST (16:00UTC) on 2023-02-07 to the same time today.

Future work

I'll probably keep a mail relay to make those situations more manageable in the future. At first I thought that mail filtering would be a problem, but that happens post queue anyways and I don't bounce mail based on Spamassassin, so back-scatter shouldn't be an issue.

I basically need Postfix, OpenDMARC, and Postgrey. I'm not even sure I need OpenDKIM as the server won't process outgoing mail, so it doesn't need to sign anything, just check incoming signatures, which OpenDMARC can (probably?) do.

Thanks to everyone who supported me through this ordeal, you know who you are (and I'm happy to give credit here if you want to be deanonymized)!

As announced previously, Plasma 5.27 will have a significantly reworked multiscreen management, and we want to make sure this will be the best LTS Plasma release we had so far.

Of course, this doesn’t mean it will be perfect from day one, and your feedback is really important, as we want to fix any potential issue as fast as they get noticed.

As you know, for our issue tracking we use Bugzilla at this address. We have different products and components that are involved in the multiscreen management.

First, under New bug, chose the “plasma” category. Then there are 4 possible combinations of products and components, depending on the symptoms:

Possible problemProductComponent

• The output of the command kscreen-doctor -o looks wrong, such as:
• The listed “priority” is not the one you set in systemsettings
• Geometries look wrong

kscreencommon

• Desktops or panels are on the wrong screen
• There are black screens but is possible to move the cursor inside them

plasmashellMulti Screen Support

• Ordinary application windows appear on the wrong screen or get moved in unexpected screens when screens are connected/disconnected
• Some screens are black and is not possible to move the mouse inside those, but they look enabled in the systemsettings displays module or in the output of the command kscreen-doctor -o

kwinmulti-screen

• The systemsettings displays module shows settings that don’t match reality
• The systemsettings displays module shows settings that don’t match the output of the command kscreen-doctor -o

systemsettingskcm_kscreen

In order to have a good complete information on the affected system, its configuration, and the configuration of our multiscreen management, if you can, the following information would be needed:

• Whether the problem happens in a Wayland or X11 session (or both)
• A good description of the scenario: how many screens, whether is a laptop or desktop, when the problem happens (startup, connecting/disconnectiong, going out of sleep and things like that)
• The output the terminal command: kscreen-doctor -o
• The output of the terminal command: kscreen-console
• The main plasma configuration file: ~/.config/plasma-org.kde.plasma.desktop-appletsrc

Those items of information already help a lot figuring out what problem is and where it resides.

Afterwards we still may ask for more informations, like an archive of the main screen config files that are the directory content of ~/.local/share/kscreen/ but normally, we wouldn’t need that.

One more word on kscreen-doctor and kscreen-console

Those 2 commands are very useful to understand what Plasma and the rest of the system thinks about every screen that’s connected and how they intend to treat them.

kscreen-doctor

Here is a typical output of the command kscreen-doctor - o:

Output: 1 eDP-1 enabled connected priority 2 Panel Modes: 0:1200x1920@60! 1:1024x768@60 Geometry: 1920,0 960x600 Scale: 2 Rotation: 8 Overscan: 0 Vrr: incapable RgbRange: Automatic Output: 2 DP-3 enabled connected priority 3 DisplayPort Modes: 0:1024x768@60! 1:800x600@60 2:800x600@56 3:848x480@60 4:640x480@60 5:1024x768@60 Geometry: 1920,600 1024x768 Scale: 1 Rotation: 1 Overscan: 0 Vrr: incapable RgbRange: Automatic Output: 3 DP-4 enabled connected priority 1 DisplayPort Modes: 0:1920x1080@60*! 1:1920x1080@60 2:1920x1080@60 3:1680x1050@60 4:1600x900@60 5:1280x1024@75 6:1280x1024@60 7:1440x900@60 8:1280x800@60 9:1152x864@75 10:1280x720@60 11:1280x720@60 12:1280x720@60 13:1024x768@75 14:1024x768@70 15:1024x768@60 16:832x624@75 17:800x600@75 18:800x600@72 19:800x600@60 20:800x600@56 21:720x480@60 22:720x480@60 23:720x480@60 24:720x480@60 25:640x480@75 26:640x480@73 27:640x480@67 28:640x480@60 29:640x480@60 30:720x400@70 31:1280x1024@60 32:1024x768@60 33:1280x800@60 34:1920x1080@60 35:1600x900@60 36:1368x768@60 37:1280x720@60 Geometry: 0,0 1920x1080 Scale: 1 Rotation: 1 Overscan: 0 Vrr: incapable RgbRange: Automatic

Here we can see we have 3 outputs, one internal and two via DisplayPort, DP-4 is the primary (priority 1) followed by eDP-1 (internal) and DP-3 (those correcpond to the new reordering UI in the systemsettings screen module).

Important data points, also the screen geometries (in italic in the snippet) which tell their relative positions.

kscreen-console

This gives a bit more verbose information, here is a sample (copied here the data of a single screen, as the output is very long):

Id: 3 Name: "DP-4" Type: "DisplayPort" Connected: true Enabled: true Priority: 1 Rotation: KScreen::Output::None Pos: QPoint(0,0) MMSize: QSize(520, 290) FollowPreferredMode: false Size: QSize(1920, 1080) Scale: 1 Clones: None Mode: "0" Preferred Mode: "0" Preferred modes: ("0") Modes: "0" "1920x1080@60" QSize(1920, 1080) 60 "1" "1920x1080@60" QSize(1920, 1080) 60 "10" "1280x720@60" QSize(1280, 720) 60 "11" "1280x720@60" QSize(1280, 720) 60 "12" "1280x720@60" QSize(1280, 720) 59.94 "13" "1024x768@75" QSize(1024, 768) 75.029 "14" "1024x768@70" QSize(1024, 768) 70.069 "15" "1024x768@60" QSize(1024, 768) 60.004 "16" "832x624@75" QSize(832, 624) 74.551 "17" "800x600@75" QSize(800, 600) 75 "18" "800x600@72" QSize(800, 600) 72.188 "19" "800x600@60" QSize(800, 600) 60.317 "2" "1920x1080@60" QSize(1920, 1080) 59.94 "20" "800x600@56" QSize(800, 600) 56.25 "21" "720x480@60" QSize(720, 480) 60 "22" "720x480@60" QSize(720, 480) 60 "23" "720x480@60" QSize(720, 480) 59.94 "24" "720x480@60" QSize(720, 480) 59.94 "25" "640x480@75" QSize(640, 480) 75 "26" "640x480@73" QSize(640, 480) 72.809 "27" "640x480@67" QSize(640, 480) 66.667 "28" "640x480@60" QSize(640, 480) 60 "29" "640x480@60" QSize(640, 480) 59.94 "3" "1680x1050@60" QSize(1680, 1050) 59.883 "30" "720x400@70" QSize(720, 400) 70.082 "31" "1280x1024@60" QSize(1280, 1024) 59.895 "32" "1024x768@60" QSize(1024, 768) 59.92 "33" "1280x800@60" QSize(1280, 800) 59.81 "34" "1920x1080@60" QSize(1920, 1080) 59.963 "35" "1600x900@60" QSize(1600, 900) 59.946 "36" "1368x768@60" QSize(1368, 768) 59.882 "37" "1280x720@60" QSize(1280, 720) 59.855 "4" "1600x900@60" QSize(1600, 900) 60 "5" "1280x1024@75" QSize(1280, 1024) 75.025 "6" "1280x1024@60" QSize(1280, 1024) 60.02 "7" "1440x900@60" QSize(1440, 900) 59.901 "8" "1280x800@60" QSize(1280, 800) 59.91 "9" "1152x864@75" QSize(1152, 864) 75 EDID Info: Device ID: "xrandr-Samsung Electric Company-S24B300-H4MD302024" Name: "S24B300" Vendor: "Samsung Electric Company" Serial: "H4MD302024" EISA ID: "" Hash: "eca6ca3c32c11a47a837d696a970b9d5" Width: 52 Height: 29 Gamma: 2.2 Red: QQuaternion(scalar:1, vector:(0.640625, 0.335938, 0)) Green: QQuaternion(scalar:1, vector:(0.31543, 0.628906, 0)) Blue: QQuaternion(scalar:1, vector:(0.15918, 0.0585938, 0)) White: QQuaternion(scalar:1, vector:(0.3125, 0.329102, 0))

Important also the section EDID Info, to see if the screen has a good and unique EDID, as invalid Edids, especially in combination with DisplayPort is a known source or problems.

Splitting a Python list into chunks is a common way of distributing the workload across multiple workers that can process them in parallel for faster results. Working with smaller pieces of data at a time may be the only way to fit a large dataset into computer memory. Sometimes, the very nature of the problem requires you to split the list into chunks.

In this tutorial, you’ll explore the range of options for splitting a Python list—or another iterable—into chunks. You’ll look at using Python’s standard modules and a few third-party libraries, as well as manually looping through the list and slicing it up with custom code. Along the way, you’ll learn how to handle edge cases and apply these techniques to multidimensional data by synthesizing chunks of an image in parallel.

In this tutorial, you’ll learn how to:

• Split a Python list into fixed-size chunks
• Split a Python list into a fixed number of chunks of roughly equal size
• Split finite lists as well as infinite data streams
• Perform the splitting in a greedy or lazy manner
• Produce lightweight slices without allocating memory for the chunks
• Split multidimensional data, such as an array of pixels

Throughout the tutorial, you’ll encounter a few technical terms, such as sequence, iterable, iterator, and generator. If these are new to you, then check out the linked resources before diving in. Additionally, familiarity with Python’s itertools module can be helpful in understanding some of the code snippets that you’ll find later.

To download the complete source code of the examples presented in this tutorial, click the link below:

Free Sample Code: Click here to download the free source code that you’ll use to split a Python list or iterable into chunks.

Split a Python List Into Fixed-Size Chunks

There are many real-world scenarios that involve splitting a long list of items into smaller pieces of equal size. The whole list may be too large to fit in your computer’s memory. Perhaps it’s more convenient or efficient to process the individual chunks separately rather than all at once. But there could be other reasons for splitting.

For example, when you search for something online, the results are usually presented to you in chunks, called pages, containing an equal number of items. This technique, known as content pagination, is common in web development because it helps improve the website’s performance by reducing the amount of data to transfer from the database at a time. It can also benefit the user by improving their browsing experience.

Most computer networks use packet switching to transfer data in packets or datagrams, which can be individually routed from the source to the destination address. This approach doesn’t require a dedicated physical connection between the two points, allowing the packets to bypass a damaged part of the network. The packets can be of variable length, but some low-level protocols require the data to be split into fixed-size packets.

Note: When splitting sequential data, you need to consider its size while keeping a few details in mind.

Specifically, if the total number of elements to split is an exact multiple of the desired chunk’s length, then you’ll end up with all the chunks having the same number of items. Otherwise, the last chunk will contain fewer items, and you may need extra padding to compensate for that.

Additionally, your data may have a known size up front when it’s loaded from a file in one go, or it can consist of an indefinite stream of bytes—while live streaming a teleconference, for example. Some solutions that you learn in this tutorial will only work when the number of elements is known before the splitting begins.

Most web frameworks, such as Django, will handle content pagination for you. Also, you don’t typically have to worry about some low-level network protocols. That being said, there are times when you’ll need to have more granular control and do the splitting yourself. In this section, you’ll take a look at how to split a list into smaller lists of equal size using different tools in Python.

Standard Library in Python 3.12: itertools.batched()

Using the standard library is almost always your best choice because it requires no external dependencies. The standard library provides concise, well-documented code that’s been tested by millions of users in production, making it less likely to contain bugs. Besides that, the standard library’s code is portable across different platforms and typically much more performant than a pure-Python equivalent, as most of it is implemented in C.

Unfortunately, the Python standard library hasn’t traditionally had built-in support for splitting iterable objects like Python lists. At the time of writing, Python 3.11 is the most recent version of the interpreter. But you can put yourself on the cutting edge by downloading a pre-release version of Python 3.12, which gives you access to the new itertools.batched(). Here’s an example demonstrating its use:

>>>>>> from itertools import batched >>> for batch in batched("ABCDEFGHIJ", 4): ... print(batch) ... ('A', 'B', 'C', 'D') ('E', 'F', 'G', 'H') ('I', 'J')

The function accepts any iterable object, such as a string, as its first argument. The chunk size is its second argument. Regardless of the input data type, the function always yields chunks or batches of elements as Python tuples, which you may need to convert to something else if you prefer working with a different sequence type. For example, you might want to join the characters in the resulting tuples to form strings again.

Note: The underlying implementation of itertools.batched() could’ve changed since the publishing of this tutorial, which was written against an alpha release of Python 3.12. For example, the function may now yield lists instead of tuples, so be sure to check the official documentation for the most up-to-date information.

Also, notice that the last chunk will be shorter than its predecessors unless the iterable’s length is divisible by the desired chunk size. To ensure that all the chunks have an equal length at all times, you can pad the last chunk with empty values, such as None, when necessary:

>>>>>> def batched_with_padding(iterable, batch_size, fill_value=None): ... for batch in batched(iterable, batch_size): ... yield batch + (fill_value,) * (batch_size - len(batch)) >>> for batch in batched_with_padding("ABCDEFGHIJ", 4): ... print(batch) ... ('A', 'B', 'C', 'D') ('E', 'F', 'G', 'H') ('I', 'J', None, None)

This adapted version of itertools.batched() takes an optional argument named fill_value, which defaults to None. If a chunk’s length happens to be less than size, then the function appends additional elements to that chunk’s end using fill_value as padding.

You can supply either a finite sequence of values to the batched() function or an infinite iterator yielding values without end:

>>>>>> from itertools import count >>> finite = batched([1, 2, 3, 4, 5, 6, 7, 8, 9, 10], 4) >>> infinite = batched(count(1), 4) >>> finite <itertools.batched object at 0x7f4e0e2ee830> >>> infinite <itertools.batched object at 0x7f4b4e5fbf10> >>> list(finite) [(1, 2, 3, 4), (5, 6, 7, 8), (9, 10)] >>> next(infinite) (1, 2, 3, 4) >>> next(infinite) (5, 6, 7, 8) >>> next(infinite) (9, 10, 11, 12)

In both cases, the function returns an iterator that consumes the input iterable using lazy evaluation by accumulating just enough elements to fill the next chunk. The finite iterator will eventually reach the end of the sequence and stop yielding chunks. Conversely, the infinite one will continue to produce chunks as long as you keep requesting them—for instance, by calling the built-in next() function on it.

Read the full article at https://realpython.com/how-to-split-a-python-list-into-chunks/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

JSON and YAML are the two most used file formats in software development. The YAML files are mainly used for configuration files whereas JSON files are used to store and transmit data. This article discusses how to convert YAML to JSON in Python.

Table of Contents

1. What is YAML?
2. What is JSON?
3. YAML String to JSON String in Python
4. YAML string to JSON file
5. Convert YAML file to JSON string
6. Convert YAML File to JSON File in Python
7. Conclusion

What is YAML?

YAML is a human-readable data serialization language used for data storage and data exchange formats. It is often used for configuration files. But, we can also use it for data exchange between systems.

YAML uses indentation to denote structure and nested elements, instead of using brackets or tags as in XML or JSON. This makes YAML files more readable and easier to edit than other data serialization formats. In YAML, a list of items is denoted by a dash (-) followed by a space, and key-value pairs are separated by a colon (:) followed by a space. YAML also supports comments, which can be added using the “#” symbol.

YAML supports various data types, including strings, integers, floating-point numbers, booleans, and arrays. It also supports nested structures, allowing for the creation of complex data structures. This makes YAML suitable for a wide range of use cases, from simple configurations to more complex data structures.

Following is a sample YAML file. It contains the details of an employee.

employee: name: John Doe age: 35 job: title: Software Engineer department: IT years_of_experience: 10 address: street: 123 Main St. city: San Francisco state: CA zip: 94102

In the above YAML file,

• The employee attribute has four nested attributes i.e. name, age, job, and address.
• The name and age attributes contain the name and age of the employee.
• The job attribute has three nested attributes i.e. title, department, and years_of_experience.
• Similarly, the address attribute has four inner attributes namely street, city, state, and zip.

Hence, you can observe that we can easily represent complex data with multiple levels of hierarchy using the YAML format.

What is JSON?

JSON (JavaScript Object Notation) is a lightweight data-interchange format that is used for exchanging data between systems. It is a text-based format that based on a subset of the JavaScript programming language. JSON is widely used for storing and exchanging data on the web, as well as for creating APIs (Application Programming Interfaces) that allow different systems to communicate with each other.

The basic structure of JSON data consists of key-value pairs, where keys are used to identify data elements, and values can be of various data types, including strings, numbers, booleans, arrays, and objects.

In JSON, data is enclosed in curly braces ({}) and separated by commas, while arrays are enclosed in square brackets ([]). JSON also supports comments, but they are not commonly used due to their limited support in various programming languages and applications.

We can represent the data shown in the previous section using JSON format as shown below.

{ "employee": { "name": "John Doe", "age": 35, "job": { "title": "Software Engineer", "department": "IT", "years_of_experience": 10 }, "address": { "street": "123 Main St.", "city": "San Francisco", "state": "CA", "zip": 94102 } } }

JSON data is lightweight and easy to parse, making it a popular choice for exchanging data on the web. It is also easily supported by many programming languages, including JavaScript, Python, Ruby, PHP, and Java, making it a versatile and universal format for exchanging data between different systems.

YAML String to JSON String in Python

To convert a YAML string to a JSON string, we will use the following steps.

• First, we will import the json and yaml modules using the import statement.
• Next, we will convert the YAML string to a python dictionary using the load() method defined in the yaml module. The load() method takes the yaml string as its first input argument and the loader type in its Loader argument. After execution, it returns a python dictionary.
• After obtaining the dictionary, we can convert it to a json string using the dumps() method defined in the json module. The dumps() method takes the python dictionary as its input argument and returns the json string.

You can observe this in the following example.

import json import yaml from yaml import SafeLoader yaml_string="""employee: name: John Doe age: 35 job: title: Software Engineer department: IT years_of_experience: 10 address: street: 123 Main St. city: San Francisco state: CA zip: 94102 """ print("The YAML string is:") print(yaml_string) python_dict=yaml.load(yaml_string, Loader=SafeLoader) json_string=json.dumps(python_dict) print("The JSON string is:") print(json_string)

Output:

The YAML string is: employee: name: John Doe age: 35 job: title: Software Engineer department: IT years_of_experience: 10 address: street: 123 Main St. city: San Francisco state: CA zip: 94102 The JSON string is: {"employee": {"name": "John Doe", "age": 35, "job": {"title": "Software Engineer", "department": "IT", "years_of_experience": 10}, "address": {"street": "123 Main St.", "city": "San Francisco", "state": "CA", "zip": 94102}}}

In this example, we have converted a YAML string to JSON string.

YAML string to JSON file

Instead of obtaining a json string, you can also convert a yaml string to a json file. For this, we will use the following steps.

• First, we will convert the yaml string to a python dictionary using the load() method defined in the yaml module.
• Next, we will open a json file in write mode using the open() function. The open() function takes the file name as its first input argument and the python literal “w” as its second input argument. After execution, it returns the file pointer.
• After opening the file, we will convert the python dictionary obtained from the yaml string into the json file. For this, we will use the dump() method defined in the json module. The dump() method takes the python dictionary as its first input argument and the file pointer returned by the open() function as the second input argument. 
• After execution of the dump() method, the json file will be saved on your machine. Then, we will close the json file using the close() method.

You can observe this in the following example.

import json import yaml from yaml import SafeLoader yaml_string="""employee: name: John Doe age: 35 job: title: Software Engineer department: IT years_of_experience: 10 address: street: 123 Main St. city: San Francisco state: CA zip: 94102 """ print("The YAML string is:") print(yaml_string) python_dict=yaml.load(yaml_string, Loader=SafeLoader) file=open("person_details.json","w") json.dump(python_dict,file) file.close() print("JSON file saved")

Output:

The YAML string is: employee: name: John Doe age: 35 job: title: Software Engineer department: IT years_of_experience: 10 address: street: 123 Main St. city: San Francisco state: CA zip: 94102 JSON file saved

In this example, we have converted the YAML string to a JSON file using json and yaml modules in python. The output JSON file looks as follows.

Output JSON file Convert YAML file to JSON string

We can also convert a  yaml file to a json string. For this, we will use the following steps.

• First, we will open the yaml file in read mode using the open() function. The open() function takes the file name of the yaml file as its first input argument and the python literal “r” as its second argument. After execution, it returns a file pointer. 
• Next, we will obtain the python dictionary from the yaml file using the load() method.  The load() method takes the file pointer as its first input argument and the loader type in its Loader argument. After execution, it returns a dictionary.
• After obtaining the dictionary, we will convert it to JSON string using the dumps() method defined in the json module.  The dumps() method takes the python dictionary as its input argument and returns the json string.

You can observe this in the following example.

import json import yaml from yaml import SafeLoader yaml_file=open("person_details.yaml","r") python_dict=yaml.load(yaml_string, Loader=SafeLoader) json_string=json.dumps(python_dict) print("The JSON string is:") print(json_string)

Output:

The JSON string is: {"employee": {"name": "John Doe", "age": 35, "job": {"title": "Software Engineer", "department": "IT", "years_of_experience": 10}, "address": {"street": "123 Main St.", "city": "San Francisco", "state": "CA", "zip": 94102}}}

In this example, we have converted a YAML file to JSON string.

Convert YAML File to JSON File in Python

Instead of converting it into the JSON string, we can also convert the YAML file into a JSON file. For this, we will use the following steps.

• First, we will obtain the python dictionary from the yaml file using the open() function and the load() method.
• Then, we will open a json file in write mode using the open() function. The open() function takes the file name as its first input argument and the literal “w” as its second input argument. After execution, it returns the file pointer.
• After opening the file, we will convert the python dictionary obtained from the yaml file into the json file. For this, we will use the dump() method defined in the json module. The dump() method takes the python dictionary as its first input argument and the file pointer returned by the open() function as its second input argument. 
• After execution of the dump() method, the json file will be saved on your machine. Then, we will close the json file using the close() method.

You can observe the entire process in the following example.

import json import yaml from yaml import SafeLoader yaml_file=open("person_details.yaml","r") python_dict=yaml.load(yaml_string, Loader=SafeLoader) file=open("person_details1.json","w") json.dump(python_dict,file) file.close() print("JSON file saved")

After execution of the above code, the YAML file person_details.yaml is saved as JSON into the file person_details1.json.

Conclusion

In this article, we have discussed how to convert a yaml string or file to json format. 

To learn more about python programming, you can read this article on how to convert JSON to YAML in Python. You might also like this article on custom json encoders in python.

I hope you enjoyed reading this article. Stay tuned for more informative articles.

Happy Learning!

The post Convert YAML to JSON in Python appeared first on PythonForBeginners.com.

Hi everyone,

I am happy to report that after solving some last-time problems we have a bunch of fresh releases for you!

Python 3.12.0 alpha 5

Check the new alpha of 3.12 with some Star Trek vibes:

https://www.python.org/downloads/release/python-3120a5/

210 new commits since 3.12.0a4 last month

Python 3.11.2

A shipment of bugfixes and security releases for the newest Python!

https://www.python.org/downloads/release/python-3112/

194 new commits since 3.11.1

Python 3.10.10

Your trusty Python3.10 just got more stable and secure!

https://www.python.org/downloads/release/python-31010/

131 new commits since 3.10.9

We hope you enjoy the new releases!

Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.

https://www.python.org/psf/ 

Your friendly release team,

Ned Deily @nad
Steve Dower @steve.dower
Pablo Galindo Salgado @pablogsal
Łukasz Langa @ambv
Thomas Wouters @thomas

I have installed GNU/Linux on many a computers in ~20 years (some automated, most individually). In the University, I used to be woken past midnight by someone knocking at the door — who reinstalled Windows — and now they can’t boot because grub was overwritten. I’d rub the eyes, pickup the bunch latest Fedora CDs and go rescue the beast machine. Linux installation, customization and grub-recovery was my specialization (no, the course didn’t have credit for that).

Technologies (libre & otherwise) have improved since then. Instead of MBR, there’s GPT (no, not that one). Instead of BIOS, there’s UEFI. Dual booting Windows with GNU/Linux has become mostly painless. Then there’s Secure Boot. Libre software works with that too. You may still run into issues; I ran into one recently and if someone is in the same position I hope this helps:

A friend of mine got an Ideapad 3 Gaming laptop and we tried to install Fedora 37 on it (of course, remotely; thanks to screensharing and cameras on mobile phones). The bootable USB pendrive was not being listed in boot options (F12), so we fiddled with TPM & Secure Boot settings in EFI settings (F2). No luck, and troubleshooting eventually concluded that the USB pendrive was faulty. Tried with another one, and this time it was detected, happily installed Fedora 37 (under 15 mins, because instead of spinning Hard Disks, there’s SSD). Fedora boots & works fine.

A day later, the friend selects Windows to boot into (from grub menu) and gets greeted by a BitLocker message: “Enter bitlocker recovery key” because “Secure boot is disabled”.

Dang. I thought we re-enabled Secure Boot, but apparently not. Go to EFI settings, and turn it back on; save & reboot; select Windows — but BitLocker kept asking for recovery key but with a different reason: “Secure Boot policy has unexpectedly changed”.

That led to scrambling & searching, as BitLocker was not enabled by the user but OEM, and thus there was no recovery key in the user’s Microsoft online account (if the user had enabled it manually, they can find the key there).

The nature of the error message made me conclude that Fedora installation with secure boot disabled has somehow altered the TPM settings and Windows (rightfully) refuses to boot. EFI settings has an option to ‘Restore Factory Keys’ which will reset the secure boot DB. I could try that to remove Fedora keys, pray Windows boots and if it works, recover grub (my specialty) or reinstall Fedora in the worst case scenario.

Enter Matthew Garret. Matthew was instrumental in making GNU/Linux systems to work with Secure Boot (and was awarded the prestigious Free Software Foundation Award). He is a security researcher who frequently writes about computer security.

I have sought Matthew’s advice before trying anything stupid, and he suggested thus (reproduced with permission):

First, how are you attempting to boot Windows? If you’re
doing this via grub then this will result in the secure boot
measurements changing and this error occurring – if you pick Windows
from the firmware boot menu (which I think should appear if you hit F12
on an Ideapad?) then this might solve the problem.

Secondly, if the owner added a Microsoft account when setting up the
Windows system, they can visit
https://account.microsoft.com/devices/recoverykey and a recovery key
should be available there.

If neither of these approaches work, then please try resetting the
factory keys, reset the firmware to its default settings, and delete any
Fedora boot entries from the firmware (you can recover them later), and
with luck that’ll work.

Thankfully, the first option of booting Windows directly via F12 — without involving grub — works. And the first thing the user does after logging in is back up the recovery keys.

Sooner or later in your data science journey, you’ll hit a point where you need to get data from a database. However, making the leap from reading a locally-stored CSV file into pandas to connecting to and querying databases can be a daunting task. In the first of a series of blog posts, we’ll explore how to read data stored in a MySQL database into pandas, and look at some nice PyCharm features that make this task easier.

Viewing the database contents

In this tutorial, we’re going to read some data about airline delays and cancellations from a MySQL database into a pandas DataFrame. This data is a version of the “Airline Delays from 2003-2016” dataset by Priank Ravichandar licensed under CC0 1.0.

One of the first things that can be frustrating about working with databases is not having an overview of the available data, as all of the tables are stored on a remote server. Therefore, the first PyCharm feature we’re going to use is the Database tool window, which allows you to connect to and fully introspect a database before doing any queries.

To connect to our MySQL database, we’re first going to navigate over to the right-hand side of PyCharm and click the Database tool window.

On the top left of this window, you’ll see a plus button. Clicking on this gives us the following dropdown dialog window, from which we’ll select Data Source | MySQL.

We now have a popup window which will allow us to connect to our MySQL database. In this case, we’re using a locally hosted database, so we leave Host as “localhost” and Port as the default MySQL port of “3306”. We’ll use the “User & Password” Authentication option, and enter “pycharm” for both the User and Password. Finally, we enter our Database name of “demo”. Of course, in order to connect to your own MySQL database you’ll need the specific host, database name, and your username and password. See the documentation for the full set of options.

Next, click Test Connection. PyCharm lets us know that we don’t have the driver files installed. Go ahead and click Download Driver Files. One of the very nice features of the Database tool window is that it automatically finds and installs the correct drivers for us.

Success! We’ve connected to our database. We can now navigate to the Schemas tab and select which schemas we want to introspect. In our example database we only have one (“demo”), but in cases where you have very large databases, you can save yourself time by only introspecting relevant ones.

With all of that done, we’re ready to connect to our database. Click OK, and wait a few seconds. You can now see that our entire database has been introspected, down to the level of table fields and their types. This gives us a great overview of what is in the database before running a single query.

Reading in the data using MySQL Connector

Now that we know what is in our database, we are ready to put together a query. Let’s say we want to see the airports that had at least 500 delays in 2016. From looking at the fields in the introspected airlines table, we see that we can get that data with the following query:

SELECT AirportCode, SUM(FlightsDelayed) AS TotalDelayed FROM airlines WHERE TimeYear = 2016 GROUP BY AirportCode HAVING SUM(FlightsDelayed) > 500;

The first way we can run this query using Python is using a package called MySQL Connector, which can be installed from either PyPI or Anaconda. See the linked documentation if you need guidance on setting up pip or conda environments or installing dependencies. Once installation is finished, we’ll open a new Jupyter notebook and import both MySQL Connector and pandas.

import mysql.connector import pandas as pd

In order to read data from our database, we need to create a connector. This is done using the connect method, to which we pass the credentials needed to access the database: the host, the database name, the user, and the password. These are the same credentials we used to access the database using the Database tool window in the previous section.

mysql_db_connector = mysql.connector.connect( host="localhost", database="demo", user="pycharm", password="pycharm" )

We now need to create a cursor. This will be used to execute our SQL queries against the database, and it uses the credentials sorted in our connector to get access.

mysql_db_cursor = mysql_db_connector.cursor()

We’re now ready to execute our query. We do this using the execute method from our cursor and passing the query as the argument.

delays_query = """ SELECT AirportCode, SUM(FlightsDelayed) AS TotalDelayed FROM airlines WHERE TimeYear = 2016 GROUP BY AirportCode HAVING SUM(FlightsDelayed) > 500; """ mysql_db_cursor.execute(delays_query)

We then retrieve the result using the cursor’s fetchall method.

mysql_delays_list = mysql_db_cursor.fetchall()

However, we have a problem at this point: fetchall returns the data as a list. To get it into pandas, we can pass it into a DataFrame, but we’ll lose our column names and will need to manually specify them when we want to create the DataFrame.

Luckily, pandas offers a better way. Rather than creating a cursor, we can read our query into a DataFrame in one step, using the read_sql method.

mysql_delays_df2 = pd.read_sql(delays_query, con=mysql_db_connector)

We simply need to pass our query and connector as arguments in order to read the data from the MySQL database. Looking at our dataframe, we can see that we have the exact same results as above, but this time our column names have been preserved.

A nice feature you might have noticed is that PyCharm applies syntax highlighting to the SQL query, even when it’s contained inside a Python string. We’ll cover another way that PyCharm allows you to work with SQL later in this blog post.

Reading in the data using SQLAlchemy

An alternative to using MySQL Connector is using a package called SQLAlchemy. This package offers a one-stop method for connecting to a range of different databases, including MySQL. One of the nice things about using SQLAlchemy is that the syntax for querying different database types remains consistent across database types, saving you from remembering a bunch of different commands if you’re working with a lot of different databases.

To get started, we need to install SQLAlchemy either from PyPI or Anaconda. We then import the create_engine method, and of course, pandas.

import pandas as pd from sqlalchemy import create_engine

We now need to create our engine. The engine allows us to tell pandas which SQL dialect we’re using (in our case, MySQL) and provide it with the credentials it needs to access our database. This is all passed as one string, in the form of [dialect]://[user]:[password]@[host]/[database]. Let’s see what this looks like for our MySQL database:

mysql_engine = create_engine("mysql+mysqlconnector://pycharm:pycharm@localhost/demo")

With this created, we simply need to use read_sql again, this time passing the engine to the con argument:

mysql_delays_df3 = pd.read_sql(delays_query, con=mysql_engine)

As you can see, we get the same result as when using read_sql with MySQL Connector.

Advanced options for working with databases

Now these connector methods are very nice for extracting a query that we already know we want, but what if we want to get a preview of what our data will look like before running the full query, or an idea of how long the whole query will take? PyCharm is here again with some advanced features for working with databases.

If we navigate back over to the Database tool window and right-click on our database, we can see that under New we have the option to create a Query Console.

This allows us to open a console which we can use to query against the database in native SQL. The console window includes SQL code completion and introspection, giving you an easier way to create your queries prior to passing them to the connector packages in Python.

Highlight your query and click the Execute button in the top left corner.

This will retrieve the results of our query in the Services tab, where it can be inspected or exported. One nice thing about running queries against the console is that only the first 500 rows are initially retrieved from the database, meaning you can get a sense of the results of larger queries without committing to pulling all of the data. You can adjust the number of rows retrieved by going to Settings/Preferences | Tools | Database | Data Editor and Viewer and changing the value under Limit page size to:.

Speaking of large queries, we can also get a sense of how long our query will take by generating an execution plan. If we highlight our query again and then right-click, we can select Explain Plan | Explain Analyse from the menu. This will generate an execution plan for our query, showing each step that the query planner is taking to retrieve our results. Execution plans are their own topic, and we don’t really need to understand everything our plan is telling us. Most relevant for our purposes is the Actual Total Time column, where we can see how long it will take to return all of the rows at each step. This gives us a good estimate of the overall query time, as well as whether any parts of our query are likely to be particularly time consuming.

You can also visualize the execution by clicking on the Show Visualization button to the left of the Plan panel.

This will bring up a flowchart that makes it a bit easier to navigate through the steps that the query planner is taking.

Getting data from MySQL databases into pandas DataFrames is straightforward, and PyCharm has a number of powerful tools to make working with MySQL databases easier. In the next blog post, we’ll look at how to use PyCharm to read data into pandas from another popular database type, PostgreSQL databases.

The PSF is pleased to announce its fourth batch of PSF Fellows for 2022! Let us welcome the new PSF Fellows for Q4! The following people continue to do amazing things for the Python community:

Chandan Kumar Twitter, GitHub, MastodonDanny AdairWebsiteHugo van Kemenade GitHub, MastodonJosef HeinenTwitter, GitHub, WebsiteNicolas Laurance
LinkedInSayan ChowdhuryTwitter, GitHub, WebsiteSoong Chee GiTwitter, GitHub, WebsiteYung-Yu ChenLinkedIn, Twitter, Website 

Thank you for your continued contributions. We have added you to our Fellow roster online.

The above members help support the Python ecosystem by being phenomenal leaders, sustaining the growth of the Python scientific community, maintaining virtual Python communities, maintaining Python libraries, creating educational material, organizing Python events and conferences, starting Python communities in local regions, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.

Let's continue recognizing Pythonistas all over the world for their impact on our community. The criteria for Fellow members is available online: https://www.python.org/psf/fellows/. If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. We are accepting nominations for quarter 1 through February 20, 2023.

Are you a PSF Fellow and want to help the Work Group review nominations? Contact us at psf-fellow at python.org.

When you think about processing tabular data in Python, what library comes to mind? Pandas, I'd guess. But there are other libraries out there and Polars is one of the more exciting new ones. It's built in Rust, embraces parallelism, and can be 10-20x faster than Pandas out of the box. <br/> <br/> We have Polars' creator, Ritchie Vink here to give us a look at this exciting new data frame library.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Ritchie on Mastodon</b>: <a href="https://fosstodon.org/@ritchie46" target="_blank" rel="noopener">@ritchie46@fosstodon.org</a><br/> <b>Ritchie on Twitter</b>: <a href="https://twitter.com/RitchieVink" target="_blank" rel="noopener">@RitchieVink</a><br/> <b>Ritchie's website</b>: <a href="https://www.ritchievink.com" target="_blank" rel="noopener">ritchievink.com</a><br/> <br/> <b>Polars</b>: <a href="https://www.pola.rs" target="_blank" rel="noopener">pola.rs</a><br/> <b>Apache Arrow</b>: <a href="https://arrow.apache.org" target="_blank" rel="noopener">arrow.apache.org</a><br/> <b>Polars Benchmarks</b>: <a href="https://www.pola.rs/benchmarks.html" target="_blank" rel="noopener">pola.rs</a><br/> <b>Coming from Pandas Guide</b>: <a href="https://pola-rs.github.io/polars-book/user-guide/coming_from_pandas.html" target="_blank" rel="noopener">github.io</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=KdCcqWAsuRs" target="_blank" rel="noopener">youtube.com</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div><br/> <strong>Sponsors</strong><br/> <a href='https://talkpython.fm/taipy'>Taipy</a><br> <a href='https://talkpython.fm/userinterviews'>User Interviews</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>

Finally Drupal has a new default admin theme — Claro. Read our post to learn how the Drupal community made its way toward the update and what theme features let this CMS progress to a higher level.

The Tryton Foundation is happy to announce the next Tryton Unconference in Berlin on May 22nd - 24th.

The first day will be mainly dedicated to presentation at Change Hub.

The second and third day will be dedicated to code sprint at inmedio Berlin.

More information and registration at Tryton - Unconference Berlin - May 22nd-24th, 2023

Many thanks to m-ds to organize the event.

2 posts - 2 participants

Read full topic

As was announced in the February newsletter we’ve migrated our development to heptapod. This means that you no longer need a Google account in order to contribute to Tryton and none of our tools are dependent on them any more. It took us 11 years to reach this point!

Now, one month on from the migration, we have more than 20 members contributing to the Tryton project, who have created more than 200 Merge Requests. These are good numbers for the project and I’m sure they will keep increasing in the future.

Such a migration was only possible with the help of many people, some of which I would like to thank publicly now.

The migration was fully sponsored by Jonathan Levy (@jonl) who contributed all the funds required to create the migration scripts. Jon is an entrepreneur who has been working with Tryton since 2012. Mr. Levy says:

Tryton is truly an undersung gem in the open-source software world. It is beautifully structured, flexible, and reliable, and I continue to be impressed by its core community. I would recommend it, either as an off-the-shelf ERP, or to anyone needing to encode custom business logic for their enterprise. I hope the recent Heptapod migration, which updates Tryton’s old contribution workflow, will help Tryton flourish in the years to come.

Thanks Jon for your contribution and for your wonderful endorsement and best wishes for our project

This migration also required lots of work from other people:

• The Octobus team, who created the scripts to import our repository and all our bug tracker history.
• The B2CK guys (@nicoe and @ced) who coordinated the work with Octobus and ensured everything went smoothly.
• Last but not least, @htgoebel who helped with the CI configuration.

This is a clear proof that a good team and hard work can achieve amazing results and that there is no limit if we share the work between us, following in the spirit of open source.

I cannot end this without giving big thanks to everyone who helped us finish this important task. Please also express your gratitude to them with some likes on this topic.

1 post - 1 participant

Read full topic

tips to improve coding