Planet Python

My self-nomination statement for the 2025 Django Software Foundation (DSF) board of directors elections

I talked about the new Python 3.13 REPL a few months ago and after 3.13 was released. I think it’s awesome.

I’d like to share a secret feature within the Python 3.13 REPL which I’ve been finding useful recently: adding custom keyboard shortcuts.

This feature involves a PYTHONSTARTUP file, use of an unsupported Python module, and dynamically evaluating code.

In short, we may be getting ourselves into trouble. But the result is very neat!

Thanks to Łukasz Llanga for inspiring this post via his excellent EuroPython keynote talk.

The goal: keyboard shortcuts in the REPL

First, I’d like to explain the end result.

Let’s say I’m in the Python REPL on my machine and I’ve typed numbers =:

1 >>> numbers =

I can now hit Ctrl-N to enter a list of numbers I often use while teaching (Lucas numbers):

1 numbers = [2, 1, 3, 4, 7, 11, 18, 29]

That saved me some typing!

Getting a prototype working

First, let’s try out an example command.

Copy-paste this into your Python 3.13 REPL:

1 2 3 4 5 6 7 8 9 10 11 from _pyrepl.simple_interact import _get_reader from _pyrepl.commands import Command class Lucas(Command): def do(self): self.reader.insert("[2, 1, 3, 4, 7, 11, 18, 29]") reader = _get_reader() reader.commands["lucas"] = Lucas reader.bind(r"\C-n", "lucas")

Now hit Ctrl-N.

If all worked as planned, you should see that list of numbers entered into the REPL.

Cool! Now let’s generalize this trick and make Python run our code whenever it starts.

But first… a disclaimer.

Here be dragons 🐉

Notice that _ prefix in the _pyrepl module that we’re importing from? That means this module is officially unsupported.

The _pyrepl module is an implementation detail and its implementation may change at any time in future Python versions.

In other words: _pyrepl is designed to be used by Python’s standard library modules and not anyone else. That means that we should assume this code will break in a future Python version.

Will that stop us from playing with this module for the fun of it?

It won’t.

Creating a PYTHONSTARTUP file

So we’ve made one custom key combination for ourselves. How can we setup this command automatically whenever the Python REPL starts?

We need a PYTHONSTARTUP file.

When Python launches, if it sees a PYTHONSTARTUP environment variable it will treat that environment variable as a Python file to run on startup.

I’ve made a /home/trey/.python_startup.py file and I’ve set this environment variable in my shell’s configuration file (~/.zshrc):

1 export PYTHONSTARTUP=$HOME/.python_startup.py

To start, we could put our single custom command in this file:

1 2 3 4 5 6 7 8 9 10 11 12 13 try: from _pyrepl.simple_interact import _get_reader from _pyrepl.commands import Command except ImportError: pass # Not in the new pyrepl OR _pyrepl implementation changed else: class Lucas(Command): def do(self): self.reader.insert("[2, 1, 3, 4, 7, 11, 18, 29]") reader = _get_reader() reader.commands["lucas"] = Lucas reader.bind(r"\C-n", "lucas")

Note that I’ve stuck our code in a try-except block. Our code only runs if those _pyrepl imports succeed.

Note that this might still raise an exception when Python starts if the reader object’s command attribute or bind method change in a way that breaks our code.

Personally, I’d like to see those breaking changes occur print out a traceback the next time I upgrade Python. So I’m going to leave those last few lines without their own catch-all exception handler.

Generalizing the code

Here’s a PYTHONSTARTUP file with a more generalized solution:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 try: from _pyrepl.simple_interact import _get_reader from _pyrepl.commands import Command except ImportError: pass else: # Hack the new Python 3.13 REPL! cmds = { r"\C-n": "[2, 1, 3, 4, 7, 11, 18, 29]", r"\C-f": '["apples", "oranges", "bananas", "strawberries", "pears"]', } from textwrap import dedent reader = _get_reader() for n, (key, text) in enumerate(cmds.items(), start=1): name = f"CustomCommand{n}" exec(dedent(f""" class _cmds: class {name}(Command): def do(self): self.reader.insert({text!r}) reader.commands[{name!r}] = {name} reader.bind({key!r}, {name!r}) """)) # Clean up all the new variables del _get_reader, Command, dedent, reader, cmds, text, key, name, _cmds, n

This version uses a dictionary to map keyboard shortcuts to the text they should insert.

Note that we’re repeatedly building up a string of Command subclasses for each shortcut, using exec to execute the code for that custom Command subclass, and then binding the keyboard shortcut to that new command class.

At the end we then delete all the variables we’ve made so our REPL will start the clean global environment we normally expect it to have:

1 2 3 4 Python 3.13.0 (main, Oct 8 2024, 10:37:56) [GCC 11.4.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> dir() ['__annotations__', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__']

Is this messy?

Yes.

Is that a needless use of a dictionary that could have been a list of 2-item tuples instead?

Yes.

Does this work?

Yes.

Doing more interesting and risky stuff

Note that there are many keyboard shortcuts that may cause weird behaviors if you bind them.

For example, if you bind Ctrl-i, your binding may trigger every time you try to indent. And if you try to bind Ctrl-m, your binding may be ignored because this is equivalent to hitting the Enter key.

So be sure to test your REPL carefully after each new binding you try to invent.

If you want to do something more interesting, you could poke around in the _pyrepl package to see what existing code you can use/abuse.

For example, here’s a very hacky way of making a binding to Ctrl-x followed by Ctrl-r to make this import subprocess, type in a subprocess.run line, and move your cursor between the empty string within the run call:

1 2 3 4 5 6 7 8 9 10 11 12 class _cmds: class Run(Command): def do(self): from _pyrepl.commands import backward_kill_word, left backward_kill_word(self.reader, self.event_name, self.event).do() self.reader.insert("import subprocess\n") code = 'subprocess.run("", shell=True)' self.reader.insert(code) for _ in range(len(code) - code.index('""') - 1): left(self.reader, self.event_name, self.event).do() reader.commands["subprocess_run"] = _cmds.Run reader.bind(r"\C-x\C-r", "subprocess_run") What keyboard shortcuts are available?

As you play with customizing keyboard shortcuts, you’ll likely notice that many key combinations result in strange and undesirable behavior when overridden.

For example, overriding Ctrl-J will also override the Enter key… at least it does in my terminal.

I’ll list the key combinations that seem unproblematic on my setup with Gnome Terminal in Ubuntu Linux.

Here are Control key shortcuts that seem to be complete unused in the Python REPL:

• Ctrl-N
• Ctrl-O
• Ctrl-P
• Ctrl-Q
• Ctrl-S
• Ctrl-V

Note that overriding Ctrl-H is often an alternative to the backspace key

Here are Alt/Meta key shortcuts that appear unused on my machine:

• Alt-A
• Alt-E
• Alt-G
• Alt-H
• Alt-I
• Alt-J
• Alt-K
• Alt-M
• Alt-N
• Alt-O
• Alt-P
• Alt-Q
• Alt-S
• Alt-V
• Alt-W
• Alt-X
• Alt-Z

You can add an Alt shortcut by using \M (for “meta”). So r"\M-a" would capture Alt-A just as r"\C-a" would capture Ctrl-A.

Here are keyboard shortcuts that can be customized but you might want to consider whether the current default behavior is worth losing:

• Alt-B: backward word (same as Ctrl-Left)
• Alt-C: capitalize word (does nothing on my machine…)
• Alt-D: kill word (delete to end of word)
• Alt-F: forward word (same as Ctrl-Right)
• Alt-L: downcase word (does nothing on my machine…)
• Alt-U: upcase word (does nothing on my machine…)
• Alt-Y: yank pop
• Ctrl-A: beginning of line (like the Home key)
• Ctrl-B: left (like the Left key)
• Ctrl-E: end of line (like the End key)
• Ctrl-F: right (like the Right key)
• Ctrl-G: cancel
• Ctrl-H: backspace (same as the Backspace key)
• Ctrl-K: kill line (delete to end of line)
• Ctrl-T: transpose characters
• Ctrl-U: line discard (delete to beginning of line)
• Ctrl-W: word discard (delete to beginning of word)
• Ctrl-Y: yank
• Alt-R: restore history (within history mode)

What fun have you found in _pyrepl?

Find something fun while playing with the _pyrepl package’s inner-workings?

I’d love to hear about it! Comment below to share what you found.

Web scraping is the automated process of extracting data from the internet. The Python libraries Requests and Beautiful Soup are powerful tools for the job. To effectively harvest the vast amount of data available online for your research, projects, or personal interests, you’ll need to become skilled at web scraping.

In this tutorial, you’ll learn how to:

• Inspect the HTML structure of your target site with your browser’s developer tools
• Decipher data encoded in URLs
• Use Requests and Beautiful Soup for scraping and parsing data from the internet
• Step through a web scraping pipeline from start to finish
• Build a script that fetches job offers from websites and displays relevant information in your console

If you like learning with hands-on examples and have a basic understanding of Python and HTML, then this tutorial is for you! Working through this project will give you the knowledge and tools you need to scrape any static website out there on the World Wide Web. You can download the project source code by clicking on the link below:

Get Your Code: Click here to download the free sample code that you’ll use to learn about web scraping in Python.

Take the Quiz: Test your knowledge with our interactive “Beautiful Soup: Build a Web Scraper With Python” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Beautiful Soup: Build a Web Scraper With Python

In this quiz, you'll test your understanding of web scraping using Python. By working through this quiz, you'll revisit how to inspect the HTML structure of a target site, decipher data encoded in URLs, and use Requests and Beautiful Soup for scraping and parsing data from the Web.

What Is Web Scraping?

Web scraping is the process of gathering information from the internet. Even copying and pasting the lyrics of your favorite song can be considered a form of web scraping! However, the term “web scraping” usually refers to a process that involves automation. While some websites don’t like it when automatic scrapers gather their data, which can lead to legal issues, others don’t mind it.

If you’re scraping a page respectfully for educational purposes, then you’re unlikely to have any problems. Still, it’s a good idea to do some research on your own to make sure you’re not violating any Terms of Service before you start a large-scale web scraping project.

Reasons for Automated Web Scraping

Say that you like to surf—both in the ocean and online—and you’re looking for employment. It’s clear that you’re not interested in just any job. With a surfer’s mindset, you’re waiting for the perfect opportunity to roll your way!

You know about a job site that offers precisely the kinds of jobs you want. Unfortunately, a new position only pops up once in a blue moon, and the site doesn’t provide an email notification service. You consider checking up on it every day, but that doesn’t sound like the most fun and productive way to spend your time. You’d rather be outside surfing real-life waves!

Thankfully, Python offers a way to apply your surfer’s mindset. Instead of having to check the job site every day, you can use Python to help automate the repetitive parts of your job search. With automated web scraping, you can write the code once, and it’ll get the information that you need many times and from many pages.

Note: In contrast, when you try to get information manually, you might spend a lot of time clicking, scrolling, and searching, especially if you need large amounts of data from websites that are regularly updated with new content. Manual web scraping can take a lot of time and be highly repetitive and error-prone.

There’s so much information on the internet, with new information constantly being added. You’ll probably be interested in some of that data, and much of it is out there for the taking. Whether you’re actually on the job hunt or just want to automatically download all the lyrics of your favorite artist, automated web scraping can help you accomplish your goals.

Challenges of Web Scraping

The internet has grown organically out of many sources. It combines many different technologies, styles, and personalities, and it continues to grow every day. In other words, the internet is a hot mess! Because of this, you’ll run into some challenges when scraping the web:

• Variety: Every website is different. While you’ll encounter general structures that repeat themselves, each website is unique and will need personal treatment if you want to extract the relevant information.

• Durability: Websites constantly change. Say you’ve built a shiny new web scraper that automatically cherry-picks what you want from your resource of interest. The first time you run your script, it works flawlessly. But when you run the same script a while later, you run into a discouraging and lengthy stack of tracebacks!

Unstable scripts are a realistic scenario because many websites are in active development. If a site’s structure changes, then your scraper might not be able to navigate the sitemap correctly or find the relevant information. The good news is that changes to websites are often small and incremental, so you’ll likely be able to update your scraper with minimal adjustments.

Still, keep in mind that the internet is dynamic and keeps on changing. Therefore, the scrapers you build will probably require maintenance. You can set up continuous integration to run scraping tests periodically to ensure that your main script doesn’t break without your knowledge.

An Alternative to Web Scraping: APIs

Some website providers offer application programming interfaces (APIs) that allow you to access their data in a predefined manner. With APIs, you can avoid parsing HTML. Instead, you can access the data directly using formats like JSON and XML. HTML is primarily a way to visually present content to users.

When you use an API, the data collection process is generally more stable than it is through web scraping. That’s because developers create APIs to be consumed by programs rather than by human eyes.

The front-end presentation of a site might change often, but a change in the website’s design doesn’t affect its API structure. The structure of an API is usually more permanent, which means it’s a more reliable source of the site’s data.

However, APIs can change as well. The challenges of both variety and durability apply to APIs just as they do to websites. Additionally, it’s much harder to inspect the structure of an API by yourself if the provided documentation lacks quality.

Read the full article at https://realpython.com/beautiful-soup-web-scraper-python/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of web scraping with Python, Requests, and Beautiful Soup.

By working through this quiz, you’ll revisit how to inspect the HTML structure of your target site with your browser’s developer tools, decipher data encoded in URLs, use Requests and Beautiful Soup for scraping and parsing data from the Web, and gain an understanding of what a web scraping pipeline looks like.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

<strong>Topics covered in this episode:</strong><br> <ul> <li><strong><a href="https://pythoninsider.blogspot.com/2024/10/python-3140-alpha-1-is-now-available.html?featured_on=pythonbytes">Python 3.14.0 alpha 1 is now available</a></strong></li> <li><a href="https://github.com/astral-sh/uv/pull/8272?featured_on=pythonbytes"><strong>uv supports dependency groups</strong></a></li> <li><strong><a href="https://github.com/wagoodman/dive?featured_on=pythonbytes">dive: A tool for exploring each layer in a docker image</a></strong></li> <li><a href="https://pypi.org/project/pytest-metadata/?featured_on=pythonbytes"><strong>pytest-metadata</strong></a></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=70OO7BMV1KE' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="407">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/?featured_on=pythonbytes"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://courses.pythontest.com/p/the-complete-pytest-course?featured_on=pythonbytes"><strong>The Complete pytest Course</strong></a> &amp; <a href="https://courses.pythontest.com/hello-pytest?featured_on=pythonbytes"><strong>Hello, pytest!</strong></a></li> <li><a href="https://www.patreon.com/pythonbytes"><strong>Patreon Supporters</strong></a></li> </ul> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually <strong>Monday</strong> at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</p> <p><strong>Michael #1:</strong> <a href="https://pythoninsider.blogspot.com/2024/10/python-3140-alpha-1-is-now-available.html?featured_on=pythonbytes">Python 3.14.0 alpha 1 is now available</a></p> <ul> <li>First of seven planned alpha releases.</li> <li>Many new features for Python 3.14 are still being planned and written. Among the new major new features and changes so far: <ul> <li><a href="https://peps.python.org/pep-0649/?featured_on=pythonbytes">PEP 649</a>: <a href="https://docs.python.org/3.14/whatsnew/3.14.html#pep-649-deferred-evaluation-of-annotations">deferred evaluation of annotations</a></li> <li><a href="https://docs.python.org/3.14/whatsnew/3.14.html#improved-error-messages">Improved error messages</a></li> </ul></li> </ul> <p><strong>Brian #2:</strong> <a href="https://github.com/astral-sh/uv/pull/8272?featured_on=pythonbytes"><strong>uv supports dependency groups</strong></a></p> <ul> <li><a href="https://pythonbytes.fm/episodes/show/406/whats-on-django-tv-tonight">we covered dependency groups in episode 406</a></li> <li>as of <a href="https://github.com/astral-sh/uv/blob/main/CHANGELOG.md?featured_on=pythonbytes">0.4.27</a>, uv supports dependency groups</li> <li>docs show <a href="https://docs.astral.sh/uv/concepts/dependencies/?featured_on=pythonbytes">how to add dependencies</a> with uv add --group <ul> <li>also “The --dev, --only-dev, and --no-dev flags are equivalent to --group dev, --only-group dev, and --no-group dev respectively.”</li> </ul></li> <li>To install a group, uv pip install --group doesn’t work yet. <ul> <li>It’s waiting for PyPA to decide on an interface for pip, and uv pip will use that interface.</li> </ul></li> <li>But sync works. <pre><code>$ uv init # create a pyproject.toml $ uv add --group foo pytest $ uv venv # create venv $ uv sync --group foo # will install all dependencies, including group "foo" </code></pre></li> </ul> <p><strong>Michael #3:</strong> <a href="https://github.com/wagoodman/dive?featured_on=pythonbytes">dive: A tool for exploring each layer in a docker image</a></p> <ul> <li>via Mike Fiedler</li> <li>Features: <ul> <li>Show Docker image contents broken down by layer</li> <li>Indicate what's changed in each layer</li> <li>Estimate "image efficiency"</li> <li>Quick build/analysis cycles</li> <li>CI Integration</li> </ul></li> </ul> <p><strong>Brian #4:</strong> <a href="https://pypi.org/project/pytest-metadata/?featured_on=pythonbytes"><strong>pytest-metadata</strong></a></p> <ul> <li>An incredibly useful plugin for adding, you guessed it, metadata, to your pytest results.</li> <li>Required for <a href="https://pypi.org/project/pytest-html/?featured_on=pythonbytes">pytest-html</a> but also useful on it’s own</li> <li>Adds metadata to <ul> <li>text output with --verbose</li> <li>xml output when using --junit-xml, handy for CI systems that support junit.xml</li> </ul></li> <li>Other plugins depend on this and report in other ways, such as pytest-html</li> <li>By default, already grabs <ul> <li>Python version</li> <li>Platform info</li> <li>List of installed packages</li> <li>List of installed pytest plugins</li> </ul></li> <li>You can add your own metadata</li> <li>You can access all metadata (and add to it) from tests, fixtures, and hook functions via a metadata fixture.</li> <li>This is in the <a href="https://pythontest.com/top-pytest-plugins/?featured_on=pythonbytes">Top pytest Plugins list</a>, currently #5.</li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li>I’ve started filtering deprecated plugins from <a href="https://pythontest.com/top-pytest-plugins/?featured_on=pythonbytes">the pytest plugin list</a>.</li> <li>I’m also going to start reviewing the list and pulling out interesting plugins as the topic of the <a href="https://testandcode.com?featured_on=pythonbytes">next season of Test &amp; Code</a>.</li> </ul> <p>Michael:</p> <ul> <li><a href="https://mastodon.social/@hugovk/113312137194438039?kjy=spring&featured_on=pythonbytes">Pillow 11 is out</a></li> <li><a href="https://hachyderm.io/@graham_knapp/113351051856672146?featured_on=pythonbytes">pip install deutschland</a></li> <li><a href="https://talkpython.fm/blog/?featured_on=pythonbytes">Talk Python has a dedicated blog</a>, please subscribe!</li> </ul> <p><strong>Joke:</strong> Dog names</p>

Salesforce API integrations and connected apps 2024-10-28, by Dariusz Suchojad Overview

This instalment in a series of articles about API integrations with Salesforce covers connected apps - how to create them and how to obtain their credentials needed to exchange REST messages with Salesforce.

In Salesforce's terminology, a connected app is, essentially, an API client. It has credentials, a set of permissions, and it works on behalf of a user in an automated manner.

In particular, the kind of a connected app that I am going to create below is one that can be used in backend, server-side integrations that operate without any direct input from end users or administrators, i.e. the app is created once, its permissions and credentials are set once, and then it is able to work uninterrupted in the background, on server side.

Server-side systems are quite unlike other kinds of apps, such as mobile ones, that assume there is a human operator involved - they have their own work characteristics, related yet different, and I am not going to cover them here.

Note that permission types and their scopes are a separate, broad subject and they will described in a separate how-to article.

Finally, I assume that you are either an administrator in a Salesforce organization or that you are preparing information for another person with similar grants in Salesforce.

Conceptually, there is nothing particularly unusual about Salesforce connected apps, it is just its own mini-world of jargon and, at the end of the day, it simply enables you to invoke APIs that Salesforce is built on. It is just that knowing where to click, what to choose and how to navigate the user interface can be a daunting challenge that this article hopes to make easier to overcome.

The steps

For an automated, server-side connected app to make use of Salesforce APIs, the requirements are:

• Having access to username/password credentials
• Creating a connected app
• Granting permissions to the app (not covered in this article)
• Obtaining a customer key and customer secret for the app

You will note that there are four credentials in total:

• Username
• Password
• Customer key
• Customer secret

Also, depending on what chapter of the Salesforce documentation you are reading, you will note that the customer key can be also known as "client_id" whereas another name for the customer secret is "client_secret". These two pairs mean the same.

Access to username/password credentials

For starters, you need to have an account in Salesforce, a combination of username + password that you can log in with and on whose behalf the connected app will be created:

Creating a connected app

Once you are logged in, go to Setup in the top right-hand corner:

In the search box, look up "app manager":

Next, click the "New Connected App" button to the right:

Fill out the basic details such as "Connect App Name" and make sure that you select "Enable OAuth Settings". Then, given that in this document we are not dealing with the subject of permissions at all, grant full access to the connected app and finally click "Save" at the bottom of the page.

Obtaining a customer key and customer secret

We have a connected app but we still do not know what its customer key and secret are. To reveal it, go to the "App Manager" once more, either via the search box or using the menu on the left hand side.

Find your app in the list and click "View" in the list of actions. Observe that it is "View", not "Edit" or "Manage", where you can check what the credentials are:

The customer key and secret van be now revealed in the "API (Enable OAuth Settings)" section:

This concludes the process - you have a connected app and all the credentials needed now.

Testing

Seeing as this document is part of a series of how-tos in the context of Zato, if you would like to integrate with Salesforce in Python, at this point you will be able to follow the steps in another where everything is detailed separately.

Just as a quick teaser, it would look akin to the below.

... # Salesforce REST API endpoint to invoke path = '/sobjects/Campaign/' # Build the request to Salesforce based on what we received request = { 'Name': input.name, 'Segment__c': input.segment, } # Create a reference to our connection definition .. salesforce = self.cloud.salesforce['My Salesforce Connection'] # .. obtain a client to Salesforce .. with salesforce.conn.client() as client: # type: SalesforceClient # .. create the campaign now. response = client.post(path, request) ...

On a much lower level, however, if you would just like to quickly test out whether you configured the connected app correctly, you can invoke from command line a Salesforce REST endpoint that will return an OAuth token, as below.

Note that, as I mentioned it previously, client_id is the same as customer key and client_secret is the same as customer secret.

curl https://example.my.salesforce.com/services/oauth2/token \ -H "X-PrettyPrint: 1" \ --header 'Content-Type: application/x-www-form-urlencoded' \ --data-urlencode 'grant_type=password' \ --data-urlencode 'username=hello@example.com' \ --data-urlencode 'password=my.password' \ --data-urlencode 'client_id=my.customer.key' \ --data-urlencode 'client_secret=my.client.secret'

The result will be, for instance:

{ "access_token" : "008e0000000PTzLPb!4Vzm91PeIWJo.IbPzoEZf2ygEM.6cavCt0YwAGSM", "instance_url" : "https://example.my.salesforce.com", "id" : "https://login.salesforce.com/id/008e0000000PTzLPb/0081fSUkuxPDrir000j1", "token_type" : "Bearer", "issued_at" : "1649064143961", "signature" : "dwb6rwNIzl76kZq8lQswsTyjW2uwvTnh=" }

Above, we have an OAuth bearer token on output - this can be used in subsequent, business REST calls to Salesforce but how to do it exactly in practice is left for another article.

Next steps:

➤ Read about how to use Python to build and integrate enterprise APIs that your tests will cover
➤ Python API integration tutorial
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

In this quiz, you’ll test your understanding of how to reset a pandas DataFrame index.

By working through the questions, you’ll review your knowledge of indexing and also expand on what you learned in the tutorial.

You’ll need to do some research outside of the tutorial to answer all the questions. Embrace this challenge and let it take you on a learning journey.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

What changes are happening under the hood in the latest versions of Python? How are these updates laying the groundwork for a faster Python in the coming years? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of The Python Standard REPL: Try Out Code and Ideas Quickly.

The Python REPL allows you to run Python code interactively, which is useful for testing new ideas, exploring libraries, refactoring and debugging code, and trying out examples.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

 

The PSF is pleased to announce its second batch of PSF Fellows for 2024! Let us welcome the new PSF Fellows for Q2! The following people continue to do amazing things for the Python community:

Leonard Richardson

Blog

Winnie Ke 

Facebook, LinkedIn

Thank you for your continued contributions. We have added you to our Fellow roster.

The above members help support the Python ecosystem by being phenomenal leaders, sustaining the growth of the Python scientific community, maintaining virtual Python communities, maintaining Python libraries, creating educational material, organizing Python events and conferences, starting Python communities in local regions, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.

Let's continue recognizing Pythonistas all over the world for their impact on our community. The criteria for Fellow members is available online: https://www.python.org/psf/fellows/. If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. Quarter 3 nominations are currently in review. We are accepting nominations for Quarter 4 through November 20th, 2024.

Are you a PSF Fellow and want to help the Work Group review nominations? Contact us at psf-fellow at python.org.

Do you struggle to make sure your code is always correct before you check it in? What about your team members' code? That one person who never wants to run the linter? Tired of dealing with tons of conflicts and spurious git changes? You need git pre-commit hooks. We're lucky to have Stefanie Molin on this episode who has done a bunch of writing and teaching of git hooks.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/bluehost'>Bluehost</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Stefanie Molin</b>: <a href="https://stefaniemolin.com/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <br/> <b>Talk Python Blog</b>: <a href="https://talkpython.fm/blog/" target="_blank" >talkpython.fm/blog</a><br/> <br/> <b>How to Set Up Pre-Commit Hooks</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/setup-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Common Pre-Commit Errors and How to Solve Them</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/troubleshooting-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>A Behind-the-Scenes Look at How Pre-Commit Works</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/behind-the-scenes/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Pre-Commit Hook Creation Guide</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/hook-creation-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>(Pre-)Commit to Better Code Workshop</b>: <a href="https://stefaniemolin.com/workshops/pre-commit-workshop/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>exif-stripper</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/exif-stripper/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>exif-stripper on GitHub</b>: <a href="https://github.com/stefmolin/exif-stripper?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>docstring-validation-using-pre-commit-hook</b>: <a href="https://numpydoc.readthedocs.io/en/latest/validation.html#docstring-validation-using-pre-commit-hook" target="_blank" >numpydoc.readthedocs.io</a><br/> <b>Data Morph: Moving Beyond the Datasaurus Dozen</b>: <a href="https://stefaniemolin.com/articles/data-science/introducing-data-morph/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Data Morph on GitHub</b>: <a href="https://github.com/stefmolin/data-morph?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=EzlzX1OL92w" target="_blank" >youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/482/pre-commit-hooks-for-python-devs" target="_blank" >talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

As is probably apparent from the sequence of blog posts about the topic in the last year, I have been thinking about and working on integer optimizations in the JIT compiler a lot. This work was mainly motivated by Pydrofoil, where integer operations matter a lot more than for your typical Python program.

In this post I'll describe my most recent change, which is a new small domain specific language that I implemented to specify peephole optimizations on integer operations in the JIT. It uses pattern matching to specify how (sequences of) integer operations should be simplified and optimized. The rules are then compiled to RPython code that then becomes part of the JIT's optimization passes.

To make it less likely to introduce incorrect optimizations into the JIT, the rules are automatically proven correct with Z3 as part of the build process (for a more hands-on intro to how that works you can look at the knownbits post). In this blog post I want to motivate why I introduced the DSL and give an introduction to how it works.

Motivation

This summer, after I wrote my scripts to mine JIT traces for missed optimization opportunities, I started implementing a few of the integer peephole rewrite that the script identified. Unfortunately, doing so led to the problem that the way we express these rewrites up to now is very imperative and verbose. Here's a snippet of RPython code that shows some rewrites for integer multiplication (look at the comments to see what the different parts actually do). You don't need to understand the code in detail, but basically it's in very imperative style and there's quite a lot of boilerplate.

def optimize_INT_MUL(self, op): arg0 = get_box_replacement(op.getarg(0)) b0 = self.getintbound(arg0) arg1 = get_box_replacement(op.getarg(1)) b1 = self.getintbound(arg1) if b0.known_eq_const(1): # 1 * x == x self.make_equal_to(op, arg1) elif b1.known_eq_const(1): # x * 1 == x self.make_equal_to(op, arg0) elif b0.known_eq_const(0) or b1.known_eq_const(0): # 0 * x == x * 0 == 0 self.make_constant_int(op, 0) else: for lhs, rhs in [(arg0, arg1), (arg1, arg0)]: lh_info = self.getintbound(lhs) if lh_info.is_constant(): x = lh_info.get_constant_int() if x & (x - 1) == 0: # x * (2 ** c) == x << c new_rhs = ConstInt(highest_bit(lh_info.get_constant_int())) op = self.replace_op_with(op, rop.INT_LSHIFT, args=[rhs, new_rhs]) self.optimizer.send_extra_operation(op) return elif x == -1: # x * -1 == -x op = self.replace_op_with(op, rop.INT_NEG, args=[rhs]) self.optimizer.send_extra_operation(op) return else: # x * (1 << y) == x << y shiftop = self.optimizer.as_operation(get_box_replacement(lhs), rop.INT_LSHIFT) if shiftop is None: continue if not shiftop.getarg(0).is_constant() or shiftop.getarg(0).getint() != 1: continue shiftvar = get_box_replacement(shiftop.getarg(1)) shiftbound = self.getintbound(shiftvar) if shiftbound.known_nonnegative() and shiftbound.known_lt_const(LONG_BIT): op = self.replace_op_with( op, rop.INT_LSHIFT, args=[rhs, shiftvar]) self.optimizer.send_extra_operation(op) return return self.emit(op)

Adding more rules to these functions is very tedious and gets super confusing when the functions get bigger. In addition I am always worried about making mistakes when writing this kind of code, and there is no feedback at all about which of these rules are actually applied a lot in real programs.

Therefore I decided to write a small domain specific language with the goal of expressing these rules in a more declarative way. In the rest of the post I'll describe the DSL (most of that description is adapted from the documentation about it that I wrote).

The Peephole Rule DSL Simple transformation rules

The rules in the DSL specify how integer operation can be transformed into cheaper other integer operations. A rule always consists of a name, a pattern, and a target. Here's a simple rule:

add_zero: int_add(x, 0) => x

The name of the rule is add_zero. It matches operations in the trace of the form int_add(x, 0), where x will match anything and 0 will match only the constant zero. After the => arrow is the target of the rewrite, i.e. what the operation is rewritten to, in this case x.

The rule language has a list of which of the operations are commutative, so add_zero will also optimize int_add(0, x) to x.

Variables in the pattern can repeat:

sub_x_x: int_sub(x, x) => 0

This rule matches against int_sub operations where the two arguments are the same (either the same box, or the same constant).

Here's a rule with a more complicated pattern:

sub_add: int_sub(int_add(x, y), y) => x

This pattern matches int_sub operations, where the first argument was produced by an int_add operation. In addition, one of the arguments of the addition has to be the same as the second argument of the subtraction.

The constants MININT, MAXINT and LONG_BIT (which is either 32 or 64, depending on which platform the JIT is built for) can be used in rules, they behave like writing numbers but allow bit-width-independent formulations:

is_true_and_minint: int_is_true(int_and(x, MININT)) => int_lt(x, 0)

It is also possible to have a pattern where some arguments needs to be a constant, without specifying which constant. Those patterns look like this:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete # more goes here => int_sub(x, C)

Variables in the pattern that start with a C match against constants only. However, in this current form the rule is incomplete, because the variable C that is being used in the target operation is not defined anywhere. We will see how to compute it in the next section.

Computing constants and other intermediate results

Sometimes it is necessary to compute intermediate results that are used in the target operation. To do that, there can be extra assignments between the rule head and the rule target.:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete C = C1 + C1 => int_sub(x, C)

The right hand side of such an assignment is a subset of Python syntax, supporting arithmetic using +, -, *, and certain helper functions. However, the syntax allows you to be explicit about unsignedness for some operations. E.g. >>u exists for unsigned right shifts (and I plan to add >u, >=u, <u, <=u for comparisons).

Here's an example of a rule that uses >>u:

urshift_lshift_x_c_c: uint_rshift(int_lshift(x, C), C) mask = (-1 << C) >>u C => int_and(x, mask) Checks

Some rewrites are only true under certain conditions. For example, int_eq(x, 1) can be rewritten to x, if x is known to store a boolean value. This can be expressed with checks:

eq_one: int_eq(x, 1) check x.is_bool() => x

A check is followed by a boolean expression. The variables from the pattern can be used as IntBound instances in checks (and also in assignments) to find out what the abstract interpretation of the JIT knows about the value of a trace variable (IntBound is the name of the abstract domain that the JIT uses for integers, despite the fact that it also stores knownbits information nowadays).

Here's another example:

mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

It expresses that x * (1 << y) can be rewritten to x << y but checks that y is known to be between 0 and LONG_BIT.

Checks and assignments can be repeated and combined with each other:

mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift)

In addition to calling methods on IntBound instances, it's also possible to access their attributes, like in this rule:

and_x_c_in_range: int_and(x, C) check x.lower >= 0 and x.upper <= C & ~(C + 1) => x Rule Ordering and Liveness

The generated optimizer code will give preference to applying rules that produce a constant or a variable as a rewrite result. Only if none of those match do rules that produce new result operations get applied. For example, the rules sub_x_x and sub_add are tried before trying sub_add_consts, because the former two rules optimize to a constant and a variable respectively, while the latter produces a new operation as the result.

The rule sub_add_consts has a possible problem, which is that if the intermediate result of the int_add operation in the rule head is used by some other operations, then the sub_add_consts rule does not actually reduce the number of operations (and might actually make things slightly worse due to increased register pressure). However, currently it would be extremely hard to take that kind of information into account in the optimization pass of the JIT, so we optimistically apply the rules anyway.

Checking rule coverage

Every rewrite rule should have at least one unit test where it triggers. To ensure this, the unit test file that mainly checks integer optimizations in the JIT has an assert at the end of a test run, that every rule fired at least once.

Printing rule statistics

The JIT can print statistics about which rule fired how often in the jit-intbounds-stats logging category, using the PYPYLOG mechanism. For example, to print the category to stdout at the end of program execution, run PyPy like this:

PYPYLOG=jit-intbounds-stats:- pypy ...

The output of that will look something like this:

int_add add_reassoc_consts 2514 add_zero 107008 int_sub sub_zero 31519 sub_from_zero 523 sub_x_x 3153 sub_add_consts 159 sub_add 55 sub_sub_x_c_c 1752 sub_sub_c_x_c 0 sub_xor_x_y_y 0 sub_or_x_y_y 0 int_mul mul_zero 0 mul_one 110 mul_minus_one 0 mul_pow2_const 1456 mul_lshift 0 ... Termination and Confluence

Right now there are unfortunately no checks that the rules actually rewrite operations towards "simpler" forms. There is no cost model, and also nothing that prevents you from writing a rule like this:

neg_complication: int_neg(x) # leads to infinite rewrites => int_mul(-1, x)

Doing this would lead to endless rewrites if there is also another rule that turns multiplication with -1 into negation.

There is also no checking for confluence (yet?), i.e. the property that all rewrites starting from the same input trace always lead to the same output trace, no matter in which order the rules are applied.

Proofs

It is very easy to write a peephole rule that is not correct in all corner cases. Therefore all the rules are proven correct with Z3 before compiled into actual JIT code, by default. When the proof fails, a (hopefully minimal) counterexample is printed. The counterexample consists of values for all the inputs that fulfil the checks, values for the intermediate expressions, and then two different values for the source and the target operations.

E.g. if we try to add the incorrect rule:

mul_is_add: int_mul(a, b) => int_add(a, b)

We get the following counterexample as output:

Could not prove correctness of rule 'mul_is_add' in line 1 counterexample given by Z3: counterexample values: a: 0 b: 1 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 0 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 1

If we add conditions, they are taken into account and the counterexample will fulfil the conditions:

mul_is_add: int_mul(a, b) check a.known_gt_const(1) and b.known_gt_const(2) => int_add(a, b)

This leads to the following counterexample:

Could not prove correctness of rule 'mul_is_add' in line 46 counterexample given by Z3: counterexample values: a: 2 b: 3 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 6 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 5

Some IntBound methods cannot be used in Z3 proofs because their control flow is too complex. If that is the case, they can have Z3-equivalent formulations defined (in every case this is done, it's a potential proof hole if the Z3 friendly reformulation and the real implementation differ from each other, therefore extra care is required to make very sure they are equivalent).

It's possible to skip the proof of individual rules entirely by adding SORRY_Z3 to its body (but we should try not to do that too often):

eq_different_knownbits: int_eq(x, y) SORRY_Z3 check x.known_ne(y) => 0 Checking for satisfiability

In addition to checking whether the rule yields a correct optimization, we also check whether the rule can ever apply. This ensures that there are some runtime values that would fulfil all the checks in a rule. Here's an example of a rule violating this:

never_applies: int_is_true(x) check x.known_lt_const(0) and x.known_gt_const(0) # impossible condition, always False => x

Right now the error messages if this goes wrong are not completely easy to understand. I hope to be able to improve this later:

Rule 'never_applies' cannot ever apply in line 1 Z3 did not manage to find values for variables x such that the following condition becomes True: And(x <= x_upper, x_lower <= x, If(x_upper < 0, x_lower > 0, x_upper < 0)) Implementation Notes

The implementation of the DSL is done in a relatively ad-hoc manner. It is parsed using rply, there's a small type checker that tries to find common problems in how the rules are written. Z3 is used via the Python API, like in the previous blog posts that are using it. The pattern matching RPython code is generated using an approach inspired by Luc Maranget's paper Compiling Pattern Matching to Good Decision Trees. See this blog post for an approachable introduction.

Conclusion

Now that I've described the DSL, here are the rules that are equivalent to the imperative code in the motivation section:

mul_zero: int_mul(x, 0) => 0 mul_one: int_mul(x, 1) => x mul_minus_one: int_mul(x, -1) => int_neg(x) mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift) mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

The current status of the DSL is that it got merged to PyPy's main branch. I rewrote a part of the integer rewrites into the DSL, but some are still in the old imperative style (mostly for complicated reasons, the easily ported ones are all done). Since I've only been porting optimizations that had existed prior to the existence of the DSL, performance numbers of benchmarks didn't change.

There are a number of features that are still missing and some possible extensions that I plan to work on in the future:

• All the integer operations that the DSL handles so far are the variants that do not check for overflow (or where overflow was proven to be impossible to happen). In regular Python code the overflow-checking variants int_add_ovf etc are much more common, but the DSL doesn't support them yet. I plan to fix this, but don't completely understand how the correctness proofs for them should be done correctly.

• A related problem is that I don't understand what it means for a rewrite to be correct if some of the operations are only defined for a subset of the input values. E.g. division isn't defined if the divisor is zero. In theory, a division operation in the trace should always be preceded by a check that the divisor isn't zero. But sometimes other optimization move the check around and the connection to the division gets lost or muddled. What optimizations can we still safely perform on the division? There's lots of prior work on this question, but I still don't understand what the correct approach in our context would be.

• Ordering comparisons like int_lt, int_le and their unsigned variants are not ported to the DSL yet. Comparisons are an area where the JIT is not super good yet at optimizing away operations. This is a pretty big topic and I've started a project with Nico Rittinghaus to try to improve the situation a bit more generally.

• A more advanced direction of work would be to implement a simplified form of e-graphs (or ae-graphs). The JIT has like half of an e-graph data structure already, and we probably can't afford a full one in terms of compile time costs, but maybe we can have two thirds or something?

Acknowledgements

Thank you to Max Bernstein and Martin Berger for super helpful feedback on drafts of the post!

In this episode of the Python Show Podcast, David Mertz is our guest. David is a prolific writer about the Python programming language. From his extremely popular IPM Developerworks articles to his multiple books on the Python language, David has been a part of the Python community for decades.

We ended up chatting about:

• The history of Python

• Book writing

• Conference speaking

• The PSF

• and more!

Show Links

• David Mertz’s website

• PyDev of the Week: David Mertz on Mouse vs Python

• David Mertz on InformIT / Pearson

Python threading allows you to run parts of your code concurrently, making the code more efficient. However, when you introduce threading to your code without knowing about thread safety, you may run into issues such as race conditions. You solve these with tools like locks, semaphores, events, conditions, and barriers.

By the end of this tutorial, you’ll be able to identify safety issues and prevent them by using the synchronization primitives in Python’s threading module to make your code thread-safe.

In this tutorial, you’ll learn:

• What thread safety is
• What race conditions are and how to avoid them
• How to identify thread safety issues in your code
• What different synchronization primitives exist in the threading module
• How to use synchronization primitives to make your code thread-safe

To get the most out of this tutorial, you’ll need to have basic experience working with multithreaded code using Python’s threading module and ThreadPoolExecutor.

Get Your Code: Click here to download the free sample code that you’ll use to learn about thread safety techniques in Python.

Take the Quiz: Test your knowledge with our interactive “Python Thread Safety: Using a Lock and Other Techniques” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Python Thread Safety: Using a Lock and Other Techniques

In this quiz, you'll test your understanding of Python thread safety. You'll revisit the concepts of race conditions, locks, and other synchronization primitives in the threading module. By working through this quiz, you'll reinforce your knowledge about how to make your Python code thread-safe.

Threading in Python

In this section, you’ll get a general overview of how Python handles threading. Before discussing threading in Python, it’s important to revisit two related terms that you may have heard about in this context:

• Concurrency: The ability of a system to handle multiple tasks by allowing their execution to overlap in time but not necessarily happen simultaneously.
• Parallelism: The simultaneous execution of multiple tasks that run at the same time to leverage multiple processing units, typically multiple CPU cores.

Python’s threading is a concurrency framework that allows you to spin up multiple threads that run concurrently, each executing pieces of code. This improves the efficiency and responsiveness of your application. When running multiple threads, the Python interpreter switches between them, handing the control of execution over to each thread.

By running the script below, you can observe the creation of four threads:

Python threading_example.py import threading import time from concurrent.futures import ThreadPoolExecutor def threaded_function(): for number in range(3): print(f"Printing from {threading.current_thread().name}. {number=}") time.sleep(0.1) with ThreadPoolExecutor(max_workers=4, thread_name_prefix="Worker") as executor: for _ in range(4): executor.submit(threaded_function) Copied!

In this example, threaded_function prints the values zero to two that your for loop assigns to the loop variable number. Using a ThreadPoolExecutor, four threads are created to execute the threaded function. ThreadPoolExecutor is configured to run a maximum of four threads concurrently with max_workers=4, and each worker thread is named with a “Worker” prefix, as in thread_name_prefix="Worker".

In print(), the .name attribute on threading.current_thread() is used to get the name of the current thread. This will help you identify which thread is executed each time. A call to sleep() is added inside the threaded function to increase the likelihood of a context switch.

You’ll learn what a context switch is in just a moment. First, run the script and take a look at the output:

Shell $ python threading_example.py Printing from Worker_0. number=0 Printing from Worker_1. number=0 Printing from Worker_2. number=0 Printing from Worker_3. number=0 Printing from Worker_0. number=1 Printing from Worker_2. number=1 Printing from Worker_1. number=1 Printing from Worker_3. number=1 Printing from Worker_0. number=2 Printing from Worker_2. number=2 Printing from Worker_1. number=2 Printing from Worker_3. number=2 Copied!

Each line in the output represents a print() call from a worker thread, identified by Worker_0, Worker_1, Worker_2, and Worker_3. The number that follows the worker thread name shows the current iteration of the loop each thread is executing. Each thread takes turns executing the threaded_function, and the execution happens in a concurrent rather than sequential manner.

For example, after Worker_0 prints number=0, it’s not immediately followed by Worker_0 printing number=1. Instead, you see outputs from Worker_1, Worker_2, and Worker_3 printing number=0 before Worker_0 proceeds to number=1. You’ll notice from these interleaved outputs that multiple threads are running at the same time, taking turns to execute their part of the code.

This happens because the Python interpreter performs a context switch. This means that Python pauses the execution state of the current thread and passes control to another thread. When the context switches, Python saves the current execution state so that it can resume later. By switching the control of execution at specific intervals, multiple threads can execute code concurrently.

You can check the context switch interval of your Python interpreter by typing the following in the REPL:

Python >>> import sys >>> sys.getswitchinterval() 0.005 Copied!

The output of calling the getswitchinterval() is a number in seconds that represents the context switch interval of your Python interpreter. In this case, it’s 0.005 seconds or five milliseconds. You can think of the switch interval as how often the Python interpreter checks if it should switch to another thread.

An interval of five milliseconds doesn’t mean that threads switch exactly every five milliseconds, but rather that the interpreter considers switching to another thread at these intervals.

The switch interval is defined in the Python docs as follows:

Read the full article at https://realpython.com/python-thread-lock/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of Python Class Constructors.

By working through this quiz, you’ll revisit the internal instantiation process, object initialization using .__init__(), and fine-tuning object creation by overriding .__new__().

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

#652 – OCTOBER 22, 2024
View in Browser »

Structural Pattern Matching in Python

In this tutorial, you’ll learn how to harness the power of structural pattern matching in Python. You’ll explore the new syntax, delve into various pattern types, and find appropriate applications for pattern matching, all while identifying common pitfalls.
REAL PYTHON

Combinatoric Iterators From itertools

The itertools module offers four combinatoric iterators that generate different combined outputs from one or more iterable. This post covers all of them: product, permutations, combinations, and combinations_with_replacement.
JUHA-MATTI SANTALA

Scrape Web Data at Scale with a 98.7% Success Rate

Extract all the data you need from any website without getting blocked with ZenRows’ Scraper API – a complete toolkit with premium proxies, anti-CAPTCHA, cloud-based scalable browsers, and more. Start your free trial now →
ZENROWS sponsor

CPython Internals: Your Guide to the Python 3 Interpreter

Unlock the inner workings of the Python language, compile the Python interpreter from source code, and participate in the development of CPython. Guido van Rossum, the creator of Python, says: “I can recommend CPython Internals to anyone who wants to get going with hacking on CPython” →
ANTHONY SHAW sponsor

SSH Scripting With Fabric and Python

Reading and writing files is a basic task that most software applications need to do, but what if you need to do that on remote machines? This tutorial introduces you to Fabric and how to connect over SSH in Python.
MIKE DRISCOLL

Python 3.14.0 Alpha 1 Released

CPYTHON DEV BLOG

Quiz: Structural Pattern Matching

REAL PYTHON

Quiz: Iterators and Iterables in Python

REAL PYTHON

Quiz: Python import: Advanced Techniques and Tips

REAL PYTHON

Articles & Tutorials Mind Your Image Metadata

Most devices record a variety of metadata when generating images. While some of that information may be innocuous, you could end up exposing the GPS coordinates to your home if you aren’t careful. In this article, Stefanie provides a brief introduction to image metadata, and then shows you how to remove it with exif-stripper.
STEFANIEMOLIN.COM • Shared by Stefanie Molin

Open Source in Python and JavaScript 2024

Python vs. JavaScript: Which open-source community is leading the way? This analysis of 36,000 GitHub repositories explores the evolution of Python and JavaScript ecosystems, highlighting key trends and popular topics. Discover how open-source communities of Python and JavaScript have shaped the tech landscape.
PYCHALLENGER.COM • Shared by Erik Nogueira Kückelheim

Accelerate Edge Devices with High-Performance AI Power

Experience the power of Edge AI—delivering lightning-fast, real-time processing where it matters. Optimize your applications to push performance and accuracy beyond limits with Intel’s OpenVINO toolkit.
INTEL CORPORATION sponsor

Using Type Hints for Multiple Return Types in Python

In this video course, you’ll learn how to define multiple return types using type hints in Python. This course covers working with single or multiple pieces of data, defining type aliases, and performing type checking using a third-party static type checker tool.
REAL PYTHON course

Narwhals: Expanding DataFrame Compatibility

How does a Python tool support all types of DataFrames and their various features? Could a lightweight library be used to add compatibility for newer formats like Polars or PyArrow? This week on the show, we speak with Marco Gorelli about his project, Narwhals.
REAL PYTHON podcast

Syntactic Sugar: Why Python Is Sweet and Pythonic

In this tutorial, you’ll learn what syntactic sugar is and how Python uses it to help you create more readable, descriptive, clean, and Pythonic code. You’ll also learn how to replace a given piece of syntactic sugar with another syntax construct.
REAL PYTHON

Entering Text in the Terminal Is Complicated

Julia asked some folks on Mastodon what they found confusing about working in a terminal. It turns out that entering text in the terminal is complicated. This post talks about why that is and how to understand it better.
JULIA EVANS

4 Lessons From Small Teams That Ship Fast

Software engineering provides a lot of leverage and small teams can do a large amount of work. This post talks about several common examples in the industry where a small group created a big product.
LEONARDO CREED

Perks of Being a Python Core Developer

Mariatta has been a Python Core Developer since 2017. If you want to know just what that means, this post talks about all the things she gets to do.
MARIATTA

Custom Dictionary Types in Pydantic

Pydantic lets you create custom types. This post talks about how to create a custom dictionary type using root models and Enums.
BRYAN ANTHONIO

How to Use Lambda Functions in Python

This article looks at some examples and best practices when using Lambda functions in Python.
FEDERICO TROTTA • Shared by AppSignal

Projects & Code ryp: R Inside Python

GITHUB.COM/WAINBERG

pyglove: Symbolic OO for Python

GITHUB.COM/GOOGLE

pipreqs: Generate requirements.txt Based on Imports

GITHUB.COM/BNDR

Bowler: Safe Code Refactoring for Modern Python

GITHUB.COM/FACEBOOKINCUBATOR

nanodjango: Full Django in a Single File

GITHUB.COM/RADIAC

Events Weekly Real Python Office Hours Q&A (Virtual)

October 23, 2024
REALPYTHON.COM

October Oslo Python Meetup

October 24, 2024
MEETUP.COM

PyCon APAC 2024

October 25 to October 27, 2024
PYCON.ID

PyCon Korea 2024

October 25 to October 28, 2024
PYCON.KR

PythonHo Conference 2024

October 26 to October 28, 2024
PYTHONHO.COM

PythOnRio Meetup

October 26, 2024
PYTHON.ORG.BR

Django Girls Aba

October 27, 2024
DJANGOGIRLS.ORG

PyCon FR 2024

October 31 to November 3, 2024
PYCON.FR

PyCon Zimbabwe

October 31 to November 3, 2024
PYCON.ORG

Happy Pythoning!
This was PyCoder’s Weekly Issue #652.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

The Python Global Interpreter Lock or GIL, in simple words, is a mutex (or a lock) that allows only one thread to hold the control of the Python interpreter.

This means that only one thread can be in a state of execution at any point in time. The impact of the GIL isn’t visible to developers who execute single-threaded programs, but it can be a performance bottleneck in CPU-bound and multi-threaded code.

Since the GIL allows only one thread to execute at a time even in a multi-threaded architecture with more than one CPU core, the GIL has gained a reputation as an “infamous” feature of Python.

In this video course you’ll learn how the GIL affects the performance of your Python programs, and how you can mitigate the impact it might have on your code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of how to define your own Python function.

You’ll revisit theoretical knowledge about passing values to functions, when to divide your program into separate user-defined functions, and all the tools you’ll need to define complex and powerful functions in Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PythonAnywhere has been around for over 10 years, and as our platform continues to grow with thousands of users, we’re committed to keeping it in top shape. Part of this involves upgrading some of the older parts of our infrastructure, with a special focus on our file storage servers—some of the oldest systems we have.

Today's exercice is just about turning a very nice example of the python soundevice module into something that works for me© to help me tune my bass.

Long story short, I suck at tuning my instrument and just lost my tuner...

This will require the python module soundevice and matplotlib.

So in order to tune my guitar I indeed need a spectrosonogram that displays the frequencies captured in real time by an audio device with an output readable enough I can actually know if I am nearing a legit frequency called a Note.

The frequencies for the notes are pretty arbitrary and I chose to only show the frequency for E, A , D, G, B since I have a 5 strings bass.
I chose the frequency between 100 and 2000 knowing that anyway any frequency below will trigger harmonics and above will trigger reasonance in the right frequency frame.

Plotting a spectrogram is done by tweaking the eponym matplotlib grapher with values chosen to fit my need and show me a laser thin beam around the right frequency. #!/usr/bin/env python3 """Show a text-mode spectrogram using live microphone data.""" import argparse import math import shutil import matplotlib.pyplot as plt from multiprocessing import Process, Queue import matplotlib.animation as animation import numpy as np import sounddevice as sd usage_line = ' press enter to quit,' def int_or_str(text): """Helper function for argument parsing.""" try: return int(text) except ValueError: return text try: columns, _ = shutil.get_terminal_size() except AttributeError: columns = 80 parser = argparse.ArgumentParser(add_help=False) parser.add_argument( '-l', '--list-devices', action='store_true', help='show list of audio devices and exit') args, remaining = parser.parse_known_args() if args.list_devices: print(sd.query_devices()) parser.exit(0) parser = argparse.ArgumentParser( description=__doc__ + '\n\nSupported keys:' + usage_line, formatter_class=argparse.RawDescriptionHelpFormatter, parents=[parser]) parser.add_argument( '-b', '--block-duration', type=float, metavar='DURATION', default=50, help='block size (default %(default)s milliseconds)') parser.add_argument( '-d', '--device', type=int_or_str, help='input device (numeric ID or substring)') parser.add_argument( '-g', '--gain', type=float, default=10, help='initial gain factor (default %(default)s)') parser.add_argument( '-r', '--range', type=float, nargs=2, metavar=('LOW', 'HIGH'), default=[50, 4000], help='frequency range (default %(default)s Hz)') args = parser.parse_args(remaining) low, high = args.range if high <= low: parser.error('HIGH must be greater than LOW') q = Queue() try: samplerate = sd.query_devices(args.device, 'input')['default_samplerate'] def plot(q): global samplerate fig, ( ax,axs) = plt.subplots(nrows=2) plt.ioff() def animate(i,q): data = q.get() ax.clear() axs.clear() axs.plot(data) ax.set_yticks([ 41.20, 82.41, 164.8, 329.6, 659.3, # E 55.00, 110.0, 220.0, 440.0, 880.0, # A 73.42, 146.8, 293.7, 587.3, # D 49.00, 98.00, 196.0, 392.0, 784.0, #G 61.74, 123.5, 246.9, 493.9, 987.8 ])#B ax.specgram(data[:,-1],mode="magnitude", Fs=samplerate*2, scale="linear",NFFT=9002) ax.set_ylim(150,1000) ani = animation.FuncAnimation(fig, animate,fargs=(q,), interval=500) plt.show() plotrt = Process(target=plot, args=(q,)) plotrt.start() def callback(indata, frames, time, status): if any(indata): q.put(indata) else: print('no input') with sd.InputStream(device=args.device, channels=1, callback=callback, blocksize=int(samplerate * args.block_duration /50 ), samplerate=samplerate) as sound: while True: response = input() if response in ('', 'q', 'Q'): break for ch in response: if ch == '+': args.gain *= 2 elif ch == '-': args.gain /= 2 else: print('\x1b[31;40m', usage_line.center(args.columns, '#'), '\x1b[0m', sep='') break except KeyboardInterrupt: parser.exit('Interrupted by user') except Exception as e: parser.exit(type(e).__name__ + ': ' + str(e))