Planet Python
PyCharm: PyCharm 2024.2.1: What’s New!
PyCharm 2024.2.1 is here! This release’s key features include initial support for Python 3.13, improvements to the Data View tool window, and enhanced code assistance for Django.
Don’t forget to visit our What’s New page to get all the new updates. Download the latest version from our website, or update your current version through our free Toolbox App.
Download PyCharm 2024.2.1 PyCharm 2024.2.1 key features Data View PROPyCharm now provides two color-scheme options for the table heatmaps in the Data View tool window: the Diverging and Sequential color schemes.
The Diverging color scheme emphasizes variation relative to a norm. It consists of two contrasting colors that deviate from a central value in two opposite directions.
The Sequential color scheme consists of a single color or a range of closely related colors that vary in intensity.
You can apply the heatmap color schemes to the whole table or to each column separately, or you can use coloring only for Boolean values.
Python 3.13PyCharm now recognizes TypeIs syntax, providing proper type inference and code completion for user-defined narrowed functions. As part of Python 3.13 support, the IDE is now also aware of ReadOnly keys in TypedDict and warns you if something is assigned to a ReadOnly member.
Read more Django: Completion for ModelAdmin fields, and more PROGet intelligent code completion, refactoring, and navigation for the fields in ModelAdmin classes. Other productivity enhancements include a warning about newly created apps that have not been added to the INSTALLED_APPS declaration, and the ability to insert an app’s tag into the manage.py console automatically when migrations are made from the Django Structure tool window.
Read our release notes for the full breakdown and more details on all of the features in 2024.2.1. If you encounter any problems, please report them in our issue tracker so we can address them promptly.
Connect with us on X (formerly Twitter) to share your thoughts on PyCharm 2024.2.1. We’re looking forward to hearing them!
Python Software Foundation: Python Developers Survey 2023 Results
We are excited to share the results of the seventh official annual Python Developers Survey. This survey is done yearly as a collaborative effort between the Python Software Foundation and JetBrains. Responses were collected from November 2023 through February 2024. This year, we kept the response period open longer to facilitate as much global representation as possible. More than 25,000 Python developers and enthusiasts from almost 200 countries and regions participated in the survey to reveal the current state of the language and the ecosystem around it.
Check out the survey results!
The survey aims to map the Python landscape and covers the following topics:
- General Python usage
- Purpose for using Python
- Python versions
- Frameworks and Libraries
- Cloud Platforms
- Data science
- Development tools
- Python packaging
- Demographics
We encourage you to check out the methodology and the raw data for this year's Python Developers Survey, as well as those from past years (2022, 2021, 2020, 2019, 2018, and 2017). We would love to hear about what you learn by digging into the numbers! Share your results and comments with us on social media by mentioning JetBrains (LinkedIn, X) and the PSF (Mastodon, LinkedIn, X) using the #pythondevsurvey hashtag. Based on the feedback we received last year, we made adjustments to the 2023 survey- so we welcome suggestions and feedback that could help us improve again for next year!
PyPy: Guest Post: How PortaOne uses PyPy for high-performance processing, connecting over 1B of phone calls every month
The PyPy project is always happy to hear about industrial use and deployments of PyPy. For the GC bug finding task earlier this year, we collaborated with PortaOne and we're super happy that Serhii Titov, head of the QA department at PortaOne, was up to writing this guest post to describe their use and experience with the project.
What does PortaOne do?We at PortaOne Inc. allow telecom operators to launch new services (or provide existing services more efficiently) using our VoIP platform (PortaSIP) and our real-time charging system (PortaBilling), which provides additional features for cloud PBX, such as call transfer, queues, interactive voice response (IVR) and more. At this moment our support team manages several thousand servers with our software installed in 100 countries, through which over 500 telecommunication service providers connect millions of end users every day. The unique thing about PortaOne is that we supply the source code of our product to our customers - something unheard of in the telecom world! Thus we attract "telco innovators", who use our APIs to build around the system and the source code to create unique tweaks of functionality, which produces amazing products.
At the core of PortaSIP is the middle-ware component (the proper name for it is "B2BUA", but that probably does not say much to anyone outside of experts in VoIP), which implements the actual handling of SIP calls, messages, etc. and all added features (for instance, trying to send a call via telco operators through which the cost per minute is lower). It has to be fast (since even a small delay in establishing a call is noticed by a customer), reliable (everyone hates when a call drops or cannot be completed) and yet easily expandable with new functionality. This is why we decided to use Python as opposed to C/C++ or similar programming languages, which are often used in telecom equipment.
The B2BUA component is a batch of similar Python processes that are looped inside a asyncore.dispatcher wrapper. The load balancing between these Python processes is done by our stateless SIP proxy server written in C++. All our sockets are served by this B2BUA. We have our custom client-wrappers around pymysql, redis, cassandra-driver and requests to communicate with external services. Some of the Python processes use cffi wrappers around C-code to improve their performance (examples: an Oracle DB driver, a client to a radius server, a custom C logger).
The I/O operations that block the main thread of the Python processes are processed in sub-threads. We have custom wrappers around threading.Thread and also asyncore.dispatcher. The results of such operations are returned to the main thread.
Improving our performance with PyPyWe started with CPython and then in 2014 switched to PyPy because it was faster. Here's an exact quote from our first testing notes: "PyPy gives significant performance boost, ~50%". Nowadays, after years of changes in all the software involved, PyPy still gives us +50% boost compared to CPython.
Taking care of real time traffic for so many people around the globe is something we're really proud of. I hope the PyPy team can be proud of it as well, as the PyPy product is a part of this solution.
Finding a garbage collector bug: stage 1, the GC hooksHowever our path with PyPy wasn't perfectly smooth. There were very rare cases of crashes on PyPy that we weren't able to catch. That's because to make coredump useful we needed to switch to PyPy with debug, but we cannot let it run in that mode on a production system for an extended period of time, and we did not have any STR (steps-to-reproduce) to make PyPy crash again in our lab. That's why we kept (and still keep) both interpreters installed just in case, and we would switch to CPython if we noticed it happening.
At the time of updating PyPy from 3.5 to 3.6 our QA started noticing those crashes more often, but we still had no luck with STR or collecting proper coredumps with debug symbols. Then it became even worse after our development played with the Garbage Collector's options to increase performance of our middleware component. The crashes started to affect our regular performance testing (controlled by QA manager Yevhenii Bovda). At that point it was decided that we can no longer live like that and so we started an intense investigation.
During the first stage of our investigation (following the best practice of troubleshooting) we narrowed down the issue as much as we could. So, it was not our code, it was definitely somewhere in PyPy. Eventually our SIP software engineer Yevhenii Yatchenko found out that this bug is connected with the use of our custom hooks in the GC. Yevhenii created ticket #4899 and within 2-3 days we got a fix from a member of the PyPy team, in true open-source fashion.
Finding a garbage collector bug: stage 2, the real bugThen came stage 2. In parallel with the previous ticket, Yevhenii created #4900 that we still see failing with coredumps quite often, and they are not connected to GC custom hooks. In a nutshell, it took us dozens of back and forward emails, three Zoom sessions and four versions of a patch to solve the issue. During the last iteration we got a new set of options to try and a new version of the patch. Surprisingly, that helped! What a relief! So, the next logical step was to remove all debug options and run PyPy only with the patch. Unfortunately, it started to fail again and we came to the obvious conclusion that what will help us is not a patch, but one of options we were testing out. At that point we found out that PYPY_GC_MAX_PINNED=0 is a necessary and sufficient condition to solve our issue. This points to another bug in the garbage collector, somehow related to object pinning.
Here's our current state: we have to add PYPY_GC_MAX_PINNED=0, but we do not face the crashes anymore.
Conclusion and next stepsGratitude is extended to Carl for his invaluable assistance in resolving the nasty bugss, because it seems we're the only ones who suffered from the last one and we really did not want to fall back to CPython due to its performance disadvantage.
Serhii Titov, head of the QA department at PortaOne Inc.
P.S. If you are a perfectionist and at this point you have mixed feelings and you are still bothered by the question "But there might still be a bug in the GC, what about that?" - Carl has some ideas about it and he will sort it out (we will help with the testing/verification part).
Matt Layman: No Frills, Just Go: Standard Library Only Web Apps
Python Morsels: Arithmetic in Python
An explanation of Python's two number types (integers and floating point numbers), supported arithmetic operations, and an explanation of operator precedence.
Table of contents
- Integers
- Floating point numbers
- Mixing integers and floating point numbers
- Arithmetic operations
- Operator precedence in Python
- Arithmetic in Python is similar to in math
Integers are used for representing whole numbers.
>>> 5 5 >>> 0 0 >>> 999999999999 999999999999 >>> -10 -10Any number that doesn't have a decimal point in it is an integer.
Floating point numbersFloating point numbers are used …
Read the full article: https://www.pythonmorsels.com/arithmetic-in-python/Real Python: Web Scraping With Scrapy and MongoDB
Scrapy is a robust Python web scraping framework that can manage requests asynchronously, follow links, and parse site content. To store scraped data, you can use MongoDB, a scalable NoSQL database, that stores data in a JSON-like format. Combining Scrapy with MongoDB offers a powerful solution for web scraping projects, leveraging Scrapy’s efficiency and MongoDB’s flexible data storage.
In this tutorial, you’ll learn how to:
- Set up and configure a Scrapy project
- Build a functional web scraper with Scrapy
- Extract data from websites using selectors
- Store scraped data in a MongoDB database
- Test and debug your Scrapy web scraper
If you’re new to web scraping and you’re looking for flexible and scalable tooling, then this is the right tutorial for you. You’ll also benefit from learning this tool kit if you’ve scraped sites before, but the complexity of your project has outgrown using Beautiful Soup and Requests.
To get the most out of this tutorial, you should have basic Python programming knowledge, understand object-oriented programming, comfortably work with third-party packages, and be familiar with HTML and CSS.
By the end, you’ll know how to get, parse, and store static data from the Internet, and you’ll be familiar with several useful tools that allow you to go much deeper.
Get Your Code: Click here to download the free code that shows you how to gather Web data with Scrapy and MongoDB.
Take the Quiz: Test your knowledge with our interactive “Web Scraping With Scrapy and MongoDB” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Web Scraping With Scrapy and MongoDBIn this quiz, you'll test your understanding of web scraping with Scrapy and MongoDB. You'll revisit how to set up a Scrapy project, build a functional web scraper, extract data from websites, store scraped data in MongoDB, and test and debug your Scrapy web scraper.
Prepare the Scraper ScaffoldingYou’ll start by setting up the necessary tools and creating a basic project structure that will serve as the backbone for your scraping tasks.
While working through the tutorial, you’ll build a complete web scraping project, approaching it as an ETL (Extract, Transform, Load) process:
- Extract data from the website using a Scrapy spider as your web crawler.
- Transform this data, for example by cleaning or validating it, using an item pipeline.
- Load the transformed data into a storage system like MongoDB with an item pipeline.
Scrapy provides scaffolding for all of these processes, and you’ll tap into that scaffolding to learn web scraping following the robust structure that Scrapy provides and that numerous enterprise-scale web scraping projects rely on.
Note: In a Scrapy web scraping project, a spider is a Python class that defines how to crawl a specific website or a group of websites. It contains the logic for making requests, parsing responses, and extracting the desired data.
First, you’ll install Scrapy and create a new Scrapy project, then explore the auto-generated project structure to ensure that you’re well-equipped to proceed with building a performant web scraper.
Install the Scrapy PackageTo get started with Scrapy, you first need to install it using pip. Create and activate a virtual environment to keep the installation separate from your global Python installation. Then, you can install Scrapy:
Shell (venv) $ python -m pip install scrapy Copied!After the installation is complete, you can verify it by running the scrapy command and viewing the output:
Shell (venv) $ scrapy Scrapy 2.11.2 - no active project Usage: scrapy <command> [options] [args] Available commands: bench Run quick benchmark test fetch Fetch a URL using the Scrapy downloader genspider Generate new spider using pre-defined templates runspider Run a self-contained spider (without creating a project) settings Get settings values shell Interactive scraping console startproject Create new project version Print Scrapy version view Open URL in browser, as seen by Scrapy [ more ] More commands available when run from project directory Use "scrapy <command> -h" to see more info about a command Copied!The command-line (CLI) program should display the help text of Scrapy. This confirms that you installed the package correctly. You’ll next run the highlighted startproject command to create a project.
Create a Scrapy ProjectScrapy is built around projects. Generally, you’ll create a new project for each web scraping project that you’re working on. In this tutorial, you’ll work on scraping a website called Books to Scrape, so you can call your project books.
As you may have already identified in the help text, the framework provides a command to create a new project:
Shell (venv) $ scrapy startproject books Copied! Read the full article at https://realpython.com/web-scraping-with-scrapy-and-mongodb/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Django Weblog: Could you host DjangoCon Europe 2026? Call for organizers
We are looking for the next group of organizers to own and lead the 2026 DjangoCon Europe conference. Could your town - or your football stadium, circus tent, private island or city hall - host this wonderful community event?
DjangoCon Europe is a major pillar of the Django community, as people from across the world meet and share. This includes many qualities that make it a unique event - unconventional and conventional venues, creative happenings, a feast of talks and a dedication to inclusion and diversity.
Hosting a DjangoCon is an ambitious undertaking. It's hard work, but each year it has been successfully run by a team of community volunteers, not all of whom have had previous experience - more important is enthusiasm, organizational skills, the ability to plan and manage budgets, time and people - and plenty of time to invest in the project.
For 2026, we want to kickstart the organization much earlier than in previous years to allow more flexibility for the organizing team, and open up more opportunities for support from our DjangoCon Europe support working group.
Step 1: Submit your expression of interestIf you’re considering organizing DjangoCon Europe (🙌 great!), fill in our DjangoCon Europe 2026 expression of interest form with your contact details. No need to fill in all the information at this stage if you don’t have it all already, we’ll reach out and help you figure it out.
Express your interest in organizing
Step 2: We’re here to help!We've set up a DjangoCon Europe support working group of previous organizers that you can reach out to with questions about organizing and running a DjangoCon Europe.
The group will be in touch with everyone submitting the expression of interest form, or you can reach out to them directly: european-organizers-support@djangoproject.com
We'd love to hear from you as soon as possible, so your proposal can be finalized and sent to the DSF board by October 6th 2024. The selected hosts will be publicly announced at DjangoCon Europe 2025 by the current organizers.
Step 3: Submitting the proposalThe more detailed and complete your final proposal is, the better. Basic details include:
- Organizing committee members: You won’t have a full team yet, probably, naming just some core team members is enough.
- The legal entity that is intended to run the conference: Even if the entity does not exist yet, please share how you are planning to set it up.
- Dates: See “What dates are possible in 2026?” below. We must avoid conflicts with major holidays, EuroPython, DjangoCon US, and PyCon US.
- Venue(s), including size, number of possible attendees, pictures, accessibility concerns, catering, etc.
- Transport links and accommodation: Can your venue be reached by international travelers?
- Budgets and ticket prices: Talk to the DjangoCon Europe Support group to get help with this, including information on past event budgets.
We also like to see:
- Timelines
- Pictures
- Plans for online participation, and other ways to make the event more inclusive and reduce its environmental footprint
- Draft agreements with providers
- Alternatives you have considered
Have a look at our proposed DjangoCon Europe 2026 Licensing Agreement for the fine print on contractual requirements and involvement of the Django Software Foundation.
Submit your completed proposal by October 6th 2024 via our DjangoCon Europe 2026 expression of interest form, this time filling in as many fields as possible. We look forward to reviewing great proposals that continue the excellence the whole community associates with DjangoCon Europe.
Q&A Can I organize a conference alone?We strongly recommend that a team of people submit an application.
I/we don’t have a legal entity yet, is that a problem?Depending on your jurisdiction, this is usually not a problem. But please share your plans about the entity you will use or form in your application.
Do I/we need experience with organizing conferences?The support group is here to help you succeed. From experience, we know that many core groups of 2-3 people have been able to run a DjangoCon with guidance from previous organizers and help from volunteers.
What is required in order to announce an event?Ultimately, a contract with the venue confirming the dates is crucial, since announcing a conference makes people book calendars, holidays, buy transportation and accommodation etc. This, however, would only be relevant after the DSF board has concluded the application process. Naturally, the application itself cannot contain any guarantees, but it’s good to check concrete dates with your venues to ensure they are actually open and currently available, before suggesting these dates in the application.
Do we have to do everything ourselves?No. You will definitely be offered lots of help by the community. Typically, conference organizers will divide responsibilities into different teams, making it possible for more volunteers to join. Local organizers are free to choose which areas they want to invite the community to help out with, and a call will go out through a blog post announcement on djangoproject.com and social media.
What kind of support can we expect from the Django Software Foundation?The DSF regularly provides grant funding to DjangoCon organizers, to the extent of $6,000 in recent editions. We also offer support via specific working groups:
- The dedicated DjangoCon Europe support working group.
- The social media working group can help you promote the event.
- The Code of Conduct working group works with all event organizers.
In addition, a lot of Individual Members of the DSF regularly volunteer at community events. If your team aren’t Individual Members, we can reach out to them on your behalf to find volunteers.
What dates are possible in 2026?For 2026, DjangoCon Europe should happen between January 5th and April 27th, or June 4th and June 28th. This is to avoid the following community events’ provisional dates:
- PyCon US 2026: May 2026
- EuroPython 2026: July 2026
- DjangoCon US 2026: September - October 2026
- DjangoCon Africa 2026: August - September 2026
We also want to avoid the following holidays:
- New Year's Day: Wednesday 1st January 2026
- Chinese New Year: Tuesday 17th February 2026
- Eid Al-Fitr: Friday 20th March 2026
- Passover: Wednesday 1st - Thursday 9th April 2026
- Easter: Sunday 5th April 2026
- Eid Al-Adha: Tuesday 26th - Friday 29th May 2026
- Rosh Hashanah: Friday 11th - Sunday 13th September 2026
- Yom Kippur: Sunday 20th - Monday 21st September 2026
Any city in Europe. This can be a city or country where DjangoCon Europe has happened in the past (Vigo, Edinburgh, Porto, Copenhagen, Heidelberg, Florence, Budapest, Cardiff, Toulon, Warsaw, Zurich, Amsterdam, Berlin), or a new locale.
References Past callsPyPy: PyPy v7.3.17 release
The PyPy team is proud to release version 7.3.17 of PyPy.
This release includes a new RISC-V JIT backend, an improved REPL based on work by the CPython team, and better JIT optimizations of integer operations. Special shout-outs to Logan Chien for the RISC-V backend work, to Nico Rittinghaus for better integer optimization in the JIT, and the CPython team that has worked on the repl.
The release includes two different interpreters:
PyPy2.7, which is an interpreter supporting the syntax and the features of Python 2.7 including the stdlib for CPython 2.7.18+ (the + is for backported security updates)
PyPy3.10, which is an interpreter supporting the syntax and the features of Python 3.10, including the stdlib for CPython 3.10.14.
The interpreters are based on much the same codebase, thus the dual release. This is a micro release, all APIs are compatible with the other 7.3 releases. It follows after 7.3.16 release on April 23, 2024.
We recommend updating. You can find links to download the releases here:
https://pypy.org/download.html
We would like to thank our donors for the continued support of the PyPy project. If PyPy is not quite good enough for your needs, we are available for direct consulting work. If PyPy is helping you out, we would love to hear about it and encourage submissions to our blog via a pull request to https://github.com/pypy/pypy.org
We would also like to thank our contributors and encourage new people to join the project. PyPy has many layers and we need help with all of them: bug fixes, PyPy and RPython documentation improvements, or general help with making RPython's JIT even better.
If you are a python library maintainer and use C-extensions, please consider making a HPy / CFFI / cppyy version of your library that would be performant on PyPy. In any case, both cibuildwheel and the multibuild system support building wheels for PyPy.
RISC-V backend for the JITPyPy's JIT has added support for generating 64-bit RISC-V machine code at runtime (RV64-IMAD, specifically). So far we are not releasing binaries for any RISC-V platforms, but there are instructions on how to cross-compile binaries.
REPL ImprovementsThe biggest user-visible change of the release is new features in the repl of PyPy3.10. CPython 3.13 has adopted and extended PyPy's pure-Python repl, adding a number of features and fixing a number or bugs in the process. We have backported and added the following features:
Prompts and tracebacks use terminal colors, as well as terminal hyperlinks for file names.
Bracketed paste enable pasting several lines of input into the terminal without auto-indentation getting in the way.
A special interactive help browser (F1), history browser (F2), explicit paste mode (F3).
Support for Ctrl-<left/right> to jump over whole words at a time.
See the CPython documentation for further details. Thanks to Łukasz Langa, Pablo Galindo Salgado and the other CPython devs involved in this work.
Better JIT optimizations of integer operationsThe optimizers of PyPy's JIT have become much better at reasoning about and optimizing integer operations. This is done with a new "knownbits" abstract domain. In many programs that do bit-manipulation of integers, some of the bits of the integer variables of the program can be statically known. Here's a simple example:
x = a | 1 ... if x & 1: ... else: ...With the new abstract domain, the JIT can optimize the if-condition to True, because it already knows that the lowest bit of x must be set. This optimization applies to all Python-integers that fit into a machine word (PyPy optimistically picks between two different representations for int, depending on the size of the value). Unfortunately there is very little impact of this change on almost all Python code, because intensive bit-manipulation is rare in Python. However, the change leads to significant performance improvements in Pydrofoil (the RPython-based RISC-V/ARM emulators that are automatically generated from high-level Sail specifications of the respective ISAs, and that use the RPython JIT to improve performance).
PyPy versions and speed.pypy.orgThe keen-eyed will have noticed no mention of Python version 3.9 in the releases above. Typically we will maintain only one version of Python3, but due to PyPy3.9 support on conda-forge we maintained multiple versions from the first release of PyPy3.10 in PyPy v7.3.12 (Dec 2022). Conda-forge is sunsetting its PyPy support, which means we can drop PyPy3.9. Since that was the major driver of benchmarks at https://speed.pypy.org, we revamped the site to showcase PyPy3.9, PyPy3.10, and various versions of cpython on the home page. For historical reasons, the "baseline" for comparison is still cpython 3.7.19.
We will keep the buildbots building PyPY3.9 until the end of August, these builds will still be available on the nightly builds tab of the buildbot.
What is PyPy?PyPy is a Python interpreter, a drop-in replacement for CPython It's fast (PyPy and CPython performance comparison) due to its integrated tracing JIT compiler.
We also welcome developers of other dynamic languages to see what RPython can do for them.
We provide binary builds for:
x86 machines on most common operating systems (Linux 32/64 bits, Mac OS 64 bits, Windows 64 bits)
64-bit ARM machines running Linux (aarch64) and macos (macos_arm64).
PyPy supports Windows 32-bit, Linux PPC64 big- and little-endian, Linux ARM 32 bit, RISC-V RV64IMAFD Linux, and s390x Linux but does not release binaries. Please reach out to us if you wish to sponsor binary releases for those platforms. Downstream packagers provide binary builds for debian, Fedora, conda, OpenBSD, FreeBSD, Gentoo, and more.
What else is new?For more information about the 7.3.17 release, see the full changelog.
Please update, and continue to help us make pypy better.
Cheers, The PyPy Team
Python Software Foundation: Ask questions or tell us what you think: Introducing monthly PSF Board Office Hours!
Greetings, Pythonistas- thank you so much for supporting the work of the Python Software Foundation (PSF) and the Python community! The current PSF Board has decided to invest more in connecting and serving the global Python community by establishing a forum to have regular conversations. The board members of the PSF with the support of PSF staff are excited to introduce monthly PSF Board Office Hours on the PSF Discord. The Office Hours will be sessions where you can share with us how we can help your community, express your perspectives, and provide feedback for the PSF.
Similar to the PSF Grants Program Office Hours where PSF staff members help to answer questions regarding the PSF Grants Program, during the PSF Board Office Hours you can participate in a text-based live chat with PSF Board Directors. This is a chance to connect, share, and collaborate with the PSF Board and staff to improve our community together. Occasionally, we will have dedicated topics such as PyCon US and the PSF Board Elections for the office hour sessions.
Here is some of the work that we collaborate with staff and volunteers on:
- Promotion and outreach for the Python programming language
- Supporting local Python communities
- Organizing PyCon US
- Diversity and Inclusion in our community
- Support handling of Code of Conduct within our communities
- Support regional Python communities via the PSF Grants Program
- Furthering the mission of the PSF
Unless we have a dedicated topic for a session, you are not limited to talking with us about the above topics, although the discussions should be focused on Python, the PSF, and our community. If you think there’s something we can help with or we should know, we welcome you to come and talk to us!
The office hour sessions will take place on the PSF Discord server in the #psf-board channel. If you are new to Discord, make sure to check out a tutorial on how you can download the Discord app and sign up for free– then join us on the PSF Discord! To make the office hours more accessible, the office hours will be scheduled at alternating times so no matter where you are based, you can find a time that is most convenient for you! Here is a list of the dates and times:
- September 10th, 2024: 1pm UTC
- October 8th, 2024: 9pm UTC
- November 12th, 2024: 2pm UTC
- December 10th, 2024: 9pm UTC
- January 14th, 2025: 2pm UTC
- February 11th, 2025: 9pm UTC
- March 11th, 2025: 1pm UTC
- April 8th, 2025: 9pm UTC
- May 13th, 2025: 1pm UTC (Live from PyCon US!)
- June 10th, 2025: 9pm UTC
- July 9th, 2025: 1pm UTC
- August 12th, 2025: 9pm UTC
Each session lasts for an hour. Make sure to check what time these sessions are for you locally so you don't miss out! Sessions after August 13th, 2025, will be announced in the future.
Some of the board members of the PSF will be attending each office hour, as well as members of the PSF Staff. The list of the PSF Board Directors can be found on our website. We are passionate Python community members who are happy to listen, help, and provide support to you. We are happy to follow up with you if there are any issues we cannot address immediately during the office hour sessions. As always, you can email us at psf-board@python.org with inquiries, feedback, or comments at any time.
PyCoder’s Weekly: Issue #644 (Aug. 27, 2024)
#644 – AUGUST 27, 2024
View in Browser »
This course uses three problems often covered in introductory astro-physics courses to play in Python. Along the way you’ll learn some astronomy and how to use a variety of datascience libraries like NumPy, Matplotlib, pandas, and pint.
REAL PYTHON course
Packaging in Python has a bit of a history and if you’ve come across the variety of ways of specifying and building packages you might wonder “why?” This article gives you the history and current best practices.
BITECODE
Discover how to create, accelerate, and deploy data pipelines with RAPIDS for GPU-accelerated data science workflows. Take this course for free when you join the NVIDIA Developer Program →
NVIDIA sponsor
This guide walks you through how to build a custom query language in Python. The example given is a language to search through song lyrics.
JAMES G
In this step-by-step project, you’ll build a blog from the ground up. You’ll turn your Django blog data models into a GraphQL API and consume it in a Vue application for users to read. You’ll end up with an admin site and a user-facing site you can continue to refine for your own use.
REAL PYTHON
Are you interested in learning robotics with Python? Can physical electronics-based projects grow a child’s interest in coding? This week on the show, we speak with author Marwan Alsabbagh about his book “Build Your Own Robot - Using Python, CRICKIT, and Raspberry Pi.”
REAL PYTHON podcast
As a user of pre-commit hooks, do you know what happens when you run pre-commit install or why you have to run it in the first place? How does pre-commit actually work with Git? In this article, Stefanie takes you behind the scenes of how your pre-commit setup works.
STEFANIEMOLIN.COM • Shared by Stefanie Molin
Master the Python range() function and learn how it works under the hood. You most commonly use ranges in loops. In this tutorial, you’ll learn how to iterate over ranges but also identify when there are better alternatives.
REAL PYTHON
The author was asked what he would considered if he wrote a in-memory cache. This article talks about the eight principles he would use, many of which he wouldn’t have considered when a younger developer.
TWO WRONGS
Every now and then you hear outrageous claims such as “Python has no preprocessor”, well it is there if you’re willing to dig deep enough. Learn how to hack Python’s compile step.
PYDONG
Work continues on removing and/or optimizing the GIL in Python. This article gives a little history so you can better understand why the GIL is there and what changes are coming.
IZZY MUERTE
“Python continues to cement its overall dominance, buoyed by things like popular libraries for hot fields such as A.I.” Read the article to see where other languages have placed.
IEEE SPECTRUM
Optimization should be your last step, but once you’re there, just what can you do? This article covers ten different techniques that address memory size and code performance.
JAMES ONONIWU
A new release of of uv is out and it has added a lot of features. This post talks about what is new and how it can simplify your packaging process.
SIMON WILLISON
“PyPI has drastically improved its malware response times, resolving 90% of issues in under 24 hours and removing 900 projects since March 2024.”
SARAH GOODING
GITHUB.COM/BEN-N93 • Shared by Ben Nour
deltadb: A Lightweight Database Built on Polars and Deltalake django-public-admin: A Public and Read-Only Django Admin sqlfluff: A Modular SQL Linter and Auto-Formatter authentik: The Authentication Glue You Need Events Weekly Real Python Office Hours Q&A (Virtual) August 28, 2024
REALPYTHON.COM
August 29 to September 1, 2024
PYCON.ORG
August 29, 2024
MEETUP.COM
August 31, 2024
PYTHON.ORG.BR
September 2, 2024
J.MP
September 4 to September 6, 2024
DATACOVE.CO.UK
September 5 to September 7, 2024
PYCON.EE
Happy Pythoning!
This was PyCoder’s Weekly Issue #644.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]
James Bennett: There can't be only one
There’s a concept that I’ve heard called by a lot of different names, but my favorite name for it is “the Highlander problem”, which refers to the catchphrase of the campy-yet-still-quite-fun Highlander movie/TV franchise. In Highlander, immortal beings secretly live amongst us and sword-fight each other in hopes of being the last one standing, who will then get to rule the world forever. And when one of them is about to eliminate another, …
Real Python: Using Astropy for Astronomy With Python
This course covers two problems from introductory astronomy to help you play with some Python libraries. You’ll use Astropy, NumPy, Matplotlib, and pandas to find planet conjunctions, and graph the best viewing times for a star.
In this course you’ll learn about:
- Astronomy concepts of conjunction and optimal viewing
- The Python package Astropy
- Using pandas to process data
- Building graphs with Matplotlib
- Python’s warning module
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Armin Ronacher: MiniJinja: Learnings from Building a Template Engine in Rust
Given that I can't stop creating template engines, I figured I might write a bit about my learnings of creating MiniJinja which is an implementation of my Jinja2 template engine for Rust. Disclaimer: this post might be a bit more technical.
There is a good chance you have come across Jinja2 templates before as they became quite common place in various places over the years. They look a bit like this:
{% extends "layout.html" %} {% block body %} <p>Hello {{ name }}!</p> {% endblock %}If you want to play around it yourself, here are some links:
- The MiniJinja playground lets you play with a WASM compiled version of MiniJinja.
- The API Documentation documents all APIs, functionality and syntax.
- The GitHub Project for all the code including lots of examples.
- minijinja and minijinja-cli on crates.io
Maybe we start with the initial question of why I wrote MiniJinja. It's the year 2024 and people don't create a ton of HTML with server side rendered template engines any more. While there is some resurgence of that model thanks to HTMX, hotwire and livewire, I personally use SolidJS for my internal UI needs. There is however always a need to generate some form of text and so somehow Jinja2's need never really went away. When I originally created it, it was clearly meant for generating HTML with some JavaScript sprinkled on top, but in the years since I have encountered Jinja templates in many more places, primarily for generating YAML and similar formats. Lately it comes up for LLM prompt generation.
My personal need for MiniJinja came out of an experiment I built for infrastructure automation. Since the templates had to be loaded dynamically I could not use a system like Askama. Askama has type-safe templates that just generate Rust code. On the other hand most Jinja inspired template engines that are dynamic in Rust really do not try very hard to be Jinja compatible. Because writing template engines is also fun, I figured I might give it another try.
Over the last two years I kept adding to the engine until it got to the point where it's at almost feature parity with Jinja2 and quite enjoyable to use.
Runtime ValuesWhen building a template engine for Rust you end up building a little dynamic programming language that is optimized for text generation. Consequently you pull in most of the challenges of building a dynamic language. Particularly when working in Rust the immediate challenge is memory management and exposing native Rust objects to the embedded language. So the interesting bit here is how to create a system that allows interactions between the template engine and the Rust world around it.
MiniJinja, unlike Jinja2 does not use code generation but has a basic stack based VM and a AST based bytecode compiler. Since MiniJinja follows Jinja2 it inherits a lot of the realities of the underlying object system that Jinja2 inherits from Python. For instance macros (functions) are first class objects and they can have closures. This has challenges because it's easy to create cycles and Rust has no garbage collector that can help with this problem.
The core object model in MiniJinja is a Value type which is represented by an enum that looks as follows (some less important variants removed):
#[derive(Clone)] pub struct Value(ValueRepr); #[derive(Clone)] pub(crate) enum ValueRepr { Undefined, None, Bool(bool), U64(u64), I64(i64), F64(f64), String(Arc<str>, StringType), SmallStr(SmallStr), Invalid(Arc<Error>), Object(DynObject), }Externaly everything is a Value. If you Clone it, you usually bump a reference count or you make a cheap memcopy. Values are either primitives such as strings, numbers etc. or objects.
For objects MiniJinja provides a tait called Object which can be implemented by most Rust types. The engine provides a DynObject wrapper is a fancy Arc<dyn Object> which supports borrowing and object safety. I wrote about this before. What you will notice is that quite a few of the types involved have an Arc. That's because these values are for the most part reference counted. Since values here are really fat (they are 24 bytes in memory) a SmallStr type is used to hold up to 22 bytes of string data inline. One byte is used to encode the length of the string, and another byte is then used by the ValueRepr to mark which enum variant is in use. In pure theory this is all wrong. We never use weak references, so the weak count in the Arc is not used and clever bit hackery could be used to greatly reduce the size of the value type. I think one could get the whole thing down to 16 bytes trivially or even 8 bytes with NaN tagging. However I did not want to walk into the world of unsafe code more than feels appropriate.
MiniJinjia is also plenty fast.
One variant that is worth calling out is Invalid. That's a value that can exist in the system but it carries an error. When you're trying to interact with it in most cases it will propagate this error. That's used in the engine in places where the API assumes infallability (particularly during iteration) but it needs a way to emit an error. This concept is quite common when writing an engine in C though typically the actual error is carried out of bounds. For instance in QuickJS there is a marker value that indicates a failure, but the actual error is held on the interpreter runtime.
The trait definition for objects looks like this:
pub trait Object: Debug + Send + Sync { fn repr(self: &Arc<Self>) -> ObjectRepr { ... } fn get_value(self: &Arc<Self>, key: &Value) -> Option<Value> { ... } fn enumerate(self: &Arc<Self>) -> Enumerator { ... } fn enumerator_len(self: &Arc<Self>) -> Option<usize> { ... } fn is_true(self: &Arc<Self>) -> bool { ... } fn call( self: &Arc<Self>, state: &State<'_, '_>, args: &[Value], ) -> Result<Value, Error> { ... } fn call_method( self: &Arc<Self>, state: &State<'_, '_>, method: &str, args: &[Value], ) -> Result<Value, Error> { ... } fn render(self: &Arc<Self>, f: &mut Formatter<'_>) -> Result where Self: Sized + 'static { ... } }Some of these methods are implemented automatically. For instance many of the methods such as is_true or enumerator_len have a default implementation that is based on object repr and the return value from enumerate. But they can be overridden to change the default behavior or to add some potential optimizations.
One of the most important types in Jinja is a map as it holds the template context. They are implemented as you can imagine as Object. The implementation is in fact pretty trivial:
impl<V> Object for BTreeMap<Value, V> where V: Into<Value> + Clone + Send + Sync + fmt::Debug + 'static, { fn get_value(self: &Arc<Self>, key: &Value) -> Option<Value> { self.get(key).cloned().map(|v| v.into()) } fn enumerate(self: &Arc<Self>) -> Enumerator { self.mapped_enumerator(|this| Box::new(this.keys().cloned())) } }This reveals two interesting aspects of the object model: First that Value implements Hash. That means any value can be used as the key in a value. While this is untypical for Rust and even not what happens in Python, it simplifies the system greatly. When in the template engine you write {{ object.key }}, behind the scenes object.get_value(Value::from("key")) is called. Since most keys are typically less than 22 characters, creating a dummy Value wrapper around is not too problematic.
The second and probably more interesting part here is that you can sort of borrow out of an object for the enumerator. The mapped_enumerator helper takes a reference to self and invokes a closure which itself can borrow from self. This adjacent borrowing is implemented with unsafe code as there is no other way to make it work. The combination of repr (defaults to Map), get_value and enumerate gives the object the behavior, shape and contents.
Vectors look quite similar:
impl<T> Object for Vec<T> where T: Into<Value> + Clone + Send + Sync + fmt::Debug + 'static, { fn repr(self: &Arc<Self>) -> ObjectRepr { ObjectRepr::Seq } fn get_value(self: &Arc<Self>, key: &Value) -> Option<Value> { self.get(key.as_usize()?).cloned().map(|v| v.into()) } fn enumerate(self: &Arc<Self>) -> Enumerator { Enumerator::Seq(self.len()) } } Enumerators and Object BehaviorsEnumeration in MiniJinja is a way to allow an object to describe what's inside of it. In combination with the return values from repr() the engine changes how iteration is performed. These are possible enumerators:
pub enum Enumerator { NonEnumerable, Empty, Iter(Box<dyn Iterator<Item = Value> + Send + Sync>), Seq(usize), Values(Vec<Value>), }It's probably easier to explain how enumerators turn into iterators by showing you the try_iter method in the engine:
impl DynObject { fn try_iter(self: &Self) -> Option<Box<dyn Iterator<Item = Value> + Send + Sync>> where Self: 'static, { match self.enumerate() { Enumerator::NonEnumerable => None, Enumerator::Empty => Some(Box::new(None::<Value>.into_iter())), Enumerator::Seq(l) => { let self_clone = self.clone(); Some(Box::new((0..l).map(move |idx| { self_clone.get_value(&Value::from(idx)).unwrap_or_default() }))) } Enumerator::Iter(iter) => Some(iter), Enumerator::Values(v) => Some(Box::new(v.into_iter())), } } }Some of the trivial enumerators are quick to explain: Enumerator::NonEnumerable just does not support iteration and Enumerator::Empty does but won't yield any values. The more interesting one is Enumerator::Seq(n) which basically tells the engine to call get_value from 0 to n to yield items from the object. This is how sequences are implemented. The rest are enumerators that just directly yield values.
So when you want to iterate over a map, you will usually use something like Enumerator::Iter and iterate over all the keys in the map.
The engine then uses ObjectRepr to figure out what to do with it. For a value marked as ObjectRepr::Seq it will display like a sequence, you can index it with integers, and that it iterates over the values in the sequence. If the repr is ObejctRepr::Map then the expectation is that it will be indexable by key and it will iterate over the keys when used in a loop. Its default rendering also is a key-value pair list wrapped in curly braces.
Now quite frankly I don't like that iteration protocol. I think it's more sensible for maps to naturally iterate over the key-value pairs, but since MiniJinja follows Jinja2 and Jinja2 follows Python emulating was important.
Enumerators are a bit different than iterators because they might only define how iteration is performed (see: Enumerator::Seq). To actually create an iterator, the object is then passed to it. They are also asked to provide a length. When an enumerator provides a length it's an indication to the engine that the object can be iterated over more than once (you can re-create the enumerator). This is why objects land in a MiniJinja template that looks like a list, but is actually just an iterable object with a known length. For this MiniJinja uses a trick where it will inspect the size hint of the iterator to make assumptions about it. Internally every enumerator allows the engine to query the length of it:
impl Enumerator { fn query_len(&self) -> Option<usize> { Some(match self { Enumerator::Empty => 0, Enumerator::Values(v) => v.len(), Enumerator::Iter(i) => match i.size_hint() { (a, Some(b)) if a == b => a, _ => return None, }, Enumerator::RevIter(i) => match i.size_hint() { (a, Some(b)) if a == b => a, _ => return None, }, Enumerator::Seq(v) => *v, Enumerator::NonEnumerable => return None, }) } }The important part here is the call to size_hint. If the upper bound is known, and the lower bound matches the upper bound then MiniJinja will assume the iterator will always have that length (for as long as not iterated). As a result it will change the way the object is interacted with. This for instance means that if you run range(10) in a template it looks like a list when printed even though iteration and number creation is lazy. On the other hand if you use the Value::make_one_shot_iterator API the length hint will always be disabled and MiniJinja will not attempt to interact with the iterator when printing it:
{{ range(4) }} -> prints [0, 1, 2, 3] {{ a_real_iterator }} -> prints <iterator> Building a VMLexing and parsing I think is not too puzzling in Rust, but making an AST and making a VM is kinda unusual. The first thing is that Rust is just not particularly amazing at tree structures. In MiniJinja I really wanted to avoid having the AST at all, but it does come in in handy to implement some of the functionality that Jinja2 requires. For instance to establish closures it will just walk the AST to figure out which names are looked up within a function. I tried a few things to improve how memory allocations work with the AST. There are great crates out there for doing this, but I really wanted MiniJinja to be light on dependencies so I ended up opting against all of them.
For the AST design I went with large enums that hold Spanned<T> values:
pub enum Expr<'a> { Var(Spanned<Var<'a>>), Const(Spanned<Const>), ... } pub struct Var<'a> { pub id: &'a str, } pub struct Const { pub value: Value, }You might now be curious what Spanned<T> is. It's a wrapper type that does two things: it boxes the inner node and it stores and adjacent Span which is basically the code location in the original input template for debugging:
pub struct Spanned<T> { node: Box<T>, span: Span, }It implements Deref like a smart pointer so you can poke right through it to interact with the node. The code generator just walks the AST and emits instructions for it.
The instructions themselves are a large enum but the number of arguments to the variants is kept rather low to not waste too much memory. The base size of the instruction is dominated by it being able to hold a Value which as we have established is a pretty hefty thing:
pub enum Instruction<'source> { EmitRaw(&'source str), StoreLocal(&'source str), Lookup(&'source str), LoadConst(Value), Jump(usize), JumpIfFalse(usize), JumpIfFalseOrPop(usize), JumpIfTrueOrPop(usize), ... }The VM keeps most of the runtime state on a State object that is passed to a few places. For instance you have already seen this in the call signature further up. The state for instance holds the loaded instructions or the template context. The VM itself maintains a stack of values and then just steps through a list of instructions on the state in a loop. Since there are a lot of instructions you can have a look on GitHub to see it in its entirety. Here however is a small part that shows roughly how this works:
let mut pc = 0; loop { let instr = state.instructions.get(pc) { Some(instr) => instr, None => break, }; let a; let b; match instr { Instruction::EmitRaw(val) => { out.write_str(val).map_err(Error::from)?; } Instruction::Emit => { self.env.format(&stack.pop(), state, out)?; } Instruction::StoreLocal(name) => { state.ctx.store(name, stack.pop()); } Instruction::Lookup(name) => { stack.push(assert_valid!(state .lookup(name) .unwrap_or(Value::UNDEFINED))); } Instruction::GetAttr(name) => { a = stack.pop(); stack.push(match a.get_attr_fast(name) { Some(value) => value, None => undefined_behavior.handle_undefined(a.is_undefined())?, }); } Instruction::LoadConst(value) => { stack.push(value.clone()); } Instruction::Jump(jump_target) => { pc = *jump_target; continue; } Instruction::JumpIfFalse(jump_target) => { a = stack.pop(); if !undefined_behavior.is_true(&a)? { pc = *jump_target; continue; } } // ... } pc += 1; }Basically the current instruction is held in pc (short for program counter), normally it's advanced by one but jump instructions can change the pc to any other location. If you run out of instructions the evaluation ends.
One piece of complexity in the VM comes down to macros. That's because lifetimes make that really tricky. A macro is just a Value that holds a Macro Object internally. So how can that macro reference the instructions, if the instructions themselves have a lifetime to the template 'source? The answer is that they can't (at least I have not found a reasonable way). So instead a macro has an ID which acts as a handle to look up the instructions dynamically from the execution state. Additionally each state has a unique ID so the engine can assert that nothing funny was happening. The downside of this is that a macro cannot be "returned" from a template. They can however be imported from one template into another.
Here is what a macro object looks like in code (abbreviated):
pub(crate) struct Macro { pub name: Value, pub arg_spec: Vec<Value>, pub macro_ref_id: usize, // id of the macro pub state_id: isize, pub closure: Value, pub caller_reference: bool, } impl Object for Macro { fn call(self: &Arc<Self>, state: &State<'_, '_>, args: &[Value]) -> Result<Value, Error> { // we can only call macros that point to loaded template state. // if a template would be returned from a template this will // fail. if state.id != self.state_id { return Err(Error::new( ErrorKind::InvalidOperation, "cannot call this macro. template state went away.", )); } // ... argument parsing let arg_values = ...; // find referenced instructions let (instructions, offset) = &state.macros[self.macro_ref_id]; // created a nested vm and evaluate the macro let vm = Vm::new(state.env()); let mut rv = String::new(); let mut out = Output::with_string(&mut rv); let closure = self.closure.clone(); ok!(vm.eval_macro( instructions, *offset, self.closure.clone(), state.ctx.clone_base(), caller, &mut out, state, arg_values )); // return rendered template as string from the call Ok(if !matches!(state.auto_escape(), AutoEscape::None) { Value::from_safe_string(rv) } else { Value::from(rv) }) } }Additionally the closure is a good source of cycles. For that reason the engine keeps track of all closures during the execution and breaks cycles caused by closures manually by clearning them out.
Cool APIsThe last part that I want to go over is the magic that makes this work:
fn slugify(value: String) -> String { value.to_lowercase().split_whitespace().collect::<Vec<_>>().join("-") } fn timeformat(state: &State, ts: f64) -> String { let configured_format = state.lookup("TIME_FORMAT"); let format = configured_format .as_ref() .and_then(|x| x.as_str()) .unwrap_or("HH:MM:SS"); format_unix_timestamp(ts, format) } let mut env = Environment::new(); env.add_filter("slugify", slugify); env.add_filter("timeformat", timeformat);You might have seem something like this in Rust before, but it's still a bit magical. How can you make functions with seemingly different signatures register with the add_filter function? How does the engine perform the type conversions (as we know the engine has Value types, so where does the String conversion take place?). This is a topic for a blog post on its own but the answer behind this lies in a a lot of clever trait hackery. The add_filter function reveals a bit of that hackery:
pub fn add_filter<N, F, Rv, Args>(&mut self, name: N, f: F) where N: Into<Cow<'source, str>>, F: Filter<Rv, Args> + for<'a> Filter<Rv, <Args as FunctionArgs<'a>>::Output>, Rv: FunctionResult, Args: for<'a> FunctionArgs<'a>, { let filter = BoxedFilter(Arc::new(move |state, args| -> Result<Value, Error> { f.apply_to(Args::from_values(Some(state), args)?).into_result() })); self.filters.insert(name.into(), filter); }Hidden behind this rather complex set of traits are some basic ideas:
- FunctionArgs is a helper trait for type conversions. It's implemented for tuples of different sizes made of ArgType values. These tuples represent the signature of the function. It has a method called from_values which performs that conversion via ArgType.
- ArgType which you can't really see in the code above, is a trait that knows how to convert a Value into whatever the function desires as argument.
- Filter is a trait implemented for function with qualifying FunctionArgs signatures returning a FunctionResult.
- A FunctionResult is a trait that represents potential return values from the function such as a Value, something that can be converted into a Value or a Result.
- The BoxedFilter type is what converts the passed closure into a reference counted object that is held in the environment.
I think a lot of the patterns in MiniJinja are useful for projects outside of MiniJinja. Quite is quite a bit more hidden in it that I have talked about before such as how MiniJinja is abusing serde. If you have a need for a Jinja2 compatible template engine I would love if you get some use out of it. If you're curious about how to build a runtime and object system in Rust, you might also find some utility in the codebase.
I myself learned quite a bit about what creative API design can look like in Rust by building it. At this point I am incredibly happy with how the public API of the engine shaped out to be. The engine is extensively documented both internally and publicly and you can read all about it in the API docs.
Real Python: How to Install Python on Your System: A Guide
Installing the latest version of Python on your computer could be a common requirement for you as a Python programmer. Fortunately, you’ll have a multitude of installation options. For example, you can download the official Python installer from Python.org, use your operating system’s package manager or app store, and more.
In this tutorial, you’ll focus on official CPython distributions, which are generally the best option for learning to program with the language. However, you’ll also learn about a few other distributions, like the one available on Homebrew for macOS users.
In this tutorial, you’ll learn how to:
- Check whether a version of Python is installed on your system
- Install or update to the latest Python on Windows, macOS, and Linux
- Install Python on mobile devices like phones or tablets
- Use Python on your browser with online interpreters
This tutorial covers installing the latest Python on the most important platforms or operating systems, such as Windows, macOS, Linux, iOS, and Android. However, it doesn’t cover all the existing Linux distributions, which would be a huge task. Anyway, you’ll find instructions for the most popular distros nowadays.
To get the most out of this tutorial, you should be comfortable using your operating system’s terminal or command line.
Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.
Take the Quiz: Test your knowledge with our interactive “Python Installation and Setup” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Python Installation and SetupIn this quiz, you'll test your understanding of how to install or update Python on your computer. With this knowledge, you'll be able to set up Python on various operating systems, including Windows, macOS, and Linux.
Windows: How to Check or Get PythonIn this section, you’ll learn to check whether Python is installed on your Windows operating system (OS) and which version you have. You’ll also explore three installation options that you can use on Windows.
Note: In this tutorial, you’ll focus on installing the latest version of Python in your current operating system (OS) rather than on installing multiple versions of Python. If you want to install several versions of Python in your OS, then check out the Managing Multiple Python Versions With pyenv tutorial. Note that on Windows machines, you’d have to use pyenv-win instead of pyenv.
For a more comprehensive guide on setting up a Windows machine for Python programming, check out Your Python Coding Environment on Windows: Setup Guide.
Checking the Python Version on WindowsTo check whether you already have Python on your Windows machine, open a command-line application like PowerShell or the Windows Terminal.
Follow the steps below to open PowerShell on Windows:
- Press the Win key.
- Type PowerShell.
- Press Enter.
Alternatively, you can right-click the Start button and select Windows PowerShell or Windows PowerShell (Admin). In some versions of Windows, you’ll find Terminal or Terminal (admin).
Note: To learn more about your options for the Windows terminal, check out Using the Terminal on Windows.
With the command line open, type in the following command and press the Enter key:
Windows PowerShell PS> python --version Python 3.x.z Copied!Using the --version switch will show you the installed version. Note that the 3.x.z part is a placeholder here. In your machine, x and z will be numbers corresponding to the specific version you have installed.
Alternatively, you can use the -V switch:
Windows PowerShell PS> python -V Python 3.x.z Copied!Using the python -V or python—-version command, you can check whether Python is installed on your system and learn what version you have. If Python isn’t installed on your OS, you’ll get an error message.
Knowing the Python Installation Options on Windows Read the full article at https://realpython.com/installing-python/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Ned Batchelder: Coverage branches instead of arcs
As I mentioned in a few recent posts, I’ve been working on some significant work in coverage.py to take advantage of new capabilities in Python.
Mark Shannon has been improving the sys.monitoring API so that branch coverage can be done with low overhead. I want to take advantage of that in coverage.py, but I needed to do some refactoring work first. The tests were focused on mapping the complete set of code pathways (which I called arcs), but using low-overhead branch monitoring won’t provide those complete pathways. If the tests continued to focus on them, they would fail with sys.monitoring.
But the complete pathways aren’t actually needed. The useful information is where the branches are, and which branches were taken. That can be measured with sys.monitoring. So a first step was to refactor the tests to focus on branches instead of arcs. That took a while, but is now done.
Not needing all those arcs also meant I could simplify the AST-based parser that found the arcs, removing about 150 lines. I suspect there’s more that could be removed. Maybe it will happen over time. Also, the new code.co_branches() method might make it all obsolete over time.
If you read Coverage at a crossroads on this blog, I talked about using ideas from SlipCover like inserting fake lines with an import hook. Those exotic ideas were appealing in their way, but are no longer needed, and they would have brought a bunch of complexity. With the two new sys.monitoring events, we can get the branch information directly without advanced shenanigans.
There’s more work to do, including attending to incoming bug reports. If you’d like to help, or learn more about any of this, we have a #coverage-py channel in the Python Discord.
Python Bytes: #398 Open source makes you rich? (and other myths)
Zato Blog: Integrating with Jira APIs
Continuing in the series of articles about newest cloud connections in Zato 3.2, this episode covers Atlassian Jira from the perspective of invoking its APIs to build integrations between Jira and other systems.
There are essentially two use modes of integrations with Jira:
- Jira reacts to events taking place in your projects and invokes your endpoints accordingly via WebHooks. In this case, it is Jira that explicitly establishes connections with and sends requests to your APIs.
- Jira projects are queried periodically or as a consequence of events triggered by Jira using means other than WebHooks.
The first case is usually more straightforward to conceptualize - you create a WebHook in Jira, point it to your endpoint and Jira invokes it when a situation of interest arises, e.g. a new ticket is opened or updated. I will talk about this variant of integrations with Jira in a future instalment as the current one is about the other situation, when it is your systems that establish connections with Jira.
The reason why it is more practical to first speak about the second form is that, even if WebHooks are somewhat easier to reason about, they do come with their own ramifications.
To start off, assuming that you use the cloud-based version of Jira (e.g. https://example.atlassian.net), you need to have a publicly available endpoint for Jira to invoke through WebHooks. Very often, this is undesirable because the systems that you need to integrate with may be internal ones, never meant to be exposed to public networks.
Secondly, your endpoints need to have a TLS certificate signed by a public Certificate Authority and they need to be accessible on port 443. Again, both of these are something that most enterprise systems will not allow at all or it may take months or years to process such a change internally across the various corporate departments involved.
Lastly, even if a WebHook can be used, it is not always a given that the initial information that you receive in the request from a WebHook will already contain everything that you need in your particular integration service. Thus, you will still need a way to issue requests to Jira to look up details of a particular object, such as tickets, in this way reducing WebHooks to the role of initial triggers of an interaction with Jira, e.g. a WebHook invokes your endpoint, you have a ticket ID on input and then you invoke Jira back anyway to obtain all the details that you actually need in your business integration.
The end situation is that, although WebHooks are a useful concept that I will write about in a future article, they may very well not be sufficient for many integration use cases. That is why I start with integration methods that are alternative to WebHooks.
Alternatives to WebHooksIf, in our case, we cannot use WebHooks then what next? Two good approaches are:
- Scheduled jobs
- Reacting to emails (via IMAP)
Scheduled jobs will let you periodically inquire with Jira about the changes that you have not processed yet. For instance, with a job definition as below:
Now, the service configured for this job will be invoked once per minute to carry out any integration works required. For instance, it can get a list of tickets since the last time it ran, process each of them as required in your business context and update a database with information about what has been just done - the database can be based on Redis, MongoDB, SQL or anything else.
Integrations built around scheduled jobs make most sense when you need to make periodic sweeps across a large swaths of business data, these are the "Give me everything that changed in the last period" kind of interactions when you do not know precisely how much data you are going to receive.
In the specific case of Jira tickets, though, an interesting alternative may be to combine scheduled jobs with IMAP connections:
The idea here is that when new tickets are opened, or when updates are made to existing ones, Jira will send out notifications to specific email addresses and we can take advantage of it.
For instance, you can tell Jira to CC or BCC an address such as zato@example.com. Now, Zato will still run a scheduled job but instead of connecting with Jira directly, that job will look up unread emails for it inbox ("UNSEEN" per the relevant RFC).
Anything that is unread must be new since the last iteration which means that we can process each such email from the inbox, in this way guaranteeing that we process only the latest updates, dispensing with the need for our own database of tickets already processed. We can extract the ticket ID or other details from the email, look up its details in Jira and the continue as needed.
All the details of how to work with IMAP emails are provided in the documentation but it would boil down to this:
# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class MyService(Service): def handle(self): conn = self.email.imap.get('My Jira Inbox').conn for msg_id, msg in conn.get(): # Process the message here .. process_message(msg.data) # .. and mark it as seen in IMAP. msg.mark_seen()The natural question is - how would the "process_message" function extract details of a ticket from an email?
There are several ways:
- Each email has a subject of a fixed form - "[JIRA] (ABC-123) Here goes description". In this case, ABC-123 is the ticket ID.
- Each email will contain a summary, such as the one below, which can also be parsed:
- Finally, each email will have an "X-Atl-Mail-Meta" header with interesting metadata that can also be parsed and extracted:
The first option is the most straightforward and likely the most convenient one - simply parse out the ticket ID and call Jira with that ID on input for all the other information about the ticket. How to do it exactly is presented in the next chapter.
Regardless of how we parse the emails, the important part is that we know that we invoke Jira only when there are new or updated tickets - otherwise there would not have been any new emails to process. Moreover, because it is our side that invokes Jira, we do not expose our internal system to the public network directly.
However, from the perspective of the overall security architecture, email is still part of the attack surface so we need to make sure that we read and parse emails with that in view. In other words, regardless of whether it is Jira invoking us or our reading emails from Jira, all the usual security precautions regarding API integrations and accepting input from external resources, all that still holds and needs to be part of the design of the integration workflow.
Creating Jira connectionsThe above presented the ways in which we can arrive at the step of when we invoke Jira and now we are ready to actually do it.
As with other types of connections, Jira connections are created in Zato Dashboard, as below. Note that you use the email address of a user on whose behalf you connect to Jira but the only other credential is that user's API token previously generated in Jira, not the user's password.
Invoking JiraWith a Jira connection in place, we can now create a Python API service. In this case, we accept a ticket ID on input (called "a key" in Jira) and we return a few details about the ticket to our caller.
This is the kind of a service that could be invoked from a service that is triggered by a scheduled job. That is, we would separate the tasks, one service would be responsible for opening IMAP inboxes and parsing emails and the one below would be responsible for communication with Jira.
Thanks to this loose coupling, we make everything much more reusable - that the services can be changed independently is but one part and the more important side is that, with such separation, both of them can be reused by future services as well, without tying them rigidly to this one integration alone.
# -*- coding: utf-8 -*- # stdlib from dataclasses import dataclass # Zato from zato.common.typing_ import cast_, dictnone from zato.server.service import Model, Service # ########################################################################### if 0: from zato.server.connection.jira_ import JiraClient # ########################################################################### @dataclass(init=False) class GetTicketDetailsRequest(Model): key: str @dataclass(init=False) class GetTicketDetailsResponse(Model): assigned_to: str = '' progress_info: dictnone = None # ########################################################################### class GetTicketDetails(Service): class SimpleIO: input = GetTicketDetailsRequest output = GetTicketDetailsResponse def handle(self): # This is our input data input = self.request.input # type: GetTicketDetailsRequest # .. create a reference to our connection definition .. jira = self.cloud.jira['My Jira Connection'] # .. obtain a client to Jira .. with jira.conn.client() as client: # Cast to enable code completion client = cast_('JiraClient', client) # Get details of a ticket (issue) from Jira ticket = client.get_issue(input.key) # Observe that ticket may be None (e.g. invalid key), hence this 'if' guard .. if ticket: # .. build a shortcut reference to all the fields in the ticket .. fields = ticket['fields'] # .. build our response object .. response = GetTicketDetailsResponse() response.assigned_to = fields['assignee']['emailAddress'] response.progress_info = fields['progress'] # .. and return the response to our caller. self.response.payload = response # ########################################################################### Creating a REST channel and testing itThe last remaining part is a REST channel to invoke our service through. We will provide the ticket ID (key) on input and the service will reply with what was found in Jira for that ticket.
We are now ready for the final step - we invoke the channel, which invokes the service which communicates with Jira, transforming the response from Jira to the output that we need:
$ curl localhost:17010/jira1 -d '{"key":"ABC-123"}' { "assigned_to":"zato@example.com", "progress_info": { "progress": 10, "total": 30 } } $And this is everything for today - just remember that this is just one way of integrating with Jira. The other one, using WebHooks, is something that I will go into in one of the future articles.
More resources➤ Python API integration tutorial
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
Matt Layman: Layman's Guide to Python Built-in Functions
Seth Michael Larson: 2024 Minnesota State Fair foods
Published 2024-08-25 by Seth Larson
Reading time: minutes
If you didn't know, I'm from Minnesota. Minnesotans love their State Fair, and I'm not an exception! My wife and I were lucky enough to go to a State Fair preview for LuLu's Public House for fried ranch dressing among a handful of new drinks. I shared my thoughts on Mastodon and a few folks seemed interested in hearing more: so here's more!
Cajun fried pickles from The Perfect PickleThese are hands-down the best food at the Minnesota State Fair. You eat an order, ponder getting more (some years we do!) and then wonder to yourself why they put the best of the best right next to the shuttle entrance. Don't go out looking for answers lest they move these further away, sometimes it's best to leave sleeping “pickle dogs” lie.
Seriously, if you like pickles even a little bit, get these pickles. You can get them quick if you're lucky and other folks don't realize there are supposed to be six lines of people taking orders.
They're ripping hot right when they hand them to you, so if you're like me and enjoy food “biting back” then don't delay! 🔥
This year included a noticeable increase in the amount of Cajun seasoning, or we got lucky and someone behind the scenes gave us an extra coating (either way we're not complaining!)
Peanut Butter Bacon Cakes and Blue Cheese & Corn Fritz from The Blue BarnCelebrating their 10th consecutive year at the Minnesota State Fair, The Blue Barn is always a fan favorite. Seriously, run over there if you get to the fair early to beat the massive lines for food and drinks.
We grabbed the new Peanut Butter Bacon Cakes along with the returning classic Blue Cheese & Corn Fritz which I had never tried before.
The Peanut Butter Bacon Cakes were really great, there was thick-cut bacon griddled inside of pancake batter strips along with jelly and a peanut butter whipped cream. Perfect combo of savory and sweet, and you're in complete control of the ratios. The bacon and pancake flavors reminded me of learning to make pancakes with my late grandfather. Although that bacon was microwave-ready Hormel bacon... I promise this one's delish!
The Blue Cheese & Corn Fritz was really great, I missed out on this one last year. Perfect amount of sweetness from the corn, really well-balanced cheesy little bite! Wish I could have had more than one of these, we were sharing amongst a big group!
Wrangler Waffle Burger, Bacon-Wrapped Pickle Dog, and “Kind of a Big Dill” Lemonade from Nordic WafflesAnother vendor that fills up immediately after opening, Nordic Waffle should be top of your list because of two returning new foods from 2023: the Bacon-Wrapped Pickle Dog and the Pickle Lemonade. Both of these are really great, the lemonade sounds strange but works really well (even if you don't love pickles). The subtle saltiness balances out the sweet and tartness which makes for a dangerously drinkable item.
The Wrangler was good, it's one of those winning combinations of flavors that is really hard to mess up: beef, cheese, caramelized onions, and a mayo-based sauce. The onions being grilled into the waffle was fun but didn't do much flavor-wise (they might as well have been a topping), honestly wish they went all-out on the onions to the point of being noticeable texture-wise in the waffle. The bacon-wrapped pickle dog is as awesome as it sounds, so much more interesting flavor-wise!
I'm also not a fan of their choice of sauce, they went with Whataburger, a famously mid-tier burger joint in Texas, of all places? This is a grave error by Nordic Waffles because Minnesota and Texas have serious State Fair beef. Minnesota seeing the highest single-day attendance over 12 days, where the Texas State Fair sees the highest total attendance over 24 days (it might be obvious which State Fair I think is the true champion).
Sweet Corn Cola Float from Blue Moon Dine-in TheaterThis one was interesting! Sweet Corn icecream and house-made “corn Cola”, so I take that to mean corn syrup Cola? Not sure. The flavor definitely gave a “not-too-sweet” vibe which was nice, there was a good amount of a corny and almost “earthy” flavor in the float.
The texture of the corn icecream was a little less smooth than a normal icecream, which landed somewhere between novel and “interesting”. I actually recommend giving this one a good mix before you drink it to blend the flavors together better, you're only given a boba straw to drink it.
Overall, would I get it again? Probably not, because Lift Bridge root beer floats exist and are much better. But worth a try!
Sweet Heat Bacon Crunch from RC's BBQHad this one side-by-side with my typical order from RC's which is a bunch of ribs and yeah, it was fine, but if I'm buying barbecue I want ribs or brisket. There was some chili crisp (but not much, maybe because it's Minnesota) and hot honey that got a bit lost in the dish. Can't recommend this one, RC's usual items are much better.
Spam breakfast sandwich from SPAMAttention all SPAM-lovers at the fair! The SPAM booth has moved from under the Grandstand bridge to the southern edge of the DNR building. I nearly had a heart-attack when I saw the SPAM booth wasn't in its usual spot, I had to sneak away with a fellow SPAM-lover from our group to snag this item.
We got ours with pickles (surprise!) and jalapeños, a little bit of kick and acid to cut through the lovely fatty grilled SPAM. Pretty sure this little sandwich was gone in 4 bites, highly recommend finding this stand if you're a long-time-enjoyer or first-timer of SPAM!
That's all for this year. At this point we kept trying new items, but I suspect not being hungry started to impact my opinions of the foods, so you'll have to try them yourself! :)
Thanks for reading! ♡ Did you find this article helpful and want more content like it?
Get notified of new posts by subscribing to the RSS feed or the email newsletter.
This work is licensed under CC BY-SA 4.0
Brian Okken: Finding the top pytest plugins
Pages
Recent Publications
- Managing Hidden Dependencies in OO Software: a Study Based on Open Source Projects
- Open Source Communities as Liminal Ecosystems
- Investigating developers' email discussions during decision-making in Python language evolution
- Developers, Quality Control and Download Volume in Open Source Software (OSS) Projects
More...FLOSS Project Planets
- Django Weblog: Django security releases issued: 5.1.4, 5.0.10, and 4.2.17
- Real Python: Expression vs Statement in Python: What's the Difference?
- Ned Batchelder: Testing some tidbits
- Django Weblog: Help us make it happen ❤️
- The Licensing and Compliance Team is fighting for freedom and we need your help
FLOSS Research
- Driving Open Source forward: make your impact in 2025
- Improving Open Source security with the new GitHub Secure Open Source Fund
- Celebrating 5 years at the Open Source Initiative: a journey of growth, challenges, and community engagement
- Highlights from the Digital Public Goods Alliance Annual Members Meeting 2024
- Give Your Input on the State of Open Source Survey