Planet Python

In this quiz, you’ll test your understanding of how to use Python’s built-in functions input() and print() for basic input and output operations.

You’ll also revisit how to use readline to improve the user experience when collecting input, and how to format output using the sep and end keyword arguments of print().

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Introduction

eGenix PyRun™ is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run Python based scripts and applications to Unix based systems simple and efficient.

eGenix PyRun's executable only needs 4-6MB on disk, but still supports most Python applications and scripts.

Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to containers, VMs, clusters, client installations, customers or end-users.

It makes "installing" Python on a Unix based system as simple as copying a single file.

eGenix has been using eGenix PyRun as run-time for the Linux version of mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product.

We provide the source archive to build your own eGenix PyRun on Github, as well as a few binary distributions to get you started on Linux x86_64. In the future, we will set up automated builds for several other platforms.

Please see the product page for more details:

    >>> eGenix PyRun - One file Python Runtime

News

This major release of eGenix PyRun comes with the following enhancements:

Enhancements / Changes

• Added support for Python 3.12
• Added support for LTO release builds
• Added dev build targets for development; these don't use PGO and thus build faster

For a complete list of changes, please see the eGenix PyRun Changelog. Downloads

Please visit the eGenix PyRun product page for downloads, instructions on installation and documentation of the product.

Support

Commercial support for this product is available directly from eGenix.com.

Please see the support section of our website for details.

More Information

For more information on eGenix PyRun, licensing and download instructions, please write to sales@egenix.com.

Enjoy !

Marc-Andre Lemburg, eGenix.com

Web scraping as an API service 2024-11-13, by Dariusz Suchojad Overview

In systems-to-systems integrations, there comes an inevitable time when we have to employ some kind of a web scraping tool to integrate with a particular application. Despite its not being our first choice, it is good to know what to use at such a time - in this article, I provide a gentle introduction to my favorite tool of this kind, called Playwright, followed by sample Python code that integrates it with an API service.

Naturally, in the context of backend integrations, web scraping should be avoided and, generally, it should be considered the last resort. The basic issue here is that while the UI term contains the "interface" part, it is not really the "Application Programming" Interface that we would like to have.

It is not that the UI cannot be programmed against. After all, a web browser does just that, it takes a web page and renders it as expected. Same goes for desktop or mobile applications. Also, anyone integrating with mainframe computers will recognize that this is basically what 3270 can be used for too.

Rather, the fundamental issue is that web scraping goes against the principles of separation of layers and roles across frontend, middleware and backend, which in turn means that authors of resources (e.g. HTML pages) do not really expect for many people to access them in automated ways.

Perhaps they actually should expect it, and web pages should finally start to resemble genuine knowledge graphs, easy to access by humans, be it manually or through automation tools, but the reality today is that it is not the case and, in comparison with backend systems, the whole of the web scraping space is relatively brittle, which is why we shun this approach in integrations.

Yet, another part of reality, particularly in enterprise integrations, is that people may be sometimes given access to a frontend application on an internal network and that is it. No API, no REST, no JSON, no POST data, no real data formats, and one is simply supposed to fill out forms as part of a business process.

Typically, such a situation will result in an integration gap. There will be fully automated parts in the business process preceding this gap, with multiple systems coordinated towards a specific goal and there will be subsequent steps in the process, also fully automated.

Or you may be given access only to a specific frontend and only through VPN via a single remote Windows desktop. Getting access to a REST API may take months or may be never realized because of some high level licensing issues. This is not uncommon in the real life.

Such a gap can be a jarring and sore point, truly ruining the whole, otherwise fluid, integration process. This creates a tension and to resolve the tension, we can, should all the attempts to find a real API fail, finally resort to web scraping.

It is mostly in this context that I am looking at Playwright below - the tool is good and it has many other uses that go beyond the scope of this text, and it is well worth knowing it, for instance for frontend testing of your backend systems, but, when we deal with API integrations, we should not overdo with web scraping.

Needless to say, if web scraping is what you do primarily, your perspective will be somewhat different - you will not need any explanation of why it is needed or when, and you may be only looking for a way to enclose up your web scraping code in API services. This article will explain that too.

Introducing Playwright

The nice part of Playwright is that we can use it to visually prepare a draft of Python code that will scrape a given resource. That is, instead of programming it in Python, we go to an address, fill out a form, click buttons and otherwise use everything as usually and Playwright generates for us code that will be later used in integrations.

That code will require a bit of clean-up work, which I will talk about below, but overall it works very nicely and is certainly useful. The result is not one of these do-not-touch auto-generated pieces of code that are better left to their own.

While there are better ways to integrate with Jira, I chose that application as an example of Playwright's usage simply because I cannot show you any internal application in a public blog post.

Below, there are two windows. One is Playwright's emulating a Blackberry device to open a resource. I was clicking around, I provided an email address and then I clicked the same email field once more. To the right, based on my actions, we can find the generated Python code, which I consider quite good and readable.

The Playwright Inspector, the tool that gave us the code, will keep recording all of our actions until we click the "Record" button which then allows us to click the button next to "Record" which is "Copy code to clipboard". We can then save the code to a separate file and run it on demand, automatically.

But first, we will need to install Playwright.

Installing and starting Playwright

The tools is written in TypeScript and can be installed using npx, which in turn is part of NodeJS.

Afterwards, the "playwright install" call is needed as well because that will potentially install runtime dependencies, such as Chrome libraries.

Finally, we install Playwright using pip as well because we want to access with Python. Note that if you are installing Playwright under Zato, the "/path/to/pip" will be typically "/opt/zato/code/bin/pip".

npx -g --yes playwright install playwright install /path/to/pip install playwright

We can now start it as below. I am using BlackBerry as an example of what Playwright is capable of. Also, it is usually more convenient to use a mobile version of a site when the main window and Inspector are opened side by side, but you may prefer to use Chrome, Firefox or anything else.

playwright codegen https://example.atlassian.net/jira --device "BlackBerry Z30"

That is practically everything as using Playwright to generate code in our context goes. Open the tool, fill out forms, copy code to a Python module, done.

What is still needed, though, is cleaning up the resulting code and embedding it in an API integration process.

Code clean-up

After you keep using Playwright for a while with longer forms and pages, you will note that the generated code tends to accumulate parts that repeat.

For instance, in the module below, which I already cleaned up, the same "[placeholder=\"Enter email\"]" reference to the email field is used twice, even if a programmer developing this could would prefer to introduce a variable for that.

There is not a good answer to the question of what to do about it. On the one hand, obviously, being programmers we would prefer not to repeat that kind of details. On the other hand, if we clean up the code too much, this may result in too much of a maintenance burden because we need to keep it mind that we do not really want to invest to much in web scraping and, should there be a need to repeat the whole process, we do not want to end up with Playwright's code auto-generated from scratch once more, without any of our clean-up.

A good compromise position is to at least extract any kind of credentials from the code to environment variables or a similar place and to remove some of the code comments that Playwright generates. The result as below is what it should like at the end. Not too much effort without leaving the whole code as it was originally either.

Save the code below as "play1.py" as this is what the API service below will use.

# -*- coding: utf-8 -*- # stdlib import os # Playwright from playwright.sync_api import Playwright, sync_playwright class Config: Email = os.environ.get('APP_EMAIL', 'zato@example.com') Password = os.environ.get('APP_PASSWORD', '') Headless = bool(os.environ.get('APP_HEADLESS', False)) def run(playwright: Playwright) -> None: browser = playwright.chromium.launch(headless=Config.Headless) # type: ignore context = browser.new_context() # Open new page page = context.new_page() # Open project boards page.goto("https://example.atlassian.net/jira/software/projects/ABC/boards/1") page.goto("https://id.atlassian.com/login?continue=https%3A%2F%2Fexample.atlassian.net%2Flogin%3FredirectCount%3D1%26dest-url%3D%252Fjira%252Fsoftware%252Fprojects%252FABC%252Fboards%252F1%26application%3Djira&application=jira") # Fill out the email page.locator("[placeholder=\"Enter email\"]").click() page.locator("[placeholder=\"Enter email\"]").fill(Config.Email) # Click #login-submit page.locator("#login-submit").click() with sync_playwright() as playwright: run(playwright) Web scraping as a standalone activity

We have the generated code so the first thing to do with it is to run it from command line. This will result in a new Chrome window's accessing Jira - it is Chrome, not Blackberry, because that is the default for Playwright.

The window will close soon enough but this is fine, that code only demonstrates a principle, it is not a full integration task.

python /path/to/play1.py

It is also useful that we can run the same Python module from our IDE, giving us the ability to step through the code line by line, observing what changes when and why.

Web scraping as an API service

Finally, we are ready to invoke the standalone module from an API service, as in the following code that we are also going to make available as a REST channel.

A couple of notes about the Python service below:

• We invoke Playwright in a subprocess, as a shell command
• We accept input through data models although we do not provide any output definition because it is not needed here
• When we invoke Playwright, we set the APP_HEADLESS to True which will ensure that it does not attempt to actually display a Chrome window. After all, we intend for this service to run on Linux servers, in backend, and such a thing will be unlikely to work in this kind of an environment.

Other than that, this is a straightforward Zato service - it receives input, carries out its work and a reply is returned to the caller (here, empty).

# -*- coding: utf-8 -*- # stdlib from dataclasses import dataclass # Zato from zato.server.service import Model, Service # ########################################################################### @dataclass(init=False) class WebScrapingDemoRequest(Model): email: str password: str # ########################################################################### class WebScrapingDemo(Service): name = 'demo.web-scraping' class SimpleIO: input = WebScrapingDemoRequest def handle(self): # Path to a Python installation that Playwright was installed under py_path = '/path/to/python' # Path to a Playwright module with code to invoke playwright_path = '/path/to/the-playwright-module.py' # This is a template script that we will invoke in a subprocess command_template = """ APP_EMAIL={app_email} APP_PASSWORD={app_password} APP_HEADLESS=True {py_path} {playwright_path} """ # This is our input data input = self.request.input # type: WebScrapingDemoRequest # Extract credentials from the input .. email = input.email password = input.password # .. build the full command, taking all the config into account .. command = command_template.format( app_email = email, app_password = password, py_path = py_path, playwright_path = playwright_path, ) # .. invoke the command in a subprocess .. result = self.commands.invoke(command) # .. if it was not a success, log the details received .. if not result.is_ok: self.logger.info('Exit code -> %s', result.exit_code) self.logger.info('Stderr -> %s', result.stderr) self.logger.info('Stdout -> %s', result.stdout) # ###########################################################################

Now, the REST channel:

The last thing to do is to invoke the service - I am using curl from the command line below but it could very well be Postman or a similar option.

curl localhost:17010/demo/web-scraping -d '{"email":"hello@example.com", "password":"abc"}' ; echo

There will be no Chrome window this time around because we run Playwright in the headless mode. There will be no output from curl either because we do not return anything from the service but in server logs we will find details such as below.

We can learn from the log that the command took close to 4 seconds to complete, that the exit code was 0 (indicating success) and that is no stdout or stderr at all.

INFO - Command ` APP_EMAIL=hello@example.com APP_PASSWORD=abc APP_HEADLESS=True /path/to/python /path/to/the-playwright-module.py ` completed in 0:00:03.844157, exit_code -> 0; len-out=0 (0 Bytes); len-err=0 (0 Bytes); cid -> zcmdc5422816b2c6ff9f10742134

We are now ready to continue to work on it - for instance, you will notice that the password is visible in logs and this should not be allowed.

But, all such works are extra in comparison with the main theme - we have Playwright, which is a a tool that allows us to quickly integrate with frontend applications and we can automate it through API services. Just as expected.

More resources

➤ Python API integration tutorial
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

Simpleadmindoc is django application that allows you to quickly create help for modules in Django admin. Goal is to be flexible enough, fast to create and easy to integrate.

Middleware that intialize specific locale for admin pages.

Importing and exporting data with included admin integration.

django-cookie-consent is a reusable application for managing various cookies and visitors consent for their use in Django project.

I find Linux virtualization stack confusing. KVM? ibvirt? QEMU? Xen? What does that even mean?

This post is my attempt at making sense of that all. I don’t claim it to be correct, but that’s the way I understand it. If I badly missed a mark somewhere, please reach out and tell me how wrong I am.

Virtualization primer

Virtual machine is a box inside which programs think they are running on different hardware and operating system. That box is like a full computer, running inside a computer. Virtualization is process of creating these boxes. Virtual machine is called guest, while operating system that runs virtual machine is called host.

Two main reasons to virtualize are security and resource management. Security - because if virtual machine is done correctly, then program inside it won’t even know it’s inside a virtual machine, and that means there’s a good chance it won’t escape to interfere with other programs. Resource management - so you can buy huge server and assign slice of resources to each box as you see fit, and dynamically change it as your needs shift.

Virtual machine pretends to be the entire computer, but most discussions revolve around CPU. Each CPU architecture has a specific set of instructions, and programs generally target only one of them. It is technically possible to translate instructions from one architecture to another, but that’s usually slow. Most modern CPUs provide extensions that allow virtual machines to run at speed comparable to non-virtualized installations. Using these extensions is only possible when the guest and the host target the same CPU architecture.

Conceptually, there is no significant difference between virtualization and emulation. Practically, emulation usually refers to guest thinking it has different CPU architecture than host, and to virtual machine pretending to be some very specific physical hardware - like a video game console. Virtualization in turn usually refers to specific case where guest thinks it has the same CPU architecture as the host.

KVM

KVM is part of the kernel, responsible for talking with hardware. As I mentioned before, most modern CPUs have special extensions that allow guests to achieve near-native performance. KVM makes it possible for Linux host to use these extensions.

QEMU

QEMU is weird, because it’s multiple different things.

First, it can run virtual machines and pass instructions from guest to KVM, where they will be handled by virtualization CPU extension. So you can think about QEMU as user space component for KVM.

Second, it can run virtual machines while emulating different CPU architecture. So you can think about QEMU as user space virtualization solution, which can run without KVM at all.

Third, it can run programs targeting one CPU architecture on a computer with another CPU architecture. It’s like lightweight virtualization, where only single program is virtualized.

Finally, it is set of standalone utilities related to virtualization, but not directly tied to anything in particular. Most of these programs work with disk image files in some way.

QEMU is probably the hardest part to understand, because it can do things that other parts of the stack are responsible for. This results in some overlap between different components and encountering program names in contexts where you would not expect them.

I think this overlap is mostly a historical artifact. The oldest commit in QEMU git repository is dated 2003, while KVM was included in Linux kernel in 2007. So it seems to me that QEMU was originally intended as software virtualization solution that did not depend on any hardware or kernel driver. Later Linux gained drivers for virtualization CPU extensions, but they were useless without something in user space that could work with them. So instead of creating completely new thing, QEMU was extended to work with KVM.

Technically, the journey towards understanding Linux virtualization stack could end here - you can use QEMU to create, start, stop and modify virtual machines. But QEMU commands tend to be verbose and long, so people rarely use it directly.

libvirt

I have ignored that so far, but KVM is not the only virtualization driver in kernel. There are many, some of them closed-source.

libvirt is intended as unified layer that abstracts virtual machine management for application developers. So if you would like your application to create, start, stop or delete virtual machine, you don’t have to support each of these operations on KVM, QEMU, Xen and VirtualBox - you can just support libvirt and trust it will do the right thing.

libvirt doesn’t do any work directly, and instead asks other programs to do it. System administrator is responsible for setting up the host and ensuring virtualization may work. This makes libvirt great for application developers, who can forget about details of different solutions; but most of libvirt users are in less fortunate position, because they have to take a role of system administrator.

libvirt will talk with QEMU, which in turn may pass CPU instructions to KVM. Some sources online claim that libvirt can talk with KVM directly. libvirt itself perpetuates this confusion, as KVM is listed alongside QEMU on the main page, giving the impression they are two equivalent components. But detailed documentation page makes it clear - libvirt only talks with QEMU. It can detect KVM and allow user to create “hardware accelerated guests”, but that case is still handled through QEMU.

It’s also worth noting that libvirt supports multiple virtualization drivers across multiple operating systems, including FreeBSD and macOS.

virsh

virsh is a command line interface for libvirt. That’s it, there’s nothing more to it. If it was created today, it might have been called libvirtctl.

Vagrant

Vagrant abstracts virtual machine management across operating systems. It’s similar to libvirt, in the way that it allows application developers to target Vagrant and leave all the details to someone else. And just like libvirt, Vagrant doesn’t do anything on its own, but pushes all the work to tools lower in the stack.

Vagrant primary audience is software developers. It allows to succinctly define multiple virtual machines, and then start and provision them all with a single command. These machines are usually considered disposable - they might be deleted and re-created multiple times a day. If you want to manually modify the virtual machine and keep these changes for longer period of time, Vagrant is probably not a tool for you.

On Linux, Vagrant usually works with libvirt, but may also work with VirtualBox or Xen.

VirtualBox

VirtualBox is full virtualization solution. It consists of kernel driver and user space programs, including one with graphical interface. You can think of VirtualBox as something parallel to all that we’ve discussed so far.

The main reason to use VirtualBox is ease of use. It offers simpler mental model for virtualization - you just install VirtualBox, start VirtualBox UI and create and run virtual machines. While it does require a kernel module, and kernel is notoriously bad at breaking modules that are not in tree, VirtualBox has a lot of tooling and integrations that will ensure that module is built every time you install or update kernel image. On distributions like Ubuntu you don’t have to think about this at all.

Technically speaking, libvirt supports VirtualBox, so you can use libvirt and virsh to manage VirtualBox machines. In practice that seems to be rare.

Xen

Xen is another virtualization stack. You can think about it as something parallel to KVM/QEMU/libvirt and VirtualBox. What sets Xen apart is that instead of running inside your operating system, it runs below your operating system.

Xen manages scheduling, interrupts and timers - that is, it manages who gets access to computer resources, when, and for how long. Xen is the very first thing that starts when you boot your computer.

But Xen is not an operating system. So right after starting, it starts the guest with one. That guest is special, because it provides drivers to hardware and is the only one with a privilege to talk with Xen. Only through that special guest virtual machine you can create other virtual machines, assign them resources etc.

Xen is one of the oldest virtualization solutions, with first release back in 2003. However, it was included in kernel only in 2010. So while many people prefer Xen and think favorably of its architecture, it seems to lost the popularity contest to KVM and QEMU.

Xen provides the tools required to manage guest virtual machines, but you can also manage them through libvirt.

Summary

I thought to put a visual aid in place of summary. Each column represents a possible stack - you generally want to pick one of them. Items with dashed line style are optional - you can use them, but don’t have to. Click to see the bigger image.

#655 – NOVEMBER 12, 2024
View in Browser »

Introduction to Web Scraping With Python

In this video course, you’ll learn all about web scraping in Python. You’ll see how to parse data from websites and interact with HTML forms using tools such as Beautiful Soup and MechanicalSoup.
REAL PYTHON course

State of Python 3.13 Performance: Free-Threading

This article does a comparison between code in single threaded, threaded, and multi-process versions under Python 3.12, 3.13, and 3.13 free-threaded with the GIL on and off.
ARTHUR PASTEL

Scrape Any Website Without Getting Blocked

The only web scraper API that extracts data at scale with a 98.7% average success rate while automatically handling all anti-bot systems. ZenRows is a complete web scraping toolkit with premium proxies, anti-CAPTCHA, headless browsers, and more. Try for free now →
ZENROWS sponsor

Tiny Great Languages: Assembly

This is the first post in a multi-part series that uses Python to build tiny interpreters for other languages.
SERGE ZAITSEV

Django Bugfix Release Issued: 5.1.3

DJANGO SOFTWARE FOUNDATION

PyCon US 2025 (Pittsburgh) Call for Proposals

PRETALX.COM

Quiz: Beautiful Soup: Build a Web Scraper With Python

REAL PYTHON

Quiz: Pydantic: Simplifying Data Validation in Python

REAL PYTHON

Articles & Tutorials Opposite of Cloud Native Is…?

Michael (from Talk Python fame) introduces the concept of “stack-native” as the opposite of “cloud-native”, and how it applies to Python web apps. Building applications with just enough full-stack building blocks to run reliably with minimal complexity, rather than relying on a multitude of cloud services.
MICHAEL KENNEDY

How to Reset a pandas DataFrame Index

In this tutorial, you’ll learn how to reset a pandas DataFrame index using various techniques. You’ll also learn why you might want to do this and understand the problems you can avoid by optimizing the index structure.
REAL PYTHON

Do Your Data Science Teams Struggle to Share Their Work With Their Colleagues Securely

Posit Connect lets data teams publish, host, & manage Python work. It provides a secure, scalable platform for sharing data insights with those who need them, including models, Jupyter notebooks, Streamlit & Shiny apps, Plotly dashboards, and more.
POSIT sponsor

Python Closures: Common Use Cases and Examples

In this tutorial, you’ll learn about Python closures. A closure is a function-like object with an extended scope. You can use closures to create decorators, factory functions, stateful functions, and more.
REAL PYTHON

In Defense of Simple Architectures

You can go surprisingly far with a simple software architecture, in fact simplicity can make it easier to scale. This post talks about some real world cases of just that.
DAN LUU

Vehicle Routing Using OpenStreetMap

This project shows you how to implement the Vehicle Routing Problem (VRP) using OpenStreetMap data and Google’s OR-Tools library to find efficient routes.
MEDIUM.COM/P • Shared by Albert Ferré

Guide to Python Project Management and Packaging

This is a deep dive article on Python project management and packaging. It covers the pyproject.toml file, modules, dependencies, locking, and more.
REINFORCED KNOWLEDGE

I Waited 10B Cycles and All I Got Was This Loading Screen

This opinion piece discusses the fact that our faster hardware still can’t keep up with our bloated software and why that is the case.
PRANAV NUTALAPTI

Thoughts on Django’s Core

This post by a Django core developer talks about the last twenty years of the library and why it has had such staying power.
CARLTON GIBSON

map and filter With List Comprehensions

This quick-hit post shows you how to use the map() and filter() functions in conjunction with list comprehensions.
JUHA-MATTI SANTALA

Don’t Return Named Tuples in New APIs

This post talks about when and when not to use a named tuple in your API and why you might make that choice.
BRETT CANNON

Projects & Code mininterface: A Minimal Interface to Python Application

GITHUB.COM/CZ-NIC

deutschland: German Dataset Collection

GITHUB.COM/BUNDESAPI

pytest-metadata: Plugin for Accessing Test Session Metadata

GITHUB.COM/PYTEST-DEV

BlendModes: Apply Blending Modes to Images

GITHUB.COM/FHPYTHONUTILS

django-admin-dracula: Dracula Themes for the Django Admin

GITHUB.COM/SJBITCODE

Events IndyPy Hosts “AI & Data for Good”

November 12, 2024
MEETUP.COM • Shared by Laura Stephens

Weekly Real Python Office Hours Q&A (Virtual)

November 13, 2024
REALPYTHON.COM

PyCon Sweden 2024

November 14 to November 16, 2024
PYCON.SE

Python Atlanta

November 15, 2024
MEETUP.COM

PyCon Hong Kong 2024

November 16 to November 17, 2024
PYCON.HK

PyCon Mini Tokai 2024

November 16 to November 17, 2024
PYCON.JP

PyCon Ireland 2024

November 16 to November 18, 2024
PYTHON.IE

PyConAU 2024

November 22 to November 27, 2024
PYCON.ORG.AU

Happy Pythoning!
This was PyCoder’s Weekly Issue #655.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

You’ll often need to format and round a Python float to display the results of your calculations neatly within strings. In earlier versions of Python, this was a messy thing to do because you needed to round your numbers first and then use either string concatenation or the old string formatting technique to do this for you.

Since Python 3.6, the literal string interpolation, more commonly known as a formatted string literal or f-string, allows you to customize the content of your strings in a more readable way.

An f-string is a literal string prefixed with a lowercase or uppercase letter f and contains zero or more replacement fields enclosed within a pair of curly braces {...}. Each field contains an expression that produces a value. You can calculate the field’s content, but you can also use function calls or even variables.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In a previous post we saw that you can speed up code significantly on a single core using SIMD: Single Instruction Multiple Data. These specialized CPU instructions allow you to, for example, add 4 values at once with a single instruction, instead of the usual one value at a time. The performance improvement you get compounds with multi-core parallelism: you can benefit from both SIMD and threading at the same time.

Unfortunately, SIMD instructions are specific both to CPU architecture and CPU model. Thus ARM CPUs as used on modern Macs have different SIMD instructions than x86-64 CPUs. And even if you only care about x86-64, different models support different instructions; the i7-12700K CPU in my current computer doesn’t support AVX-512 SIMD, for example.

One way to deal with this is to write custom versions for each variation of SIMD instructions. Another is to use a portable SIMD library, that provides an abstraction layer on top of these various instruction sets.

In the previous post we used the std::simd library. It is built-in to the Rust standard library… but unfortunately it’s currently only available when using an unstable (“nightly”) compiler.

How you can write portable SIMD if you want to use stable Rust? In this article we’ll:

• Introduce the wide crate, that lets you write portable SIMD in stable Rust.
• Show how it can be used to reimplement the Mandelbrot algorithm we previously implemented with std::simd.
• Go over the pros and cons of these alternatives.

Read more...

Machine learning models are rapidly becoming more capable; how can we make use of these powerful new tools in our Go applications?

For top-of-the-line commercial LLMs like ChatGPT, Gemini or Claude, the models are exposed as language agnostic REST APIs. We can hand-craft HTTP requests or use client libraries (SDKs) provided by the LLM vendors. If we need more customized solutions, however, some challenges arise. Completely bespoke models are typically trained in Python using tools like TensorFlow, JAX or PyTorch that don't have real non-Python alternatives.

In this post, I will present some approaches for Go developers to use ML models in their applications - with increasing level of customization. The summary up front is that it's pretty easy, and we only have to deal with Python very minimally, if at all - depending on the circumstances.

Internet LLM services

This is the easiest category: multimodal services from Google, OpenAI and others are available as REST APIs with convenient client libraries for most leading languages (including Go), as well as third-party packages that provide abstractions on top (e.g. langchaingo).

Check out the official Go blog titled Building LLM-powered applications in Go that was published earlier this year. I've written about it before on this blog as well: #1, #2, #3 etc.

Go is typically as well supported as other programming languages in this domain; in fact, it's uniquely powerful for such applications because of its network-native nature; quoting from the Go blog post:

Working with LLM services often means sending REST or RPC requests to a network service, waiting for the response, sending new requests to other services based on that and so on. Go excels at all of these, providing great tools for managing concurrency and the complexity of juggling network services.

Since this has been covered extensively, let's move on to the more challenging scenarios.

Locally-running LLMs

There's a plethora of high-quality open models [1] one can choose from to run locally: Gemma, Llama, Mistral and many more. While these models aren't quite as capable as the strongest commercial LLM services, they are often surprisingly good and have clear benefits w.r.t. cost and privacy.

The industry has begun standardizing on some common formats for shipping and sharing these models - e.g. GGUF from llama.cpp, safetensors from Hugging Face or the older ONNX. Additionally, there are a number of excellent OSS tools that let us run such models locally and expose a REST API for an experience that's very similar to the OpenAI or Gemini APIs, including dedicated client libraries.

The best known such tool is probably Ollama; I've written extensively about it in the past: #1, #2, #3.

Ollama lets us customize an LLM through a Modelfile, which includes things like setting model parameters, system prompts etc. If we fine-tuned a model [2], it can also be loaded into Ollama by specifying our own GGUF file.

If you're running in a cloud environment, some vendors already have off-the-shelf solutions like GCP's Cloud Run integration that may be useful.

Ollama isn't the only player in this game, either; recently a new tool emerged with a slightly different approach. Llamafile distributes the entire model as a single binary, which is portable across several OSes and CPU architectures. Like Ollama, it provides REST APIs for the model.

If such a customized LLM is a suitable solution for your project, consider just running Ollama or Llamafile and using their REST APIs to communicate with the model. If you need higher degrees of customization, read on.

A note about the sidecar pattern

Before we proceed, I want to briefly discuss the sidecar pattern of application deployment. That k8s link talks about containers, but the pattern isn't limited to these. It applies to any software architecture in which functionality is isolated across processes.

Suppose we have an application that requires some library functionality; using Go as an example, we could find an appropriate package, import it and be on our way. Suppose there's no suitable Go package, however. If libraries exist with a C interface, we could alternatively use cgo to import it.

But say there's no C API either, for example if the functionality is only provided by a language without a convenient exported interface. Maybe it's in Lisp, or Perl, or... Python.

A very general solution could be to wrap the code we need in some kind of server interface and run it as a separate process; this kind of process is called a sidecar - it's launched specifically to provide additional functionality for another process. Whichever inter-process communication (IPC) mechanism we use, the benefits of this approach are many - isolation, security, language independence, etc. In today's world of containers and orchestration this approach is becoming increasingly more common; this is why many of the links about sidecars lead to k8s and other containerized solutions.

The Ollama approach outlined in the previous section is one example of using the sidecar pattern. Ollama provides us with LLM functionality but it runs as a server in its own process.

The solutions presented in the rest of this post are more explicit and fully worked-out examples of using the sidecar pattern.

Locally-running LLM with Python and JAX

Suppose none of the existing open LLMs will do for our project, even fine-tuned. At this point we can consider training our own LLM - this is hugely expensive, but perhaps there's no choice. Training usually involves one of the large ML frameworks like TensorFlow, JAX or PyTorch. In this section I'm not going to talk about how to train models; instead, I'll show how to run local inference of an already trained model - in Python with JAX, and use that as a sidecar server for a Go application.

The sample (full code is here) is based on the official Gemma repository, using its sampler library [3]. It comes with a README that explains how to set everything up. This is the relevant code instantiating a Gemma sampler:

# Once initialized, this will hold a sampler_lib.Sampler instance that # can be used to generate text. gemma_sampler = None def initialize_gemma(): """Initialize Gemma sampler, loading the model into the GPU.""" model_checkpoint = os.getenv("MODEL_CHECKPOINT") model_tokenizer = os.getenv("MODEL_TOKENIZER") parameters = params_lib.load_and_format_params(model_checkpoint) print("Parameters loaded") vocab = spm.SentencePieceProcessor() vocab.Load(model_tokenizer) transformer_config = transformer_lib.TransformerConfig.from_params( parameters, cache_size=1024, ) transformer = transformer_lib.Transformer(transformer_config) global gemma_sampler gemma_sampler = sampler_lib.Sampler( transformer=transformer, vocab=vocab, params=parameters["transformer"], ) print("Sampler ready")

The model weights and tokenizer vocabulary are files downloaded from Kaggle, per the instructions in the Gemma repository README.

So we have LLM inference up and running in Python; how do we use it from Go?

Using a sidecar, of course. Let's whip up a quick web server around this model and expose a trivial REST interface on a local port that Go (or any other tool) can talk to. As an example, I've set up a Flask-based web server around this inference code. The web server is invoked with gunicorn - see the shell script for details.

Excluding the imports, here's the entire application code:

def create_app(): # Create an app and perform one-time initialization of Gemma. app = Flask(__name__) with app.app_context(): initialize_gemma() return app app = create_app() # Route for simple echoing / smoke test. @app.route("/echo", methods=["POST"]) def echo(): prompt = request.json["prompt"] return {"echo_prompt": prompt} # The real route for generating text. @app.route("/prompt", methods=["POST"]) def prompt(): prompt = request.json["prompt"] # For total_generation_steps, 128 is a default taken from the Gemma # sample. It's a tradeoff between speed and quality (higher values mean # better quality but slower generation). # The user can override this value by passing a "sampling_steps" key in # the request JSON. sampling_steps = request.json.get("sampling_steps", 128) sampled_str = gemma_sampler( input_strings=[prompt], total_generation_steps=int(sampling_steps), ).text return {"response": sampled_str}

The server exposes two routes:

• prompt: a client sends in a textual prompt, the server runs Gemma inference and returns the generated text in a JSON response
• echo: used for testing and benchmarking

Here's how it all looks tied together:

The important takeaway is that this is just an example. Literally any part of this setup can be changed: one could use a different ML library (maybe PyTorch instead of JAX); one could use a different model (not Gemma, not even an LLM) and one can use a different setup to build a web server around it. There are many options, and each developer will choose what fits their project best.

It's also worth noting that we've written less than 100 lines of Python code in total - much of it piecing together snippets from tutorials. This tiny amount of Python code is sufficient to wrap an HTTP server with a simple REST interface around an LLM running locally through JAX on the GPU. From here on, we're safely back in our application's actual business logic and Go.

Now, a word about performance. One of the concerns developers may have with sidecar-based solutions is the performance overhead of IPC between Python and Go. I've added a simple echo endpoint to measure this effect; take a look at the Go client that exercises it; on my machine the latency of sending a JSON request from Go to the Python server and getting back the echo response is about 0.35 ms on average. Compared to the time it takes Gemma to process a prompt and return a response (typically measured in seconds, or maybe hundreds of milliseconds on very powerful GPUs), this is entirely negligible.

That said, not every custom model you may need to run is a full-fledged LLM. What if your model is small and fast, and the overhead of 0.35 ms becomes significant? Worry not, it can be optimized. This is the topic of the next section.

Locally-running fast image model with Python and TensorFlow

The final sample of this post mixes things up a bit:

• We'll be using a simple image model (instead of an LLM)
• We're going to train it ourselves using TensorFlow+Keras (instead of JAX)
• We'll use a different IPC method between the Python sidecar server and clients (instead of HTTP+REST)

The model is still implemented in Python, and it's still driven as a sidecar server process by a Go client [4]. The idea here is to show the versatility of the sidecar approach, and to demonstrate a lower-latency way to communicate between the processes.

The full code of the sample is here. It trains a simple CNN (convolutional neural network) to classify images from the CIFAR-10 dataset:

The neural net setup with TensorFlow and Keras was taken from an official tutorial. Here's the entire network definition:

model = models.Sequential() model.add(layers.Conv2D(32, (3, 3), activation="relu", input_shape=(32, 32, 3))) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation="relu")) model.add(layers.MaxPooling2D((2, 2))) model.add(layers.Conv2D(64, (3, 3), activation="relu")) model.add(layers.Flatten()) model.add(layers.Dense(64, activation="relu")) model.add(layers.Dense(10))

CIFAR-10 images are 32x32 pixels, each pixel being 3 values for red, green and blue. In the original dataset, these values are bytes in the inclusive range 0-255 representing color intensity. This should explain the (32, 32, 3) shape appearing in the code. The full code for training the model is in the train.py file in the sample; it runs for a bit and saves the serialized model along with the trained weights into a local file.

The next component is an "image server": it loads the trained model+weights file from disk and runs inference on images passed into it, returning the label the model thinks is most likely for each.

The server doesn't use HTTP and REST, however. It creates a Unix domain socket and uses a simple length-prefix encoding protocol to communicate:

Each packet starts with a 4-byte field that specifies the length of the rest of the contents. A type is a single byte, and the body can be anything [5]. In the sample image server two commands are currently supported:

• 0 means "echo" - the server will respond with the same packet back to the client. The contents of the packet body are immaterial.
• 1 means "classify" - the packet body is interpreted as a 32x32 RGB image, encoded as the red channel for each pixel in the first 1024 bytes (32x32, row major), then green in the next 1024 bytes and finally blue in the last 1024 bytes. Here the server will run the image through the model, and reply with the label the model thinks describes the image.

The sample also includes a simple Go client that can take a PNG file from disk, encode it in the required format and send it over the domain socket to the server, recording the response.

The client can also be used to benchmark the latency of a roundtrip message exchange. It's easier to just show the code instead of explaining what it does:

func runBenchmark(c net.Conn, numIters int) { // Create a []byte with 3072 bytes. body := make([]byte, 3072) for i := range body { body[i] = byte(i % 256) } t1 := time.Now() for range numIters { sendPacket(c, messageTypeEcho, body) cmd, resp := readPacket(c) if cmd != 0 || len(resp) != len(body) { log.Fatal("bad response") } } elapsed := time.Since(t1) fmt.Printf("Num packets: %d, Elapsed time: %s\n", numIters, elapsed) fmt.Printf("Average time per request: %d ns\n", elapsed.Nanoseconds()/int64(numIters)) }

In my testing, the average latency of a roundtrip is about 10 μs (that's micro-seconds). Considering the size of the message and it being Python on the other end, this is roughly in-line with my earlier benchmarking of Unix domain socket latency in Go.

How long does a single image inference take with this model? In my measurements, about 3 ms. Recall that the communication latency for the HTTP+REST approach was 0.35 ms; while this is only 12% of the image inference time, it's close enough to be potentially worrying. On a beefy server-class GPU the time can be much shorter [6].

With the custom protocol over domain sockets, the latency - being 10 μs - seems quite negligible no matter what you end up running on your GPU.

Code

The full code for the samples in this post is on GitHub.

[1]To be pedantic, these models are not entirely open: their inference architecture is open-source and their weights are available, but the details of their training remain proprietary. [2]The details of fine-tuning models are beyond the scope of this post, but there are plenty resources about this online. [3]"Sampling" in LLMs means roughly "inference". A trained model is fed an input prompt and then "sampled" to produce its output. [4]In my samples, the Python server and Go client simply run in different terminals and talk to each other. How service management is structured is very project-specific. We could envision an approach wherein the Go application launches the Python server to run in the background and communicates with it. Increasingly likely these days, however, would be a container-based setup, where each program is its own container and an orchestration solution launches and manages these containers. [5]You may be wondering why I'm implementing a custom protocol here instead of using something established. In real life, I'd definitely recommend using something like gRPC. However, for the sake of this sample I wanted something that would be (1) simple without additional libraries and (2) very fast. FWIW, I don't think the latency numbers would be very much different for gRPC. Check out my earlier post about RPC over Unix domain sockets in Go. [6]On the other hand, the model I'm running here is really small. It's fair to say realistic models you'll use in your application will be much larger and hence slower.

The latest Python developments all point to the same thing—Python is currently thriving. The recent GitHub Octoverse 2024 report has revealed that Python is now the most used language on GitHub. Also, last month saw the release of Python 3.13, which is already laying the groundwork for some exciting future improvements.

While Python core developers have been busy exploring the language’s features as they tinker with upcoming enhancements, it’s good to know that working on Python’s source code isn’t the only way you can contribute to Python’s future. Another way to shape the focus of upcoming releases is to join the Python Developers Survey 2024.

And with the end of the year in sight, you may want to venture a look at next year’s calendar and mark some dates, such as the PyCon US conference in May or the Python 3.14 release in October 2025.

Now that you know the highlights, it’s time to dive into the most important Python news for November.

Join Now: Click here to join the Real Python Newsletter and you'll never miss another Python tutorial, course update, or post.

Python’s Popularity Shines in GitHub’s Octoverse 2024

The latest Octoverse report for 2024 shows that Python remains one of the most widely used languages on GitHub, securing its place as a core language in open-source and professional development. Python ranked among the top three most-used languages, demonstrating its continued appeal across industries and experience levels:

As GitHub’s annual report illustrates, Python’s popularity is fueled by its solid role in developing machine learning and artificial intelligence frameworks.

Another takeaway from the Octoverse survey is Python’s strong community engagement. Python developers are not only active in contributing code but also in participating in discussions, filing issues, and reviewing pull requests.

Read the full article at https://realpython.com/python-news-november-2024/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Hi there!

I'm happy to announce version 3.15 of reader, a Python feed reader library.

What's new? #

Here are the highlights since reader 3.13.

Retry-After #

Now that it supports scheduled updates, reader can honor the Retry-After HTTP header sent with 429 Too Many Requests or 503 Service Unavailable responses.

Adding this required an extensive rework of the parser internal API, but I'd say it was worth it, since we're getting quite close to it becoming stable.

Next up in HTTP compliance is to do more on behalf of the user: bump the update interval on repeated throttling, and handle gone and redirected feeds accordingly.

Faster tag filters, feed slugs #

OR-only tag filters like get_feeds(tags=[['one', 'two']]) now use an index.

This is useful for maintaining a reverse mapping to feeds/entries, like the feed slugs recipe does to add support for user-defined short URLs:

>>> url = 'https://death.andgravity.com/_feed/index.xml' >>> reader.set_feed_slug(url, 'andgravity') >>> reader.get_feed_by_slug('andgravity') Feed(url='https://death.andgravity.com/_feed/index.xml', ...)

(Interested in adopting this recipe as a real plugin? Submit a pull request!)

enclosure_tags improvements #

The enclosure_tags plugin fixes ID3 tags for MP3 enclosures like podcasts.

I've changed the implementation to rewrite tags on the fly, instead of downloading the entire file, rewriting tags, and then sending it to the user; this should allow browsers to display accurate download progress.

Some other, smaller improvements:

• Set genre to Podcast if the feed has any tag containing "podcast".
• Prefer feed user title to feed title if available.
• Use feed title as artist, instead of author.

Using the installed feedparser #

Because feedparser makes PyPI releases at a lower cadence, reader has been using a vendored version of feedparser's develop branch for some time. It is now possible to opt out of this behavior and make reader use the installed feedparser package.

Python versions #

reader 3.14 (released back in July) adds support for Python 3.13.

That's it for now. For more details, see the full changelog.

Want to contribute? Check out the docs and the roadmap.

Learned something new today? Share this with others, it really helps! PyCoder's Weekly HN Reddit linkedin Twitter

What is reader? #

reader takes care of the core functionality required by a feed reader, so you can focus on what makes yours different.

reader allows you to:

• retrieve, store, and manage Atom, RSS, and JSON feeds
• mark articles as read or important
• add arbitrary tags/metadata to feeds and articles
• filter feeds and articles
• full-text search articles
• get statistics on feed and user activity
• write plugins to extend its functionality

...all these with:

• a stable, clearly documented API
• excellent test coverage
• fully typed Python

To find out more, check out the GitHub repo and the docs, or give the tutorial a try.

Why use a feed reader library? #

Have you been unhappy with existing feed readers and wanted to make your own, but:

• never knew where to start?
• it seemed like too much work?
• you don't like writing backend code?

Are you already working with feedparser, but:

• want an easier way to store, filter, sort and search feeds and entries?
• want to get back type-annotated objects instead of dicts?
• want to restrict or deny file-system access?
• want to change the way feeds are retrieved by using Requests?
• want to also support JSON Feed?
• want to support custom information sources?

... while still supporting all the feed types feedparser does?

If you answered yes to any of the above, reader can help.

The reader philosophy #

• reader is a library
• reader is for the long term
• reader is extensible
• reader is stable (within reason)
• reader is simple to use; API matters
• reader features work well together
• reader is tested
• reader is documented
• reader has minimal dependencies

Why make your own feed reader? #

So you can:

• have full control over your data
• control what features it has or doesn't have
• decide how much you pay for it
• make sure it doesn't get closed while you're still using it
• really, it's easier than you think

Obviously, this may not be your cup of tea, but if it is, reader can help.

A Python f-string (formatted string literal) allows you to insert variables or expressions directly into a string by placing them inside curly braces {}. In this tutorial, you will learn about the Pyt

We're thrilled to announce the much-anticipated return of DjangoCon Europe, set to take place in the vibrant city of Dublin, Ireland, in 2025! DjangoCon Europe has been a cornerstone of the Django community, bringing together developers and enthusiasts from all over Europe and beyond to celebrate and advance the Django web framework.

Save the Dates

Mark your calendars for DjangoCon Europe 2025, which will be held from April 23th to 27th. The conference will host a balanced mix of insightful talks, hands-on workshops, and ample opportunities for networking and socialising with fellow Django enthusiasts.

Explore Dublin

With its rich history and vibrant tech scene, Dublin is the perfect backdrop for this year's conference. Dublin's thriving tech community and innovative spirit make it an ideal host for DjangoCon Europe. Plus, the city's lively culture, breathtaking architecture, and friendly locals are sure to provide an unforgettable experience.

Call for Proposals

DjangoCon Europe wouldn't be the same without the insightful and diverse talks contributed by our community. We encourage you to consider submitting a proposal to share your knowledge, experiences, and insights with the Django community. Keep an eye out for the Call for Proposals (CFP) announcement. This is your chance to contribute to the conference program and help make DjangoCon Europe 2025 exceptional.

Get Involved

DjangoCon Europe is a community-driven event, and we rely on the active participation and support of our community members. Here are a few ways you can get involved:

• Attend: Join us in Dublin for a week of learning, networking, and fun.
• Speak: Share your expertise by submitting a talk proposal when the CFP opens.
• Sponsor: Support the conference financially and gain visibility in the Django community (email us at sponsors@djangocon.eu)
• Volunteer: Help us make the conference run smoothly by volunteering your time and skills (https://forms.gle/xmwxssiheMa1oCvPA)

Stay tuned for updates on registration, sponsorship opportunities, and more by following DjangoCon Europe on Twitter and Linkedin.

Stay Informed

To stay up-to-date with the latest DjangoCon Europe 2025 news, visit our website and follow us on Twitter & Linkedin. We will be sharing details about the schedule, speakers, and more in the coming months, so make sure you're on the list!

We can't wait to see you in Dublin for DjangoCon Europe 2025. Get ready for a week of learning, networking, and celebrating all things Django. It's going to be an unforgettable event, and we look forward to sharing this experience with you. Thank you for being a part of our amazing Django community!

See you in Dublin! 🍀

PS: Keep an eye on our social media for special offer we will have during the upcoming holiday season 😉