Planet Python

The 2024 version of the EuroPython conference took place both online and in person in July 2024. This was the second conference under our new Code of Conduct (CoC), and we had Code of Conduct working group members continuously available both online and in person.

Reports

We had 4 Code of Conduct working group members continuously available both online and in person. Over the course of the conference the Code of Conduct team was made aware of the following issues:

• A disabled person had requested reserved seating for talks, but when he arrived the first day, there was none. He reported this to a CoC member, who filed a report with Ops. It turned out that while the request had been gathered on the web form, there was no mechanism to get that information to the people involved. Once they were informed, the issue was quickly resolved, and the reporter expressed satisfaction with the way it was handled.
• One person was uncomfortable with having their last name shown on Discord. They were informed that they could change that as soon as the registration bot ran, limiting the exposure to a minute or so, or that they could come to the registration desk for assistance. The report came via email and there was no response to the email suggesting those options.
• An attendee reported that one talk&aposs slides included a meme that seemed to reflect a racist trope. The CoC team reviewed that talk&aposs slides, and agreed that the meme might be interpreted that way. A member of the CoC team contacted the presenter who immediately agreed to remove that meme before uploading the slides, and the video team was alerted to edit that meme out of the talk video before final publication.
• There were multiple reports that the toilet signage was confusing and causing people to be uncomfortable with choosing a toilet. Once this was reported the signage was adjusted to make the gender designation visible and no further reports were received. It should be noted that none of the complaints objected to the text of the signs, just to the fact that covering of gender markers led to people entering a toilet they didn&apost want to.
• The CoC team also were presented with a potential lightning talk topic that had caused complaints at another conference due to references to current wars that some viewers found disturbing. Since lightning talks are too short for content warnings to be effective, and since they are not reviewed in any detail by the programme committee, the CoC team counselled the prospective presenter against using the references that had been problematic at a prior conference. Given that advice, the presenter elected not to submit that topic.

This is a cross-post from Max Bernstein from his excellent blog where he writes about programming languages, compilers, optimizations, virtual machines. He's looking for a (dynamic language runtime or compiler related) job too.

CF Bolz-Tereick wrote some excellent posts in which they introduce a small IR and optimizer and extend it with allocation removal. We also did a live stream together in which we did some more heap optimizations.

In this blog post, I'm going to write a small abtract interpreter for the Toy IR and then show how we can use it to do some simple optimizations. It assumes that you are familiar with the little IR, which I have reproduced unchanged in a GitHub Gist.

Abstract interpretation is a general framework for efficiently computing properties that must be true for all possible executions of a program. It's a widely used approach both in compiler optimizations as well as offline static analysis for finding bugs. I'm writing this post to pave the way for CF's next post on proving abstract interpreters correct for range analysis and known bits analysis inside PyPy.

Before we begin, I want to note a couple of things:

• The Toy IR is in SSA form, which means that every variable is defined exactly once. This means that abstract properties of each variable are easy to track.
• The Toy IR represents a linear trace without control flow, meaning we won't talk about meet/join or fixpoints. They only make sense if the IR has a notion of conditional branches or back edges (loops).

Alright, let's get started.

Welcome to abstract interpretation

Abstract interpretation means a couple different things to different people. There's rigorous mathematical formalism thanks to Patrick and Radhia Cousot, our favorite power couple, and there's also sketchy hand-wavy stuff like what will follow in this post. In the end, all people are trying to do is reason about program behavior without running it.

In particular, abstract interpretation is an over-approximation of the behavior of a program. Correctly implemented abstract interpreters never lie, but they might be a little bit pessimistic. This is because instead of using real values and running the program---which would produce a concrete result and some real-world behavior---we "run" the program with a parallel universe of abstract values. This abstract run gives us information about all possible runs of the program.1

Abstract values always represent sets of concrete values. Instead of literally storing a set (in the world of integers, for example, it could get pretty big...there are a lot of integers), we group them into a finite number of named subsets.2

Let's learn a little about abstract interpretation with an example program and example abstract domain. Here's the example program:

v0 = 1 v1 = 2 v2 = add(v0, v1)

And our abstract domain is "is the number positive" (where "positive" means nonnegative, but I wanted to keep the words distinct):

top / \ positive negative \ / bottom

The special top value means "I don't know" and the special bottom value means "empty set" or "unreachable". The positive and negative values represent the sets of all positive and negative numbers, respectively.

We initialize all the variables v0, v1, and v2 to bottom and then walk our IR, updating our knowledge as we go.

# here v0:bottom = 1 v1:bottom = 2 v2:bottom = add(v0, v1)

In order to do that, we have to have transfer functions for each operation. For constants, the transfer function is easy: determine if the constant is positive or negative. For other operations, we have to define a function that takes the abstract values of the operands and returns the abstract value of the result.

In order to be correct, transfer functions for operations have to be compatible with the behavior of their corresponding concrete implementations. You can think of them having an implicit universal quantifier forall in front of them.

Let's step through the constants at least:

v0:positive = 1 v1:positive = 2 # here v2:bottom = add(v0, v1)

Now we need to figure out the transfer function for add. It's kind of tricky right now because we haven't specified our abstract domain very well. I keep saying "numbers", but what kinds of numbers? Integers? Real numbers? Floating point? Some kind of fixed-width bit vector (int8, uint32, ...) like an actual machine "integer"?

For this post, I am going to use the mathematical definition of integer, which means that the values are not bounded in size and therefore do not overflow. Actual hardware memory constraints aside, this is kind of like a Python int.

So let's look at what happens when we add two abstract numbers:

top positive negative bottom top top top top bottom positive top positive top bottom negative top top negative bottom bottom bottom bottom bottom bottom

As an example, let's try to add two numbers a and b, where a is positive and b is negative. We don't know anything about their values other than their signs. They could be 5 and -3, where the result is 2, or they could be 1 and -100, where the result is -99. This is why we can't say anything about the result of this operation and have to return top.

The short of this table is that we only really know the result of an addition if both operands are positive or both operands are negative. Thankfully, in this example, both operands are known positive. So we can learn something about v2:

v0:positive = 1 v1:positive = 2 v2:positive = add(v0, v1) # here

This may not seem useful in isolation, but analyzing more complex programs even with this simple domain may be able to remove checks such as if (v2 < 0) { ... }.

Let's take a look at another example using an sample absval (absolute value) IR operation:

v0 = getarg(0) v1 = getarg(1) v2 = absval(v0) v3 = absval(v1) v4 = add(v2, v3) v5 = absval(v4)

Even though we have no constant/concrete values, we can still learn something about the states of values throughout the program. Since we know that absval always returns a positive number, we learn that v2, v3, and v4 are all positive. This means that we can optimize out the absval operation on v5:

v0:top = getarg(0) v1:top = getarg(1) v2:positive = absval(v0) v3:positive = absval(v1) v4:positive = add(v2, v3) v5:positive = v4

Other interesting lattices include:

• Constants (where the middle row is pretty wide)
• Range analysis (bounds on min and max of a number)
• Known bits (using a bitvector representation of a number, which bits are always 0 or 1)

For the rest of this blog post, we are going to do a very limited version of "known bits", called parity. This analysis only tracks the least significant bit of a number, which indicates if it is even or odd.

Parity

The lattice is pretty similar to the positive/negative lattice:

top / \ even odd \ / bottom

Let's define a data structure to represent this in Python code:

class Parity: def __init__(self, name): self.name = name def __repr__(self): return self.name

And instantiate the members of the lattice:

TOP = Parity("top") EVEN = Parity("even") ODD = Parity("odd") BOTTOM = Parity("bottom")

Now let's write a forward flow analysis of a basic block using this lattice. We'll do that by assuming that a method on Parity is defined for each IR operation. For example, Parity.add, Parity.lshift, etc.

def analyze(block: Block) -> None: parity = {v: BOTTOM for v in block} def parity_of(value): if isinstance(value, Constant): return Parity.const(value) return parity[value] for op in block: transfer = getattr(Parity, op.name) args = [parity_of(arg.find()) for arg in op.args] parity[op] = transfer(*args)

For every operation, we compute the abstract value---the parity---of the arguments and then call the corresponding method on Parity to get the abstract result.

We need to special case Constants due to a quirk of how the Toy IR is constructed: the constants don't appear in the instruction stream and instead are free-floating.

Let's start by looking at the abstraction function for concrete values---constants:

class Parity: # ... @staticmethod def const(value): if value.value % 2 == 0: return EVEN else: return ODD

Seems reasonable enough. Let's pause on operations for a moment and consider an example program:

v0 = getarg(0) v1 = getarg(1) v2 = lshift(v0, 1) v3 = lshift(v1, 1) v4 = add(v2, v3) v5 = dummy(v4)

This function (which is admittedly a little contrived) takes two inputs, shifts them left by one bit, adds the result, and then checks the least significant bit of the addition result. It then passes that result into a dummy function, which you can think of as "return" or "escape".

To do some abstract interpretation on this program, we'll need to implement the transfer functions for lshift and add (dummy will just always return TOP). We'll start with add. Remember that adding two even numbers returns an even number, adding two odd numbers returns an even number, and mixing even and odd returns an odd number.

class Parity: # ... def add(self, other): if self is BOTTOM or other is BOTTOM: return BOTTOM if self is TOP or other is TOP: return TOP if self is EVEN and other is EVEN: return EVEN if self is ODD and other is ODD: return EVEN return ODD

We also need to fill in the other cases where the operands are top or bottom. In this case, they are both "contagious"; if either operand is bottom, the result is as well. If neither is bottom but either operand is top, the result is as well.

Now let's look at lshift. Shifting any number left by a non-zero number of bits will always result in an even number, but we need to be careful about the zero case! Shifting by zero doesn't change the number at all. Unfortunately, since our lattice has no notion of zero, we have to over-approximate here:

class Parity: # ... def lshift(self, other): # self << other if other is ODD: return EVEN return TOP

This means that we will miss some opportunities to optimize, but it's a tradeoff that's just part of the game. (We could also add more elements to our lattice, but that's a topic for another day.)

Now, if we run our abstract interpretation, we'll collect some interesting properties about the program. If we temporarily hack on the internals of bb_to_str, we can print out parity information alongside the IR operations:

v0:top = getarg(0) v1:top = getarg(1) v2:even = lshift(v0, 1) v3:even = lshift(v1, 1) v4:even = add(v2, v3) v5:top = dummy(v4)

This is pretty awesome, because we can see that v4, the result of the addition, is always even. Maybe we can do something with that information.

Optimization

One way that a program might check if a number is odd is by checking the least significant bit. This is a common pattern in C code, where you might see code like y = x & 1. Let's introduce a bitand IR operation that acts like the & operator in C/Python. Here is an example of use of it in our program:

v0 = getarg(0) v1 = getarg(1) v2 = lshift(v0, 1) v3 = lshift(v1, 1) v4 = add(v2, v3) v5 = bitand(v4, 1) # new! v6 = dummy(v5)

We'll hold off on implementing the transfer function for it---that's left as an exercise for the reader---and instead do something different.

Instead, we'll see if we can optimize operations of the form bitand(X, 1). If we statically know the parity as a result of abstract interpretation, we can replace the bitand with a constant 0 or 1.

We'll first modify the analyze function (and rename it) to return a new Block containing optimized instructions:

def simplify(block: Block) -> Block: parity = {v: BOTTOM for v in block} def parity_of(value): if isinstance(value, Constant): return Parity.const(value) return parity[value] result = Block() for op in block: # TODO: Optimize op # Emit result.append(op) # Analyze transfer = getattr(Parity, op.name) args = [parity_of(arg.find()) for arg in op.args] parity[op] = transfer(*args) return result

We're approaching this the way that PyPy does things under the hood, which is all in roughly a single pass. It tries to optimize an instruction away, and if it can't, it copies it into the new block.

Now let's add in the bitand optimization. It's mostly some gross-looking pattern matching that checks if the right hand side of a bitwise and operation is 1 (TODO: the left hand side, too). CF had some neat ideas on how to make this more ergonomic, which I might save for later.3

Then, if we know the parity, optimize the bitand into a constant.

def simplify(block: Block) -> Block: parity = {v: BOTTOM for v in block} def parity_of(value): if isinstance(value, Constant): return Parity.const(value) return parity[value] result = Block() for op in block: # Try to simplify if isinstance(op, Operation) and op.name == "bitand": arg = op.arg(0) mask = op.arg(1) if isinstance(mask, Constant) and mask.value == 1: if parity_of(arg) is EVEN: op.make_equal_to(Constant(0)) continue elif parity_of(arg) is ODD: op.make_equal_to(Constant(1)) continue # Emit result.append(op) # Analyze transfer = getattr(Parity, op.name) args = [parity_of(arg.find()) for arg in op.args] parity[op] = transfer(*args) return result

Remember: because we use union-find to rewrite instructions in the optimizer (make_equal_to), later uses of the same instruction get the new optimized version "for free" (find).

Let's see how it works on our IR:

v0 = getarg(0) v1 = getarg(1) v2 = lshift(v0, 1) v3 = lshift(v1, 1) v4 = add(v2, v3) v6 = dummy(0)

Hey, neat! bitand disappeared and the argument to dummy is now the constant 0 because we know the lowest bit.

Wrapping up

Hopefully you have gained a little bit of an intuitive understanding of abstract interpretation. Last year, being able to write some code made me more comfortable with the math. Now being more comfortable with the math is helping me write the code. It's nice upward spiral.

The two abstract domains we used in this post are simple and not very useful in practice but it's possible to get very far using slightly more complicated abstract domains. Common domains include: constant propagation, type inference, range analysis, effect inference, liveness, etc. For example, here is a a sample lattice for constant propagation:

"-inf"; bottom -> "-2"; bottom -> "-1"; bottom -> 0; bottom -> 1; bottom -> 2; bottom -> "+inf"; "-inf" -> negative; "-2" -> negative; "-1" -> negative; 0 -> top; 1 -> nonnegative; 2 -> nonnegative; "+inf" -> nonnegative; negative -> nonzero; nonnegative -> nonzero; nonzero->top; {rank=same; "-inf"; "-2"; "-1"; 0; 1; 2; "+inf"} {rank=same; nonnegative; negative;} } -->

It has multiple levels to indicate more and less precision. For example, you might learn that a variable is either 1 or 2 and be able to encode that as nonnegative instead of just going straight to top.

Check out some real-world abstract interpretation in open source projects:

• Known bits in LLVM
• Constant range in LLVM
• But I am told that the ranges don't form a lattice (see Interval Analysis and Machine Arithmetic: Why Signedness Ignorance Is Bliss)
• Tristate numbers for known bits in Linux eBPF
• Range analysis in Linux eBPF
• GDB prologue analysis of assembly to understand the stack and find frame pointers without using DWARF (some docs)

If you have some readable examples, please share them so I can add.

Acknowledgements

Thank you to CF Bolz-Tereick for the toy optimizer and helping edit this post!

1. In the words of abstract interpretation researchers Vincent Laviron and Francesco Logozzo in their paper Refining Abstract Interpretation-based Static Analyses with Hints (APLAS 2009):

   The three main elements of an abstract interpretation are: (i) the abstract elements ("which properties am I interested in?"); (ii) the abstract transfer functions ("which is the abstract semantics of basic statements?"); and (iii) the abstract operations ("how do I combine the abstract elements?").

   We don't have any of these "abstract operations" in this post because there's no control flow but you can read about them elsewhere! ↩

2. These abstract values are arranged in a lattice, which is a mathematical structure with some properties but the most important ones are that it has a top, a bottom, a partial order, a meet operation, and values can only move in one direction on the lattice.

   Using abstract values from a lattice promises two things:

   • The analysis will terminate
   • The analysis will be correct for any run of the program, not just one sample run

   ↩

3. Something about __match_args__ and @property... ↩

Transformers is a powerful Python library created by Hugging Face that allows you to download, manipulate, and run thousands of pretrained, open-source AI models. These models cover multiple tasks across modalities like natural language processing, computer vision, audio, and multimodal learning. Using pretrained open-source models can reduce costs, save the time needed to train models from scratch, and give you more control over the models you deploy.

In this tutorial, you’ll learn how to:

• Navigate the Hugging Face ecosystem
• Download, run, and manipulate models with Transformers
• Speed up model inference with GPUs

Throughout this tutorial, you’ll gain a conceptual understanding of Hugging Face’s AI offerings and learn how to work with the Transformers library through hands-on examples. When you finish, you’ll have the knowledge and tools you need to start using models for your own use cases. Before starting, you’ll benefit from having an intermediate understanding of Python and popular deep learning libraries like pytorch and tensorflow.

Get Your Code: Click here to download the free sample code that shows you how to use Hugging Face Transformers to leverage open-source AI in Python.

Take the Quiz: Test your knowledge with our interactive “Hugging Face Transformers” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Hugging Face Transformers

In this quiz, you'll test your understanding of the Hugging Face Transformers library. This library is a popular choice for working with transformer models in natural language processing tasks, computer vision, and other machine learning applications.

The Hugging Face Ecosystem

Before using Transformers, you’ll want to have a solid understanding of the Hugging Face ecosystem. In this first section, you’ll briefly explore everything that Hugging Face offers with a particular emphasis on model cards.

Exploring Hugging Face

Hugging Face is a hub for state-of-the-art AI models. It’s primarily known for its wide range of open-source transformer-based models that excel in natural language processing (NLP), computer vision, and audio tasks. The platform offers several resources and services that cater to developers, researchers, businesses, and anyone interested in exploring AI models for their own use cases.

There’s a lot you can do with Hugging Face, but the primary offerings can be broken down into a few categories:

• Models: Hugging Face hosts a vast repository of pretrained AI models that are readily accessible and highly customizable. This repository is called the Model Hub, and it hosts models covering a wide range of tasks, including text classification, text generation, translation, summarization, speech recognition, image classification, and more. The platform is community-driven and allows users to contribute their own models, which facilitates a diverse and ever-growing selection.

• Datasets: Hugging Face has a library of thousands of datasets that you can use to train, benchmark, and enhance your models. These range from small-scale benchmarks to massive, real-world datasets that encompass a variety of domains, such as text, image, and audio data. Like the Model Hub, 🤗 Datasets supports community contributions and provides the tools you need to search, download, and use data in your machine learning projects.

• Spaces: Spaces allows you to deploy and share machine learning applications directly on the Hugging Face website. This service supports a variety of frameworks and interfaces, including Streamlit, Gradio, and Jupyter notebooks. It is particularly useful for showcasing model capabilities, hosting interactive demos, or for educational purposes, as it allows you to interact with models in real time.

• Paid offerings: Hugging Face also offers several paid services for enterprises and advanced users. These include the Pro Account, the Enterprise Hub, and Inference Endpoints. These solutions offer private model hosting, advanced collaboration tools, and dedicated support to help organizations scale their AI operations effectively.

These resources empower you to accelerate your AI projects and encourage collaboration and innovation within the community. Whether you’re a novice looking to experiment with pretrained models, or an enterprise seeking robust AI solutions, Hugging Face offers tools and platforms that cater to a wide range of needs.

This tutorial focuses on Transformers, a Python library that lets you run just about any model in the Model Hub. Before using transformers, you’ll need to understand what model cards are, and that’s what you’ll do next.

Understanding Model Cards

Model cards are the core components of the Model Hub, and you’ll need to understand how to search and read them to use models in Transformers. Model cards are nothing more than files that accompany each model to provide useful information. You can search for the model card you’re looking for on the Models page:

Hugging Face Models page

On the left side of the Models page, you can search for model cards based on the task you’re interested in. For example, if you’re interested in zero-shot text classification, you can click the Zero-Shot Classification button under the Natural Language Processing section:

Hugging Face Models page filtered for zero-shot text classification models

In this search, you can see 266 different zero-shot text classification models, which is a paradigm where language models assign labels to text without explicit training or seeing any examples. In the upper-right corner, you can sort the search results based on model likes, downloads, creation dates, updated dates, and popularity trends.

Each model card button tells you the model’s task, when it was last updated, and how many downloads and likes it has. When you click a model card button, say the one for the facebook/bart-large-mnli model, the model card will open and display all of the model’s information:

A Hugging Face model card

Even though a model card can display just about anything, Hugging Face has outlined the information that a good model card should provide. This includes detailed information about the model, its uses and limitations, the training parameters and experiment details, the dataset used to train the model, and the model’s evaluation performance.

A high-quality model card also includes metadata such as the model’s license, references to the training data, and links to research papers that describe the model in detail. In some model cards, you’ll also get to tinker with a deployed instance of the model via the Inference API. You can see an example of this in the facebook/bart-large-mnli model card:

Tinker with Hugging Face models using the Inference API Read the full article at https://realpython.com/huggingface-transformers/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of Python’s logging module.

Logging is a very useful tool in a programmer’s toolbox. It can help you develop a better understanding of the flow of a program and discover scenarios that you might not have thought of while developing.

Logs provide developers with an extra set of eyes that are constantly looking at the flow an application is going through.

They can store information, like which user or IP accessed the application. If an error occurs, then they can provide more insights than a stack trace by telling you what the state of the program was before it arrived and the line of code where the error occurred.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Test your understanding of how to create and use Python protocols while providing type hints for your functions, variables, classes, and methods.

Take this quiz after reading our Python Protocols: Leveraging Structural Subtyping tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of Hugging Face Transformers. This library is a popular choice for working with transformer models in natural language processing tasks, computer vision, and other machine learning applications.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Django 5.1 release candidate 1 is the final opportunity for you to try out a kaleidoscope of improvements before Django 5.1 is released.

The release candidate stage marks the string freeze and the call for translators to submit translations. Provided no major bugs are discovered that can't be solved in the next two weeks, Django 5.1 will be released on or around August 7. Any delays will be communicated on the on the Django forum.

Please use this opportunity to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the release candidate package from our downloads page or on PyPI.

The PGP key ID used for this release is Natalia Bidart: 2EE82A8D9470983E

Author: Reshama Shaikh , Adam Li

BIO: Adam is currently a Postdoctoral Research Scientist at Columbia University in the Causal Artificial Intelligence Lab, directed by Dr. Elias Bareinboim. He is an NSF-funded Computing Innovation Research Fellow. He did his PhD in biomedical engineering, specializing in computational neuroscience and machine learning at Johns Hopkins University working with Dr. Sridevi V. Sarma in the Neuromedical Control Systems group. He also jointly obtained a MS in Applied Mathematics and Statistics with a focus in statistical learning theory, optimization and matrix analysis. He was fortunate to be a NSF-GRFP fellow, Whitaker International Fellow, Chateaubriand Fellow and ARCS Chapter Scholar during his time at JHU. Adam officially joined the scikit-learn team as a maintainer in July 2024.

• GitHub: @adam2392
• LinkedIn: @adam2392
• Website: https://adam2392.github.io

Link to scikit-learn contributions (issues, pull requests):

• FEA Add missing-value support for ExtaTreeClassifier and ExtaTreeRegressor
• DOC Fix tree explanation of tree_.value in example
• ENH Enable prediction of isolation forest in parallel
• ENH Adding estimators_samples_ attribute to forest models
• FEA SLEP006: Metadata routing for SelfTrainingClassifier
• FEAT SLEP006 permutation_test_score to support metadata routing
• FEA Categorical split support for DecisionTree, ExtraTree, RandomForest* and `ExtraTrees* #29437
• Issue: Adding Oblique Trees (Forest-RC) to the Cythonized Tree Module

1. Tell us about yourself.

   I currently live in New York City, where I work on theoretical and applied AI research through the lens of causal inference, statistical modeling, dynamical systems and signal processing. My current research is focused on telling a causal story, specifically in the case one has multiple distributions of data from the same causal system. For example, one may have access to brain recordings from monkeys and humans. Given these heterogeneous datasets, I am interested in answering: what causal relationships can we learn. This is known as the causal discovery problem, where given data, one attempts to learn what causes what. Another problem that I work on that is highly relevant to generative AI is the problem of causal representation learning. Here, I develop theory and train deep neural networks to understand causality among latent factors. Specifically, we demonstrate how to leverage multiple datasets and a causal neural network to generate data that is causally realistic. This can enable more robust data generation from general latent variable models.

2. How did you first become involved in open source and scikit-learn?

   I first got involved in open source as a user. I was making the switch from Matlab to Python and started using packages like numpy and scipy pretty regularly. In my PhD research, I dealt with a lot of electrophysiological data (i.e. EEG brain recordings). I was writing hundreds of lines of code to load and preprocess data, and it was always changing based on different constraints. That was when I discovered MNE-BIDS, a Python package within the MNE framework for reading and writing brain recording data in a structured format. This changed my life because now my preprocessing and data loading code was a few lines of code that adhered to an open standard tested by thousands of researchers. I realized the value of open source, and began contributing in my spare time.

3. We would love to learn of your open source journey.

   I first started contributing to open-source in the MNE organization. This package implements data structures for the processing and analysis of neural recording data (e.g. MEG, EEG, iEEG data). I contributed over 70 pull requests in the MNE-BIDS package, and subsequently was invited to be a maintainer for MNE-BIDS and MNE-Python. Later one, I participated in a Google Summer of Code to port the connectivity submodule within MNE-Python to a new package, known as MNE-Connectivity. I added new data structures, and algorithms for the sake of improving the feature developments for connectivity algorithms among neural recording data. Later on, I also worked with a team on porting a neural network architecture from Matlab to the MNE framework to automatically classify ICA derived components. This became known as MNE-ICALabel. These experiences gave me the experience necessary to work in a large asynchronous team environment that is common in OSS. It also taught me how to begin contributing to an OSS project. This led me to scikit-learn.

   I first got involved in scikit-learn as a user, who was heavily interested in the decision tree model in scikit-learn (random forest, randomized trees). Here, I was interested in contributing a new oblique decision tree model that was a generalization of the existing random forest model. However, the code was not easily added to scikit-learn, and currently the decision to include it is inconclusive. Throughout this process, I learned about the challenges and intricacies of maintaining such a large OSS project as scikit-learn. It is not trivial to simply add new features to a large OSS project because code comes with a maintenance cost, and should fit with the current internal design. At this point in time, there were very few maintainers that were able to maintain the tree submodule, and as such new features are included conservatively.

   I was eager to improve the project to enable more exciting features for the community, so I began contributing to scikit-learn starting with smaller issues such as documentation improvements, or minor bug fixes to get acquainted with the codebase. I also refactored various Cython code to begin upgrading the codebase, especially in the tree submodule. Throughout this process, I identified other projects the maintainers team were working on, and also contributed there. For example, I added metadata routing to a variety of different functions and estimators in scikit-learn. I also began reviewing PRs for the tree submodule and metadata routing where I had knowledge. I also added missing-value support for extremely randomized tree models (called ExtraTrees in scikit-learn). This allows users to pass in data that contains missing values (encoded as np.nan) to ExtraTrees. Around this time, I was invited to join the maintainer team of scikit-learn. More recently, I have taken on the project to add categorical data support to the decision tree models, which will make random forests and extremely randomized tree models more performant and capable to handle real world settings where there is commonly categorical data.

4. To which OSS projects and communities do you contribute?

   I currently primarily contribute to scikit-learn, PyWhy (a community for causal inference in Python), and also develop my own OSS project: treeple. Treeple is an exciting package that implements different decision tree models beyond those offered in scikit-learn with an efficient Cython implementation stemming from the scikit-learn tree internals.

5. What do you find alluring about OSS?

   OSS is so exciting because of the impact it has. Everyone from private projects to other OSS projects will use OSS. Any fixes to documentation, performance improvements, or new features will potentially impact the workflows of potentially millions of people. This is what makes contributing to OSS so exciting. Moreover, this impact ensures that best practices are usually carried out in these projects, and it’s a great playground to learn from the best, while giving back to the larger community.

6. What pain points do you observe in community-led OSS?

   Right now, community lead OSS moves very slowly in most places. This is for a number of very good reasons: i) not releasing buggy features that may impact millions of people, and ii) backwards compatibility. One of the challenges of maintaining a high-quality OSS project is that you would like to satisfy your users, who may all utilize different components of the project from different versions. As such, many community led OSS projects take a conservative approach when implementing new features and new ideas. However, there may be many exciting better features that are already known by the community, but still lack an OSS implementation.

   I think this can be partially solved by increased funding for OSS, so OSS maintainers and developers are able to dedicate more time to maintaining and improving the projects. In addition, I think this can be improved if more developers in the community contribute to said OSS projects. I hope that I have convinced you though that contributing to OSS is impactful and highly educational.

7. If we discuss how far OS has evolved in 10 years, what would you like to see happen?

   I think more interoperability and integrated workflows for projects will make projects that utilize OSS more streamlined and efficient. For example, right now there are different array libraries (e.g. numpy, cupy, xarray, pytorch, etc.), which all support some manner of a n-dimensional array, but with a slightly different API. This makes it very painful to transition across different libraries that use different arrays. In addition, there are multiple dataframe libraries, such as pandas and polars, and this problem of API consistency also arises there.

   Some work has been made on the Array-API front to allow different array libraries to serve as backends given a common API. This will enable GPU acceleration for free without a single code change, which is great! This will be exciting because users will eventually only have to write code in a single way, and can then leverage any array/dataframe library that has different advantages and disadvantages based on the user use case.

8. What are your hobbies, outside of work and open source?

   I enjoy running, trying new restaurants and bars, cooking and reading. I’m currently training for a half-marathon, where my goal is to run under 8 minutes per mile. I’m also trying to perfect a salad with an asian-themed dressing. In a past life, I was a bboy (breakdancer) for ten years until I stopped in graduate school because I got busy (and old).

#639 – JULY 23, 2024
View in Browser »

Asyncio gather() Handle Exceptions

The asyncio.gather() function takes an optional argument return_exceptions which changes the behavior of the co-routine when an error occurs. This article shows you how to use it.
JASON BROWNLEE

Python Protocols: Leveraging Structural Subtyping

In this tutorial, you’ll learn about Python’s protocols and how they can help you get the most out of using Python’s type hint system and static type checkers.
REAL PYTHON

Prod Alerts? You Should be Autoscaling

Let Judoscale solve your scaling issues. We support Django, Flask, and FastAPI, and we also autoscale your Celery and RQ task queues. Traffic spike? Scaled up. Quiet night? Scaled down. Work queue backlog? No problem →
JUDOSCALE sponsor

Trying Out Free-Threaded Python on macOS

Free-threaded mode is available as a compile option in the beta of Python 3.13. Simon decided to try it out on his Mac.
SIMON WILLISON

Quiz: Python’s Built-in Functions: A Complete Exploration

Take this quiz to test your knowledge about the available built-in functions in Python. By taking this quiz, you’ll deepen your understanding of how to use these functions and the common programming problems they cover, from mathematical computations to Python-specific features.
REAL PYTHON

Python 3.13.0 Beta 4 Released

CPYTHON DEV BLOG

Articles & Tutorials Exercises Course: Introduction to Web Scraping With Python

In this course, you’ll practice the main steps of the web scraping process. You’ll write a script that uses Python’s requests library to scrape and parse data from a website. You’ll also interact with HTML forms using tools like Beautiful Soup and Mechanical Soup to extract specific information.
REAL PYTHON course

When Generators Get Cleaned Up

When playing with generators in asynchronous code, Sam ran into some things he didn’t quite expect with how and when resources get cleaned up. This article shows you what you can and can’t control, and in some cases whether you should.
SAM EISENHANDLER

Posit Connect - Help Your Data Science Team Share and Collaborate

Tired of emailing files around & trying to use general-purpose collaboration tools? Posit Connect makes it easy to share, collaborate, & get feedback on your data science work including Jupyter notebooks, Streamlit, & other analytics applications.
POSIT sponsor

Different Ways I Start Writing New Code

In this article, the author talks about the different approaches they use to create new code depending on the mood, the feature or bug at hand, and their confidence level. Five different approaches are covered.
JUHA-MATTI SANTALA

A Python Epoch Timestamp Timezone Trap

Date and times are problematic and there are all sorts of problems you can get into when you use them in Python. This post talks about one challenge when using epoch timestamp numbers.
JOËL PERRAS

How to Convert a Python Script Into a Web App

This post shows you how to take a Python script and turn it into a web application using FastAPI. It includes information about how to choose endpoints and how to do authentication.
AHMED LEMINE

Instrumenting Python GIL With eBPF

eBPF is a tool available on Linux to dig into the profile of your code. In this article, Nikolay uses it to inspect when the GIL gets invoked in CPython.
NIKOLAY SIVOK

Approximate Counting in Django and Postgres

Pagination in Django uses the rather slow SELECT COUNT(*) SQL call. This article looks at how to speed up counting with Django and PostgreSQL.
NIK TOMAZIC

Assignment vs. Mutation in Python

In Python, “change” can mean two different things. Assignment changes which object a variable points to. Mutation, changes the object itself.
TREY HUNNER

Flask vs Django in 2024

This post compares Flask and Django, from what they specialize in to what it takes to do a personal web site in each.
WILL VINCENT

Making an Iterator Out of a Function

You can use the Python built-in function iter with two arguments to create an iterator from a function.
RODRIGO GIRÃO SERRÃO

Developing GraphQL APIs in Django With Strawberry

This tutorial details how to integrate GraphQL with Django using Strawberry.
TESTDRIVEN.IO • Shared by Michael Herman

Projects & Code cbfa: Class-Based Views for FastAPI

GITHUB.COM/POMPONCHIK • Shared by Evgeniy Blinov (pomponchik)

Satyrn: macOS App for Jupyter Nobtebooks

SATYRN.APP

Datasketch: Probabilistic Data Structures

PYPI.ORG

pyNES: Python Programming for Nintendo 8 Bits

GITHUB.COM/GUTOMAIA

staged-script: Divide Automation Scripts Into Stages

GITHUB.COM/SANDIALABS • Shared by Jason M. Gates

Events Weekly Real Python Office Hours Q&A (Virtual)

July 24, 2024
REALPYTHON.COM

SPb Python Drinkup

July 25, 2024
MEETUP.COM

PyOhio 2024

July 27 to July 28, 2024
PYOHIO.ORG

PyDelhi User Group Meetup

July 27, 2024
MEETUP.COM

Lightning Talks

July 27, 2024
MEETUP.COM

Happy Pythoning!
This was PyCoder’s Weekly Issue #639.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

ObjectListView is a third-party wxPython widget that wraps the wx.ListCtrl. I have used it for over 10 years in quite a few different GUI applications because it works much nicer than wx.ListCtrl does. Unfortunately, ObjectListView was never integrated into wxPython core like some other amazing third-party packages were, and so it has become broken over the past couple of years such that you can’t use it with the latest version of Python / wxPython.

So, I decided to fork the project, as its defects were simple enough that I could resolve them quickly. The new version of ObjectListView is now called ObjectListView3.

You can get the new version of ObjectListview here:

• ObjectListView3 on the Python Packaging Index
• ObjectListView3 on GitHub

Note that this version of ObjectListView works with Python 3.11+ and wxPython 4.2+. It may work with earlier versions as well.

Installation

If you’d like to install ObjectListView3, you can use Python’s pip to do so like this:

python -m pip install ObjectListView3

Now let’s see how an example of using ObjectListView!

Sample Usage

The following is an application that starts out by listing two books in an ObjectListView3 widget. When you press the “Update OLV” button, it will add three more books to the ObjectListView3 widget.

This demonstrates how to create the ObjectListview3 widget and update it after the application is running:

import wx from ObjectListView3 import ObjectListView, ColumnDefn class Book(object): """ Model of the Book object Contains the following attributes: 'ISBN', 'Author', 'Manufacturer', 'Title' """ def __init__(self, title, author, isbn, mfg): self.isbn = isbn self.author = author self.mfg = mfg self.title = title class MainPanel(wx.Panel): def __init__(self, parent): super().__init__(parent=parent, id=wx.ID_ANY) self.products = [ Book("wxPython in Action", "Robin Dunn", "1932394621", "Manning"), Book("Hello World", "Warren and Carter Sande", "1933988495", "Manning"), ] self.dataOlv = ObjectListView( self, wx.ID_ANY, style=wx.LC_REPORT | wx.SUNKEN_BORDER ) self.setBooks() # Allow the cell values to be edited when double-clicked self.dataOlv.cellEditMode = ObjectListView.CELLEDIT_SINGLECLICK # create an update button updateBtn = wx.Button(self, wx.ID_ANY, "Update OLV") updateBtn.Bind(wx.EVT_BUTTON, self.updateControl) # Create some sizers mainSizer = wx.BoxSizer(wx.VERTICAL) mainSizer.Add(self.dataOlv, 1, wx.ALL | wx.EXPAND, 5) mainSizer.Add(updateBtn, 0, wx.ALL | wx.CENTER, 5) self.SetSizer(mainSizer) def updateControl(self, event): """ Update the ObjectListView """ product_dict = [ { "title": "Core Python Programming", "author": "Wesley Chun", "isbn": "0132269937", "mfg": "Prentice Hall", }, { "title": "Python Programming for the Absolute Beginner", "author": "Michael Dawson", "isbn": "1598631128", "mfg": "Course Technology", }, { "title": "Learning Python", "author": "Mark Lutz", "isbn": "0596513984", "mfg": "O'Reilly", }, ] data = self.products + product_dict self.dataOlv.SetObjects(data) def setBooks(self, data=None): self.dataOlv.SetColumns( [ ColumnDefn("Title", "left", 220, "title"), ColumnDefn("Author", "left", 200, "author"), ColumnDefn("ISBN", "right", 100, "isbn"), ColumnDefn("Mfg", "left", 180, "mfg"), ] ) self.dataOlv.SetObjects(self.products) class MainFrame(wx.Frame): def __init__(self): wx.Frame.__init__( self, parent=None, id=wx.ID_ANY, title="ObjectListView Demo", size=(800, 600), ) panel = MainPanel(self) class OLVDemoApp(wx.App): def __init__(self, redirect=False, filename=None): super().__init__(redirect, filename) def OnInit(self): # create frame here frame = MainFrame() frame.Show() return True def main(): """ Run the demo """ app = OLVDemoApp() app.MainLoop() if __name__ == "__main__": main()

When you initially run this code, you will see the following application:

When you press the “Update OLV” button, the application will update to look like the following:

Wrapping Up

The ObjectListView3 widget makes working with tabular data easy in wxPython. You also get nice editing, sorting and more from this widget via different mixins or styles. Give ObjectListView3 a try when you create your next wxPython GUI application.

The post ANN: ObjectListView3 for wxPython appeared first on Mouse Vs Python.

Whether you’ve just started working with pandas and want to master one of its core capabilities, or you’re looking to fill in some gaps in your understanding about .groupby(), this course will help you to break down and visualize a pandas GroupBy operation from start to finish.

This course is meant to complement the official pandas documentation and the pandas Cookbook, where there are self-contained, bite-sized examples. Here, however, you’ll focus on three more involved walkthroughs that use real-world datasets.

In this course, you’ll cover:

• How to use pandas GroupBy operations on real-world data
• How the split-apply-combine chain of operations works
• How to decompose the split-apply-combine chain into steps
• How to categorize methods of a pandas GroupBy object based on their intent and result

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

<strong>Topics covered in this episode:</strong><br> <ul> <li><a href="https://marimo.io"><strong>Marimo:</strong></a><a href="https://marimo.io"> </a><a href="https://marimo.io"><strong>“Future</strong></a><a href="https://marimo.io"> <strong>of Notebooks”</strong></a></li> <li><a href="https://docs.pytest.org/en/stable/changelog.html#pytest-8-3-1-2024-07-20"><strong>pytest 8.3.0 &amp; 8.3.1 are out</strong></a></li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024.html"><strong>Python Language Summit 2024</strong></a></li> <li><a href="https://github.com/wolandark/bash-dungeon/"><strong>bash-dungeon</strong></a></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=pwTJpSvrGls' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="393">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by us! Support our work through:</p> <ul> <li>Our <a href="https://training.talkpython.fm/"><strong>courses at Talk Python Training</strong></a></li> <li><a href="https://courses.pythontest.com/p/the-complete-pytest-course"><strong>The Complete pytest Course</strong></a></li> <li><a href="https://www.patreon.com/pythonbytes"><strong>Patreon Supporters</strong></a></li> </ul> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually Tuesdays at 10am PT. Older video versions available there too.</p> <p>Finally, if you want an artisanal, hand-crafted digest of every week of the show notes in email form? Add your name and email to <a href="https://pythonbytes.fm/friends-of-the-show">our friends of the show list</a>, we'll never share it.</p> <p><strong>Michael #1:</strong> <a href="https://marimo.io"><strong>Marimo:</strong></a><a href="https://marimo.io"> </a><a href="https://marimo.io"><strong>“Future</strong></a><a href="https://marimo.io"> <strong>of Notebooks”</strong></a></p> <ul> <li>via Matt Wilkie</li> <li>An <strong>open-source</strong> reactive notebook for Python</li> <li>Run one cell and marimo reacts by <strong>automatically running affected cells</strong>, eliminating the error-prone chore of managing notebook state.</li> <li>Marimo's reactive <strong>UI elements</strong>, like dataframe GUIs and plots, make working with data feel refreshingly fast, futuristic, and intuitive.</li> <li>Rapidly experiment with code and models</li> <li>Bind UI elements to Python values</li> <li>Pick-up-and-play design, with depth for power users</li> <li>See <a href="https://docs.marimo.io/faq.html">the FAQ</a></li> </ul> <p><strong>Brian #2:</strong> <a href="https://docs.pytest.org/en/stable/changelog.html#pytest-8-3-1-2024-07-20"><strong>pytest 8.3.0 &amp; 8.3.1 are out</strong></a></p> <ul> <li>Real excited to get --xfail-tb flag added <ul> <li>This detaches xfail tracebacks from -rx/-ra (which was how it was pre-8.0)</li> </ul></li> <li>Keyword matching for marker expressions, that’s fun. <ul> <li>pytest -v -m "device(serial='123')"</li> </ul></li> <li>--no-fold-skipped allows for explit reporting of names of skipped tests</li> <li>Plus many more improvements, bug fixes, and doc improvements</li> </ul> <p><strong>Michael #3:</strong> <a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024.html"><strong>Python Language Summit 2024</strong></a></p> <ul> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-should-python-adopt-calver.html">Should Python adopt Calendar Versioning?</a>: talk by Hugo van Kemenade</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-python-security-model-after-xz.html">Python's security model after the xz-utils backdoor</a>: talk by Pablo Galindo Salgado</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-c-api.html">Native Interface and Limited C API</a>: talks by Petr Viktorin and Victor Stinner</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-free-threading-ecosystems.html">Free-threading ecosystems</a>: talk by Daniele Parmeggiani</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-python-on-mobile.html">Python on Mobile</a>: talk by Malcolm Smith</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-pyrepl-new-default-repl-for-python.html">PyREPL</a><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-pyrepl-new-default-repl-for-python.html"> </a><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-pyrepl-new-default-repl-for-python.html">--</a><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-pyrepl-new-default-repl-for-python.html"> New default REPL written in Python</a>: talk by Pablo Galindo Salgado, Łukasz Langa, and Lysandros Nikolaou</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-pyrepl-new-pdb.html">Should we make pdb better?</a>: talk by Tian Gao</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-limiting-yield-in-async-generators.html">Limiting yield in async generators</a>: talk by Zac Hatfield-Dodds</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-annotations-as-transforms.html">Annotations as Transforms</a>: talk by Jason R. Coombs</li> <li><a href="https://pyfound.blogspot.com/2024/06/python-language-summit-2024-lightning-talks.html">Lightning Talks</a>, featuring talks by Petr Viktorin, David Hewitt, Emily Morehouse, Łukasz Langa, Pablo Galindo Salgado, and Yury Selivanov</li> </ul> <p><strong>Brian #4:</strong> <a href="https://github.com/wolandark/bash-dungeon/"><strong>bash-dungeon</strong></a></p> <ul> <li>“<em>This game is intended to teach new users how to use their shell in a fun and interactive way.”</em></li> <li>Just clone the repo and start exploring with cd, ls, and cat.</li> <li>First moves <ul> <li>cd bash-dungeon</li> <li>ls</li> <li>cd Enter</li> <li>ls</li> <li>cat parchment</li> </ul></li> <li>A fun way to learn some commands you might need and/or might have forgotten about.</li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li><a href="https://discuss.python.org/t/python-3-13-0b4-now-available/58565">Python 3.12.0b4, final beta, is out</a></li> <li>If hanging out on discuss.python.org, please checkout <ul> <li><a href="https://discuss.python.org/guidelines/">Community Guidelines</a></li> </ul></li> <li>And if it’s still not clear why we need these, check out <ul> <li><a href="https://discuss.python.org/t/inclusive-communications-expectations-in-python-spaces/57950">Inclusive communications expectations in Python spaces</a></li> </ul></li> <li>Google Chrome news</li> </ul> <p>Michael:</p> <ul> <li><a href="https://fosstodon.org/@ddsmit/112793568816585948">PySimpleGUI goes commercial with obfuscated</a><a href="https://fosstodon.org/@ddsmit/112793568816585948"> </a><a href="https://fosstodon.org/@ddsmit/112793568816585948">“source</a><a href="https://fosstodon.org/@ddsmit/112793568816585948"> open”?</a></li> <li>Still have seats for <a href="https://talkpython.fm/castle">Code in a Castle event</a></li> <li><a href="https://training.talkpython.fm/courses/reactive-web-dashboards-with-shiny-for-data-science">Reactive Dashboards with Shiny for Python free course</a></li> </ul> <p><strong>Joke:</strong> </p> <ul> <li><a href="https://www.smbc-comics.com/comic/investment-2?utm_source=pocket_saves">40 Million in in Series A Funding</a> - may be a lot of reading, but I found it funny <ul> <li>Thanks to VM Brasseur for sharing this one.</li> </ul></li> <li>Also a few from <a href="https://pypi.org/project/pyjokes/0.7.2/#history">pyjokes 0.7.2</a> (first new version since 2019) <ul> <li>If at first you don't succeed, call it version 1.0.</li> <li>A product manager walks into a bar, asks for drink. Bartender says no, but will consider adding later.</li> <li>Triumphantly, Beth removed Python 2.7 from her server in 2030. 'Finally!' she said with glee, only to see the announcement for Python 4.4.1 <ul> <li>Although, if <a href="https://peps.python.org/pep-2026/">CalVer, PEP 2026</a>, happens, that’ll just be Python 3.30.0.</li> </ul></li> </ul></li> </ul>

Learn how to serve a static site using FastAPI. Perfect for locally testing statically generated websites, for instance, with `httpie`.

Meet the interns joining us for our third-annual summer internship.

In the last couple of days, I've created variations of a simple algorithm to demonstrate how Celcius and Fahrenheit seem to relate to each other if you "mirror the number".
It wasn't supposed to be about the programming language. Still, I used Python in the first one and I noticed that since the code is simple, it could be fun to write variants of it in other languages.

1. Converting Celsius to Fahrenheit with Python
2. Converting Celsius to Fahrenheit with TypeScript
3. Converting Celsius to Fahrenheit with Go
4. Converting Celsius to Fahrenheit with Ruby
5. Converting Celsius to Fahrenheit with Crystal
6. Converting Celsius to Fahrenheit with Rust

It was a fun exercise.

And speaking of fun, I couldn't help but to throw in a benchmark using hyperfine that measures, essentially, how fast these CLIs can start up. The results look like this:

Summary ./conversion-rs ran 1.31 ± 1.30 times faster than ./conversion-go 1.88 ± 1.33 times faster than ./conversion-cr 7.15 ± 4.64 times faster than bun run conversion.ts 14.27 ± 9.48 times faster than python3.12 conversion.py 18.10 ± 12.35 times faster than node conversion.js 67.75 ± 43.80 times faster than ruby conversion.rb

It doesn't prove much, that you didn't expect. But it's fun to see how fast Python 3.12 has become at starting up.

Head on over to https://github.com/peterbe/temperature-conversion to play along. Perhaps you can see some easy optimizations (speed and style).

Recording relevant information during the execution of your program is a good practice as a Python developer when you want to gain a better understanding of your code. This practice is called logging, and it’s a very useful tool for your programmer’s toolbox. It can help you discover scenarios that you might not have thought of while developing.

These records are called logs, and they can serve as an extra set of eyes that are constantly looking at your application’s flow. Logs can store information, like which user or IP accessed the application. If an error occurs, then logs may provide more insights than a stack trace by telling you the state of the program before the error and the line of code where it occurred.

Python provides a logging system as part of its standard library. You can add logging to your application with just a few lines of code.

In this tutorial, you’ll learn how to:

• Work with Python’s logging module
• Set up a basic logging configuration
• Leverage log levels
• Style your log messages with formatters
• Redirect log records with handlers
• Define logging rules with filters

When you log useful data from the right places, you can debug errors, analyze the application’s performance to plan for scaling, or look at usage patterns to plan for marketing.

You’ll do the coding for this tutorial in the Python standard REPL. If you prefer Python files, then you’ll find a full logging example as a script in the materials of this tutorial. You can download this script by clicking the link below:

Get Your Code: Click here to download the free sample code that you’ll use to learn about logging in Python.

Take the Quiz: Test your knowledge with our interactive “Logging in Python” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Logging in Python

In this quiz, you'll test your understanding of Python's logging module. With this knowledge, you'll be able to add logging to your applications, which can help you debug errors and analyze performance.

If you commonly use Python’s print() function to get information about the flow of your programs, then logging is the natural next step for you. This tutorial will guide you through creating your first logs and show you how to make logging grow with your projects.

Starting With Python’s Logging Module

The logging module in Python’s standard library is a ready-to-use, powerful module that’s designed to meet the needs of beginners as well as enterprise teams.

Note: Since logs offer a variety of insights, the logging module is often used by other third-party Python libraries, too. Once you’re more advanced in the practice of logging, you can integrate your log messages with the ones from those libraries to produce a homogeneous log for your application.

To leverage this versatility, it’s a good idea to get a better understanding of how the logging module works under the hood. For example, you could take a stroll through the logging module’s source code

The main component of the logging module is something called the logger. You can think of the logger as a reporter in your code that decides what to record, at what level of detail, and where to store or send these records.

Exploring the Root Logger

To get a first impression of how the logging module and a logger work, open the Python standard REPL and enter the code below:

Python >>> import logging >>> logging.warning("Remain calm!") WARNING:root:Remain calm! Copied!

The output shows the severity level before each message along with root, which is the name the logging module gives to its default logger. This output shows the default format that can be configured to include things like a timestamp or other details.

In the example above, you’re sending a message on the root logger. The log level of the message is WARNING. Log levels are an important aspect of logging. By default, there are five standard levels indicating the severity of events. Each has a corresponding function that can be used to log events at that level of severity.

Note: There’s also a NOTSET log level, which you’ll encounter later in this tutorial when you learn about custom logging handlers.

Here are the five default log levels, in order of increasing severity:

Log Level Function Description DEBUG logging.debug() Provides detailed information that’s valuable to you as a developer. INFO logging.info() Provides general information about what’s going on with your program. WARNING logging.warning() Indicates that there’s something you should look into. ERROR logging.error() Alerts you to an unexpected problem that’s occured in your program. CRITICAL logging.critical() Tells you that a serious error has occurred and may have crashed your app.

The logging module provides you with a default logger that allows you to get started with logging without needing to do much configuration. However, the logging functions listed in the table above reveal a quirk that you may not expect:

Python >>> logging.debug("This is a debug message") >>> logging.info("This is an info message") >>> logging.warning("This is a warning message") WARNING:root:This is a warning message >>> logging.error("This is an error message") ERROR:root:This is an error message >>> logging.critical("This is a critical message") CRITICAL:root:This is a critical message Copied!

Notice that the debug() and info() messages didn’t get logged. This is because, by default, the logging module logs the messages with a severity level of WARNING or above. You can change that by configuring the logging module to log events of all levels.

Read the full article at https://realpython.com/python-logging/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

If you want to get better at something, often times the path is pretty clear. If you get better at swimming, you go to the pool and practice your strokes and put in time doing the laps. If you want to get better at mountain biking, hit the trails and work on drills focusing on different aspects of riding. You can do the same for programming. Reuven Lerner is back on the podcast to talk about his book Pandas Workout. We dive into strategies for learning Pandas and Python as well as some of his workout exercises.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/scalablepath'>Scalable Path</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Reuven Lerner on Twitter</b>: <a href="https://twitter.com/reuvenmlerner" target="_blank" rel="noopener">@reuvenmlerner</a><br/> <b>Pandas Workout Book</b>: <a href="https://www.manning.com/books/pandas-workout" target="_blank" rel="noopener">manning.com</a><br/> <b>Bamboo Weekly: Solar eclipse</b>: <a href="https://www.bambooweekly.com/bw-61-solar-eclipse/" target="_blank" rel="noopener">bambooweekly.com</a><br/> <b>Bamboo Weekly: Avocado hand</b>: <a href="https://www.bambooweekly.com/bw-73-avocado-hand/" target="_blank" rel="noopener">bambooweekly.com</a><br/> <b>Scaling data science across Python and R</b>: <a href="https://talkpython.fm/episodes/show/236/scaling-data-science-across-python-and-r" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=x1-t4xRS9XA" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/471/learning-and-teaching-pandas" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

LDAP and Active Directory as Python API Services 2024-07-22, by Dariusz Suchojad

LDAP and Active Directory often play key a role in the management of a company's network resources yet it is not always very convenient to query a directory directly using the LDAP syntax and protocol that few people truly specialize in. This is why in this article we are using Zato to offer a REST API on top of directory services so that API clients can use REST and JSON instead.

Installing Zato

Start off by installing Zato - if you are not sure what to choose, pick the Docker Quickstart option and this will set up a working environment in a few minutes.

Creating connections

Once Zato is running, connections can be easily created in its Dashboard (by default, http://127.0.0.1:8183). Navigate to Connections -> Outgoing -> LDAP ..

.. and then click Create a new connection which will open a form as below:

The same form works for both regular LDAP and Active Directory - in the latter case, make sure that Auth type is set to NTLM.

The most important information is:

• User credentials
• Authentication type
• Server or servers to connect to

Note that if authentication type is not NTLM, user credentials can be provided using the LDAP syntax, e.g. uid=MyUser,ou=users,o=MyOrganization,dc=example,dc=com.

Right after creating a connection be sure to set its password too - the password asigned by default is a randomly generated one.

Pinging

It is always prudent to ping a newly created connection to ensure that all the information entered was correct.

Note that if you have more than one server in a pool then the first available one of them will be pinged - it is the whole pool that is pinged, not a particular part of it.

Active Directory as a REST service

As the first usage example, let's create a service that will translate JSON queries into LDAP lookups - given username or email the service will basic information about the person's account, such as first and last name.

Note that the conn object returned by client.get() below is capable of running any commands that its underlying Python library offers - in this case we are only using searches but any other operation can also be used, e.g. add or modify as well.

# -*- coding: utf-8 -*- # stdlib from json import loads # Bunch from bunch import bunchify # Zato from zato.server.service import Service # Where in the directory we expect to find the user search_base = 'cn=users, dc=example, dc=com' # On input, we are looking users up by either username or email search_filter = '(&(|(uid={user_info})(mail={user_info})))' # On output, we are interested in username, first name, last name and the person's email query_attributes = ['uid', 'givenName', 'sn', 'mail'] class ADService(Service): """ Looks up users in AD by their username or email. """ class SimpleIO: input_required = 'user_info' output_optional = 'message', 'username', 'first_name', 'last_name', 'email' response_elem = None skip_empty_keys = True def handle(self): # Connection name to use conn_name = 'My AD Connection' # Get a handle to the connection pool with self.out.ldap[conn_name].conn.client() as client: # Get a handle to a particular connection with client.get() as conn: # Build a filter to find a user by user_info = self.request.input['user_info'] user_filter = search_filter.format(user_info=user_info) # Returns True if query succeeds and has any information on output if conn.search(search_base, user_filter, attributes=query_attributes): # This is where the actual response can be found response = conn.entries # In this case, we expect at most one user matching input criteria entry = response[0] # Convert it to JSON for easier handling .. entry = entry.entry_to_json() # .. and load it from JSON to a Python dict entry = loads(entry) # Convert to a Bunch instance to get dot access to dictionary keys entry = bunchify(entry['attributes']) # Now, actually produce a JSON response. For simplicity's sake, # assume that users have only one of email or other attributes. self.response.payload.message = 'User found' self.response.payload.username = entry.uid[0] self.response.payload.first_name = entry.givenName[0] self.response.payload.last_name = entry.sn[0] self.response.payload.email = entry.mail[0] else: # No business response = no such user found self.response.payload.message = 'No such user'

After creating a REST channel, we can invoke the service from command line, thus confirming that we can offer the directory as a REST service:

$ curl "localhost:11223/api/get-user?user_info=MyOrganization\\MyUser" ; echo { "message": "User found", "username": "MyOrganization\\MyUser", "first_name": "First", "last_name": "Last", "email": "address@example.com" } $ More resources

➤ Python API integration tutorial
➤ What is a Network Packet Broker? How to automate networks in Python?
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

YouTube without YouTube Shorts About • Blog • Newsletter • Links YouTube without YouTube Shorts

Published 2024-07-22 by Seth Larson
Reading time: minutes

Back in October 2023 I wrote about how “For You” wasn't for me, and my dislike for infinitely-scrolling algorithmically-suggested content. In that post I mentioned that I removed YouTube from my phone due to the “YouTube Shorts” feature which functions similar to TikTok.

That didn't last long. I enjoy many creators on YouTube, so eventually I ended up reinstalling the app. But my problem persisted: I would find myself opening up YouTube and occasionally scrolling through Shorts for longer than I wanted to.

What I wanted was YouTube without Shorts, but there didn't seem to be any way to disable them in the app. I was complaining about this to a friend, who out of nowhere mentioned that disabling your “Watch History” will disable the YouTube Shorts infinite feed. My mind was blown! 🤯

No more YouTube Shorts! 🥳

I disabled my Watch History and deleted my existing history and sure enough, no Shorts! (also no home page, which I can also live without). Here's how to do it:

• Open YouTube app, select “You” (bottom-right on iOS)
• Settings cog in the top-right
• Manage all history, wait for the Google Account page to load
• Select “Turn Off” for YouTube History
• Select “Manage History” and delete all history
• Go back to the home page and see that Shorts no longer loads!
• Might have to restart the app, if I remember correctly I had to?

Lesson learned: complain about hyper-specific problems to your friends more often.

Thanks for reading! ♡ Did you find this article helpful and want more content like it? Get notified of new posts by subscribing to the RSS feed or the email newsletter.

This work is licensed under CC BY-SA 4.0