Planet Python
PyCoder’s Weekly: Issue #567 (March 7, 2023)
#567 – MARCH 7, 2023
View in Browser »
Python Enhancement Proposal 709 covers a change to how comprehensions are handled. Currently they are compiled as nested functions. Benchmarking shows that treating list, dict, and set comprehensions as in-line code can result in a 2x speedup on the comprehension.
PYTHON.ORG
In this tutorial, you’ll learn what iterators and iterables are in Python. You’ll learn how they differ and when to use them in your code. You’ll also learn how to create your own iterators and iterables to make data processing more efficient.
REAL PYTHON
Discover how Cisco teams use Python and InfluxDB to create custom DevOps and network monitoring solutions →
INFLUXDATA sponsor
This article details the changes in the Pandas 2.0 release, with emphasis on the underlying adoption of Apache Arrow.
MARC GARCIA
Last week’s issue of PyCoders included a link to Python’s multiprocessing performance problem which now has a Hacker News conversation to go along with it.
HACKER NEWS
The articles discusses whether Python is as slow as so many authors claim. When doing so, it mentions highly optimized Python frameworks for numerical computation, efficient compilers - but also coding time as opposed to execution time. All in all, Python is much faster than most think.
MARCIN KOZAK • Shared by Marcin
In this introductory tutorial, you will build a recurrent neural network (RNN) with PyTorch. The RNN will be trained to read names and it will output the natural language they belong to. This is a modern spin on a tutorial from the PyTorch documentation.
RODRIGO GIRÃO SERRÃO • Shared by Rodrigo Girão Serrão
Level up your code game with SonarQube 9.9 LTS! Included is 2x faster Pull Request analysis and up to 90% faster first analysis of your Python projects if you are using Git SCM. Also, 21 rules to help you write error-free regex, new rules for detecting path-sensitive bugs, and more. Discover Now →
SONARQUBE sponsor
In this video course, you’ll learn what Python’s namedtuple is and how to use it in your code. You’ll also learn about the main differences between named tuples and other data structures, such as dictionaries, data classes, and typed named tuples.
REAL PYTHON course
In this tutorial, you’ll learn how to use NumPy reshape() to rearrange the data in an array. You’ll learn to increase and decrease the number of dimensions and to configure the data in the new array to suit your requirements.
REAL PYTHON
Flask is approaching its 13th birthday, and to celebrate, Phillip has written 13 tips for writing modern Flask apps. It covers dealing with JSON, environment based configuration, auto-generated docs, and more.
PHILIP JONES
There are many speed improvements in CPython 3.11, but that doesn’t mean the Python alternatives don’t still have some advantages. Pyston and PyPy are still better in some cases.
MICHAEL LARABEL
This tutorial shows you how to build a system that takes a question template and writes a resume as output. The example is built on Streamlit and OpenAI.
PIERO PAIALUNGA
Elixir is a functional based programming language with similarities to Ruby and Python. This article is an intro guide for programmers who know Python.
RICH JONES
Python’s addition of typing has created “two” languages: typed and untyped Python. Tin tells you why that’s a great thing.
TIN TVRTKOVIĆ
At CData, we simplify connectivity between the application and data sources that power business, making it easier to unlock the value of data.
CDATA SOFTWARE sponsor
This step-by-step tutorial shows you how to build a rust tool and deliver it through Python packaging mechanisms.
MAKSUDUL HAQUE' • Shared by Maksudul Haque
Quick instructions on using nox with parameters to test multiple versions of Python against your test suite.
SETH LARSON
GITHUB.COM/YILING-J • Shared by yiling
RWKV-LM: RNN With Transformer-Level LLM Performance nosqlapi: NOSQL Database APINOSQLAPI.READTHEDOCS.IO • Shared by Matteo Guadrini
paperless-ngx: Scan, Index & Archive Your Physical Docs Events Santa Cruz Python Meetup March 8, 2023
MEETUP.COM
March 8, 2023
MEETUP.COM
March 8, 2023
REALPYTHON.COM
March 8, 2023
MEETUP.COM
March 8, 2023
PYTHONNORTHEAST.COM
March 11, 2023
MEETUP.COM
March 13 to March 18, 2023
PYTHONWEBCONF.COM
March 18 to March 20, 2023 in Vancouver, BC + Remote
PYCASCADES.COM
Happy Pythoning!
This was PyCoder’s Weekly Issue #567.
View in Browser »
[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]
Real Python: Manipulating ZIP Files With Python
Python’s zipfile is a standard-library module intended to manipulate ZIP files. This file format is a widely adopted industry standard when it comes to archiving and compressing digital data. You can use it to package together several related files. It also allows you to reduce the size of your files and save disk space. Most importantly, it facilitates data exchange over computer networks.
Knowing how to create, read, write, populate, extract, and list ZIP files using the zipfile module is a useful skill to have as a Python developer or a DevOps engineer.
In this video course, you’ll learn how to:
- Read, write, and extract files from ZIP files with Python’s zipfile
- Read metadata about the content of ZIP files using zipfile
- Use zipfile to manipulate member files in existing ZIP files
- Create new ZIP files to archive and compress files
If you commonly deal with ZIP files, then this knowledge can help to streamline your workflow to process your files confidently.
To get the most out of this video course, you should know the basics of working with files, using the with statement, handling file system paths with pathlib, and working with classes and object-oriented programming.
If you’d like to learn more about the REPL used in this course, check out Discover bpython: A Python REPL With IDE-Like Features. Note that all the code in this course will also run in the Python standard REPL.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Stack Abuse: How to Compare Tuples in Python
What do you do when you have two or more tuples and you need to compare them? This may seem like a pretty straightforward thing to do, but when you scratch the surface, you'll find out that there's much more to it than you initially realize.
Comparing tuples in Python can be used in a variety of applications, such as sorting and filtering data, comparing lists, and even testing software. However, not all tuple comparison methods are created equal, and choosing the wrong method can lead to unexpected results or inefficient code.
In this article, we'll take a look at a couple of ways to compare tuples in Python - using the comparison operator, and the all() function. We'll explain how each method works, provide examples, and discuss the pros and cons of each approach.
Comparing Tuples Using Comparison OperatorsOne of the most common ways to compare tuples in Python is to use the comparison operators. Comparison operators in Python are:
- == → "Equal" operator
- != → "Not equal" operator
- > → "Greater than" operator
- < → "Less than" operator
- >= → "Greater than or equal to" operator
- <= → "Less than or equal to" operator
Note: The tuple comparison in Python is performed lexicographically.
Let's take a look at an example of comparing tuples using the comparison operators:
tuple1 = (1, 2, 3) tuple2 = (4, 5, 6) print(tuple1 < tuple2) # Output: True print(tuple1 > tuple2) # Output: False print(tuple1 == tuple2) # Output: FalseLet's take a look at the < operator first. The first element of tuple1 is compared to the first element of tuple2. Since 1 < 4, the comparison operator returns True. The same comparison is then performed for the second and third elements of the tuples. and since all three of them return True, the result is also True - all elements of the tuple1 are "less than" the corresponding elements of the tuple2.
Similarly, for the > operator, we start by comparing the first elements of given tuples. Since 1 > 4, returns False, the final result of the comparison is also False. The same can be said for the == operator.
Note that the comparison works by comparing every pair of corresponding tuple elements. As long as there is at least one element comparison evaluated as False, the whole comparison is also evaluated as False. Sounds similar? Something like logical AND, right?
But what if you have the following scenario:
tuple1 = (1, 2, 3, 4) tuple2 = (1, 2, 3, 6)And we want to check if those tuples are equal. All corresponding elements up until the last elements are equal - 1 == 1, 2 == 2, and so on. All of those comparisons are evaluated as True. And then, we come across the last comparison - 4 == 6. That's False, and, even though all of the previous comparisons were evaluated as True, this one essentially converts the result of the whole comparison to False.
Another interesting scenario is when you try to compare two tuples of different lengths:
tuple1 = (1, 2, 3) tuple2 = (1, 2, 3, 6)Even though the first two elements of those tuples are the same, the tuple1 == tuple2 comparison will still return False because the tuple2 has one more element than tuple1 and the == operator has nothing to compare 6 with.
One more situation where comparing tuples with comparison operators may show interesting behavior is when tuples to be compared have elements of different types:
tuple1 = (1, 2, 3) tuple2 = (1, 2, "3") print(tuple1 == tuple2) # Output: FalseThis also returns False because the integer 3 is not lexicographically equal to the string "3".
Note: Another thing to note is that this method only compares the tuples element-wise, without considering the overall structure of the tuples. Two tuples with the same elements in a different order will be considered unequal (which is in line with the fact that tuples are ordered data structures).
Comparing tuples using the all() functionAnother way to compare tuples in Python is to use the built-in all() function. The all() function takes an iterable (like a tuple) as input and returns True if all elements in the iterable evaluate to True, and False otherwise. To compare two tuples using all(), we can convert the comparison of each element in the tuples to a Boolean value and pass them as arguments to the all() function:
tuple1 = (1, 2, 3) tuple2 = (4, 5, 6) print(all(a < b for a, b in zip(tuple1, tuple2))) # Output: True print(all(a > b for a, b in zip(tuple1, tuple2))) # Output: False print(all(a == b for a, b in zip(tuple1, tuple2))) # Output: FalseWe used the built-in zip() function to iterate over both tuples element-by-element and compare each element using a generator expression. The generator expression creates a tuple of two elements, a and b, which correspond to the elements of tuple1 and tuple2 at the current position. The comparison operator <, >, or == is used to compare a and b, and the result is converted to a Boolean value (True or False). The resulting Boolean values are then passed as arguments to the all() function, which returns True if all values are True, and False otherwise.
The described behavior is pretty similar to the behavior of the simple comparison operators, but this approach allows you to specify a custom comparison function, instead of relying on the default element-wise comparison. This can be useful if you need to compare tuples with complex structures or non-standard data types.
ConclusionIn this article, we've explored several ways to compare tuples in Python. We've discussed using the comparison operator and the all() function. When choosing a method for comparing tuples, it's important to consider the structure of your data and the specific requirements of your use case. If you have simple tuples with identical structures, the comparison operator may be the most straightforward solution. If you need more control over the comparison process, or if your tuples have complex structures, the all() function may be more suitable.
Tryton News: Security release for issue12108
A vulnerability in trytond has been found by José Antonio Díaz Miralles (@tiyujopite).
Due to issue12108, the Tryton server does not refresh the authenticated user data but instead uses the values from the first request.
A fix for all supported versions has been released.
Affected versions per supported series:
trytond: 6.6: <= 6.6.5 6.4: <= 6.4.12 6.0: <= 6.0.28Non affected versions per supported series:
trytond: 6.6: >= 6.6.6 6.4: >= 6.4.13 6.0: >= 6.0.29We encourage everyone to upgrade the trytond package to latest released version.
Reference Concerns?Any security concerns should be reported on the bug-tracker at [Issues · Tryton / Tryton · GitLab) marking them as confidential.
1 post - 1 participant
Read the Docs: Read the Docs newsletter - March 2023
⭐️ We passed our 10,000th issue/pull request on GitHub. And it’s pretty much an equal split between the 5039 issues and 4872 pull requests now registered. Thanks to the whole community for building this together through code, issues, suggestions… and documentation!
🌪️ Follow up: Build times have gone rapidly down after last month’s introduction of parallel uploading of artifacts with rclone. Depending on the number of files in your build output, build times may have gone down several seconds or several minutes. For instance, a large project like Write the Docs has gone down from ~7 minutes to under 3 minutes. If you want to see the change for your project, have a look at your build times before and after February 8.
1️⃣️ We are going to have 1 primary marketing website in the future, while still running two different product for our Community & Business users. The first change that is now done is redirecting logged out users on https://readthedocs.com/ to about.readthedocs.com. We will soon be doing this for logged out users going to https://readthedocs.org/, excepting project-specific subpages.
🧹️ We are now able to delist unofficial projects from search engines. If you find an unofficial and unmaintained project, please see our policy on unofficial and unmaintained projects.
🐞 Custom 404s with Sphinx DirHTML builder are now supported.
🔒️ Vulnerability fixed: Serving content from pull requests previews on main docs domains.
🔒️ Vulnerability fixed: Path traversal: access to files from any project.
You can always see the latest changes to our platforms in our Read the Docs Changelog.
Upcoming🪄️ The Django application that handles uncached requests on all our hosted documentation is called El Proxito. And that application is getting a major refactor, improving both speed and features. Most of the refactoring work is already merged and within the coming weeks, we will start to enable it gradually while monitoring if there are issues.
📚️ Documentation refactor underway: A regular reader of our user docs might notice that our navigation has been greatly changed and now contains only the 4 Diátaxis categories. The first work from our Diátaxis documentation refactor is now live and the general work has moved into shorter iterations. Once we have gathered all the final experience from the refactor, we will announce it together with practical inputs for how other documentation projects might approach such a refactor.
💬️ …If you are planning your own refactor of an existing documentation project, we’d love to hear what your questions and concerns are and address those future blog posts and talks.
🐞 Pull request builds should point to previews, not build pages. A fix will is ready in #10085.
🎬️ We’re slowly moving towards a public beta of our v2 dashboard experience. Here’s a screenshot of what’s to come…
A screenshot of the build page on the Read the Docs Dashboard v2.
Want to follow along with our development progress? View our full Roadmap 📍️
Awesome project of the monthThe Haskell Tool Stack is commonly known in the world of Haskell as simply Stack and is this month’s chosen entry from Awesome Read the Docs Projects 🕶️. See our chosen highlights from Stack’s documentation in the following Twitter thread:
The Haskell Tool Stack is a packaging tool for #haskell. Because their documentation is so awesome, it’s also their main website 💯
Stack’s website is maintained with GitHub, MkDocs, and Read the Docs: https://t.co/GaCTgxTUcm
Here is a 🤏 (small) 🧵 about why it’s awesome 🕶️ pic.twitter.com/wdAQ3NigHK
When you post links to your documentation on chat and social media, you will probably see a generic preview. Enter sphinxext-opengraph!
This Sphinx extension allows you to configure your own preview card, compatible with all major chat and social media platforms. A particular feature which we really enjoy is that the extension will detect and use the first illustration in a documentation page. If you want to see it in action, try sharing a link to this newsletter blog post (marketing unintended). Our user docs are also using sphinxext-opengraph.
Questions? Comments? Ideas for the next newsletter? Contact us!
Robin Wilson: Travel times, over time
A fun analysis I did a while back was using the Google Maps API to look at travel times between certain locations over time. I originally got interested in this because I found that travelling from my house to the university (yes, that’s how long ago this started…) seemed to either take a very short time, or a very long time, but rarely anything in the middle. I wondered if the histogram of travel times might be bi-modal – so I thought I’d investigate. This then led to doing various other analyses of local travel times.
It was actually very easy to gather the data for this. Google Maps will give you an estimated travel time for any route that you plan, and I’ve usually found these estimated times quite accurate – so I’ve relied on these for my data here. There is a googlemaps package for Python that wraps the Google Maps API and you can get the travel time between two locations using code like this:
def get_travel_time(from_loc, to_loc): now = datetime.now() directions_result = gmaps.directions(from_loc, to_loc, mode="driving", departure_time=now) return directions_result[0]['legs'][0]['duration_in_traffic']['value'] / 60Back when I did this, the Google Maps API didn’t require authentication for most uses, and had a very generous free tier. This changed after a while, and now I suspect you’d need to give an API key, and have a credit card set up on your API account, and so on.
Anyway, as you can see, this code is just one API call, and then extracting the ‘duration_in_traffic’ from the result (it comes in seconds, we convert it to minutes).
To get a dataset of travel times over time, you just need to run this regularly (using chron, or equivalent), giving a sensible set of from and to locations. These locations can be anything that Google Maps recognises: an address, a lat/lon pair, a business etc. One warning is that you must pick the from and to locations carefully if you’re starting on a dual carriageway or a motorway: if you give the starting location on the wrong carriageway (just a small change in the latitude and longitude values) then your route will be wrong as it will show you travelling down that carriage way until a junction where you can turn onto the correct carriageway.
So, lets look at some results:
Home to University histogramsLooking at my initial question, here is a histogram of travel times from my home to the university (click on any image to see a larger version):
You can see that my hypothesis was wrong: the distribution is not bimodal, but it is definitely one-sided. The peak is around 11-12 minutes, and then there is a long tail extending to the right as delays increase, with a few journeys taking almost double that time.
Southampton to BournemouthA more interesting analysis is the travel time between Southampton and Bournemouth (for those not familiar with UK geography, Southampton is a city on the south coast of the UK but without a beach, and Bournemouth has a nice beach about 45mins drive from Southampton). Again, this idea came from personal experience: I was going over to the Bournemouth area relatively frequently during the spring/summer to go to the beach, and was interested in how long it would take. It’s well-known that the traffic between Southampton and Bournemouth is particularly bad on a summer weekend, and I wondered how bad it was on bank holidays.
So, I plotted travel time between Southampton and Bournemouth on normal Mondays, compared to bank holiday Mondays:
There are a few interesting patterns in this graph: firstly, and most obviously, there is a big difference in travel times on bank holidays – travel in mid-morning can take over 50% longer than on a normal Monday. You can see the rush hour peaks on normal Mondays for journeys leaving at around 7am and 4:30pm. These peaks are still there on bank holiday Mondays, but are significantly smaller. The travel time on a bank holiday starts to exceed a normal day from around 8-9am, and reaches its peak for journeys starting from Southampton around 11am (after all, who wants to get up early on a bank holiday!).
Looking at the reverse journey, from Bournemouth back to Southampton, you can see a far broader peak on bank holidays:
Again, there is a standard rush hour peak (though less well-defined) on normal Mondays at around 7am and 4-5pm, but the bank holiday peak starts around 10am (presumably for people travelling from Bournemouth for some other reason than to go directly to Southampton), and then there is a broad peak starting in the early afternoon of people travelling back to Southampton, extending right up until around 8-9pm.
So, from a practical purpose, if you’re going to the beach in Bournemouth on a bank holiday Monday, when should you leave Southampton? Traffic is probably best if you leave before 8am (a bit early for a bank holiday!) or after 3pm, and you’re likely to hit the traffic if you come home any time after lunch.
Just for amusement purposes, I decided to look at the travel time between Southampton and Bournemouth on Fridays, Saturdays and Sundays and compare it to the temperature. I got the temperature data for Bournemouth from the Weather Underground API (which doesn’t seem to work any more, as I found when I tried to re-run this analysis). Plotting temperature against travel time gives this graph:
The straight line fit doesn’t look great, but apparently it has a R^2 of 0.67!
The Avenue in SouthamptonFinally, let’s look at The Avenue in Southampton. This is a long, straight road running north-south from the end of the M3 at the north of Southampton, right down to the city centre in the south. It gets fairly busy during rush hour, so I thought I’d look at travel time along it. Here’s the graph:
Firstly, you can see that it is quicker to travel southbound on the Avenue than it is to travel northbound. I think this is due to the way that traffic lights and junctions wait: there are various right turns when travelling northbound that cause traffic to queue, and travelling southbound most junctions are either No Right Turn, or have filter lanes for right turns.
Also, the peak in the morning is offset: travel time peaks earlier for traffic going northbound, and later for traffic going southbound. I assume this is because traffic going northbound is leaving Southampton to go somewhere else, so it is the beginning of their journey, whereas traffic going southbound is arriving in Southampton as their destination. There are some other interesting patterns on this graph that I haven’t quite worked out yet – so I’ll leave you to ponder them.
Stack Abuse: Guide to Tuples in Python
Welcome to the world of Python tuples, where parentheses are the key to unlocking the power of data organization and manipulation! As a Python programmer, you might already be familiar with lists, dictionaries, and sets - but don't overlook tuples! They are often overshadowed by more popular data types, but tuples can be incredibly useful in many situations.
In this guide, we'll take a deep dive into Python tuples and explore everything you need to know to use tuples in Python. We'll cover the basics of creating and accessing tuples, as well as more advanced topics like tuple operations, methods, and unpacking.
How to Create Tuples in PythonIn Python, tuples can be created in several different ways. The simplest one is by enclosing a comma-separated sequence of values inside of parentheses:
# Create a tuple of integers my_tuple = (1, 2, 3, 4, 5) # Create a tuple of strings fruits = ('apple', 'banana', 'cherry')Alternatively, you can create a tuple using the built-in tuple() function, which takes an iterable as an argument and returns a tuple:
# Create a tuple from a list my_list = [1, 2, 3, 4, 5] my_tuple = tuple(my_list) # Create a tuple from a string my_string = "hello" my_tuple = tuple(my_string)This method is a bit more explicit and might be easier to read for Python novices.
You can also create an empty tuple by simply using the parentheses:
# Create an empty tuple my_tuple = ()It's worth noting that even a tuple with a single element must include a trailing comma:
# Create a tuple with a single element my_tuple = (1,)Note: Without the trailing comma, Python will interpret the parentheses as simply grouping the expression, rather than creating a tuple.
With the basics out of the way, we can take a look at how to access elements within a tuple.
How to Access Tuple Elements in PythonOnce you have created a tuple in Python, you can access its elements using indexing, slicing, or looping. Let's take a closer look at each of these methods.
IndexingYou can access a specific element of a tuple using its index. In Python, indexing starts at 0, so the first element of a tuple has an index of 0, the second element has an index of 1, and so on:
# Create a tuple my_tuple = ('apple', 'banana', 'cherry') # Access the first element (index 0) print(my_tuple[0]) # Output: 'apple' # Access the third element (index 2) print(my_tuple[2]) # Output: 'cherry'If you try to access an element that is outside the bounds of the tuple, you'll get an IndexError:
# Try to access an element outside the bounds of the tuple print(my_tuple[3]) # Raises an IndexErrorAnother interesting way you can access an element from the tuple is by using negative indices. That way, you are effectively indexing a tuple in reversed order, from the last element to the first:
# Access the last element of the tuple print(my_tuple[-1]) # Output: ('cherry')Note: Negative indexing starts with -1. The last element is accessed by the -1 index, the second-to-last by the -2, and so on.
SlicingYou can also access a range of elements within a tuple using slicing. Slicing works by specifying a start index and an end index, separated by a colon. The resulting slice includes all elements from the start index up to (but not including) the end index:
# Create a tuple my_tuple = ('apple', 'banana', 'cherry', 'date', 'elderberry') # Access a slice of the tuple (indices 1 to 3) print(my_tuple[1:4]) # Output: ('banana', 'cherry', 'date')You can also use negative indices to slice from the end of the tuple:
# Access the last three elements of the tuple print(my_tuple[-3:]) # Output: ('cherry', 'date', 'elderberry')Advice: If you want to learn more about slicing in Python, you should definitely take a look at our article "Python: Slice Notation on List".
Looping Through TuplesFinally, you can simply loop through all the elements of a tuple using a for loop:
# Create a tuple my_tuple = ('apple', 'banana', 'cherry') # Loop through the tuple and print each element for fruit in my_tuple: print(fruit)This will give us:
apple banana cherryIn the next section, we'll explore the immutability of tuples and how to work around it.
Can I Modify Tuples in Python?One of the defining characteristics of tuples in Python is their immutability. Once you have created a tuple, you cannot modify its contents. This means that you cannot add, remove, or change elements within a tuple. Let's look at some examples to see this in action:
# Create a tuple my_tuple = (1, 2, 3) # Try to modify the tuple my_tuple[0] = 4 # Raises a TypeError # Try to add an element to the tuple my_tuple.append(4) # Raises a AttributeError # Try to remove an element from the tuple del my_tuple[1] # Raises a TypeErrorAs you can see, attempting to modify a tuple raises appropriate errors - TypeError or AttributeError. So what can you do if you need to change the contents of a tuple?
Note: It's important to note that all of the methods demonstrated below are simply workarounds. There is no direct way to modify a tuple in Python, and the methods discussed here effectively create new objects that simulate the modification of tuples.
One approach is to convert the tuple to a mutable data type, such as a list, make the desired modifications, and then convert it back to a tuple:
# Convert the tuple to a list my_list = list(my_tuple) # Modify the list my_list[0] = 4 # Convert the list back to a tuple my_tuple = tuple(my_list) # Check the contents of the modified tuple print(my_tuple) # Output: (4, 2, 3)This approach allows you to make modifications to the contents of the tuple, but it comes with a trade-off - the conversion between the tuple and list can be expensive in terms of time and memory. So use this technique sparingly, and only when absolutely necessary.
Another approach is to use tuple concatenation to create a new tuple that includes the desired modifications:
my_tuple = (1, 2, 3, 5) # Create a new tuple with a modified first element new_tuple = (4,) + my_tuple[1:] # Check the contents of the modified tuple print(new_tuple) # Output: (4, 2, 3)In this example, we used tuple concatenation to create a new tuple that includes the modified element (4,) followed by the remaining elements of the original tuple. This approach is less efficient than modifying a list, but it can be useful when you only need to make a small number of modifications.
Remember, tuples are immutable, and examples shown in this section are just (very inefficient) workarounds, so always be careful when modifying tuples. More specifically, if you find yourself in need of changing a tuple in Python, you probably shouldn't be using a tuple in the first place.
What Operations Can I Use on Tuples in Python?Even though tuples are immutable, there are still a number of operations that you can perform on them. Here are some of the most commonly used tuple operations in Python:
Tuple ConcatenationYou can concatenate two or more tuples using the + operator. The result is a new tuple that contains all of the elements from the original tuples:
# Concatenate two tuples tuple1 = (1, 2, 3) tuple2 = (4, 5, 6) result = tuple1 + tuple2 print(result) # Output: (1, 2, 3, 4, 5, 6) Tuple RepetitionYou can repeat a tuple a certain number of times using the * operator. The result is a new tuple that contains the original tuple repeated the specified number of times:
# Repeat a tuple my_tuple = (1, 2, 3) result = my_tuple * 3 print(result) # Output: (1, 2, 3, 1, 2, 3, 1, 2, 3) Tuple MembershipYou can check if an element is present in a tuple using the in operator. The result is a Boolean value (True or False) indicating whether or not the element is in the tuple:
# Check if an element is in a tuple my_tuple = (1, 2, 3) print(2 in my_tuple) # Output: True print(4 in my_tuple) # Output: False Tuple ComparisonYou can compare two tuples using the standard comparison operators (<, <=, >, >=, ==, and !=). The comparison is performed element-wise, and the result is a Boolean value indicating whether or not the comparison is true:
# Compare two tuples tuple1 = (1, 2, 3) tuple2 = (4, 5, 6) print(tuple1 < tuple2) # Output: True print(tuple1 == tuple2) # Output: False Tuple UnpackingYou can unpack a tuple into multiple variables using the assignment operator (=). The number of variables must match the number of elements in the tuple, otherwise a ValueError will be raised. Here's an example:
# Unpack a tuple into variables my_tuple = (1, 2, 3) a, b, c = my_tuple print(a) # Output: 1 print(b) # Output: 2 print(c) # Output: 3 Tuple MethodsIn addition to the basic operations that you can perform on tuples, there are also several built-in methods that are available for working with tuples in Python. In this section, we'll take a look at some of the most commonly used tuple methods.
count()The count() method returns the number of times a specified element appears in a tuple:
# Count the number of occurrences of an element in a tuple my_tuple = (1, 2, 2, 3, 2) count = my_tuple.count(2) print(count) # Output: 3 index()The index() method returns the index of the first occurrence of a specified element in a tuple. If the element is not found, a ValueError is raised:
# Find the index of the first occurrence of an element in a tuple my_tuple = (1, 2, 3, 2, 4) index = my_tuple.index(2) print(index) # Output: 1 len()The len() function returns the number of elements in a tuple:
# Get the length of a tuple my_tuple = (1, 2, 3, 4, 5) length = len(my_tuple) print(length) # Output: 5 sorted()The sorted() function returns a new sorted list containing all elements from the tuple:
# Sort the elements of a tuple my_tuple = (3, 1, 4, 1, 5, 9, 2, 6, 5) sorted_tuple = tuple(sorted(my_tuple)) print(sorted_tuple) # Output: (1, 1, 2, 3, 4, 5, 5, 6, 9)Note: The sorted() function returns a list, which is then converted back to a tuple using the tuple() constructor.
min() and max()The min() and max() functions return the smallest and largest elements in a tuple, respectively:
# Find the smallest and largest elements in a tuple my_tuple = (3, 1, 4, 1, 5, 9, 2, 6, 5) min_element = min(my_tuple) max_element = max(my_tuple) print(min_element) # Output: 1 print(max_element) # Output: 9These are just a few examples of the methods that are available for working with tuples in Python. By combining these methods with the various operations available for tuples, you can perform a wide variety of tasks with these versatile data types.
Tuple UnpackingOne of the interesting features of tuples in Python that we've discussed is that you can "unpack" them into multiple variables at once. This means that you can assign each element of a tuple to a separate variable in a single line of code. This can be a convenient way to work with tuples when you need to access individual elements or perform operations on them separately.
Let's recall the example from the previous section:
# Tuple unpacking example my_tuple = (1, 2, 3) a, b, c = my_tuple print(a) # Output: 1 print(b) # Output: 2 print(c) # Output: 3In this example, we created a tuple my_tuple with three elements. Then, we "unpack" the tuple by assigning each element to a separate variables a, b, and c in a single line of code. Finally, we verified that the tuple has been correctly unpacked.
One interesting use case of tuple unpacking is that we can use it to swap the values of two variables, without needing a temporary variable:
# Swap the values of two variables using tuple unpacking a = 5 b = 10 print("Before swapping:") print("a =", a) print("b =", b) # Swap the values a, b = b, a print("After swapping:") print("a =", a) print("b =", b)Output:
Before swapping: a = 5 b = 10 After swapping: a = 10 b = 5Here, we use tuple unpacking to swap the values of a and b. The expression a, b = b, a creates a tuple with the values of b and a, which is then unpacked into the variables a and b in a single line of code.
Another useful application of tuple unpacking is unpacking a tuple into another tuple. This can be helpful when you have a tuple with multiple elements, and you want to group some of those elements together into a separate tuple:
# Unpacking a tuple into another tuple my_tuple = (1, 2, 3, 4, 5) a, b, *c = my_tuple print(a) # Output: 1 print(b) # Output: 2 print(c) # Output: [3, 4, 5]We have a tuple my_tuple with five elements. We use tuple unpacking to assign the first two elements to the variables a and b, and the remaining elements to the variable c using the * operator. The * operator is used to "unpack" the remaining elements of the tuple into a new tuple, which is assigned to the variable c.
This is also an interesting way to return multiple values/variables from a function, allowing the caller to then decide how the return values should be unpacked and assigned from their end.
ConclusionTuples are one of fundamental data types in Python. They allow you to store a collection of values in a single container. They're similar to lists, but with a few important differences - tuples are immutable, and they're usually used to store a fixed set of values that belong together.
In this guide, we've covered the basics of working with tuples in Python, including creating tuples, accessing their elements, modifying them, and performing operations on them. We've also explored some of the more advanced features of tuples, such as tuple unpacking.
Tuples may not be the most glamorous data type in Python, but they're certainly effective when you know how and when to use them. So the next time you're working on a Python project, remember to give tuples a try. Who knows, they may just become your new favorite data type!
PyCon: PyCon US 2023 Call for Volunteers
Looking to make a meaningful contribution to the Python community? Look no further than PyCon US 2023! Whether you're a seasoned Python pro or a newcomer to the community and looking to get involved, there's a volunteer opportunity that's perfect for you.
This year, the sign-up for volunteer roles is done directly through the PyCon US website! This way, you can view and manage shifts you sign up for through your personal dashboard! You can read up on the different roles to volunteer for and how to sign up on the PyCon US website here.
PyCon US is largely organized and run by volunteers. Every year, we ask to fill over 300 onsite volunteer hours to ensure that everything runs smoothly at the event. And the best part? You don't need to commit a lot of time to make a difference– some shifts are as short as one hour long! You can sign up for as many or as few shifts as you’d like. Even a couple of hours of your time can go a long way in helping us create an amazing experience for attendees.
Keep in mind that you need to be registered for the event in order to sign up for a volunteer role.
One important way to get involved is to sign up as a Session Chair or Session Runner. This is an excellent opportunity to meet and interact with speakers while helping to ensure that sessions run smoothly. And who knows, you might just learn something new along the way! You can sign up for these roles directly on the Talks schedule. For more details on how to sign up as a Session Chair or Session Runner, click here.
Volunteer your time at PyCon US 2023 and you’ll be part of a fantastic community that's passionate about Python programming and help us make the 20th anniversary conference a huge success. Sign up today for the shifts that call to you and join the fun!
PyBites: Learning to code is a lot like learning a language
This content was first sent as part of our friends email list, you can subscribe here to get this type of content early and fresh every week …
Let me tell you the story of how I effectively learned foreign languages (there are a lot of parallels with how I became a developer).
I was not a good language learning when I was young, heck I even flunked 6th grade in part because my English was terrible!
But being bad at something does not mean it’s game over, I was using the wrong “software”.
My approach was flawed: I tried to memorize things, almost learning from a dictionary, highly inefficient and not fun.
Learning practical things like languages, driving a car or programming doesn’t work like that.
Everything changed when I was introduced to “interrailing“. For those that don’t know, it’s a 3-4 week train pass that let’s you travel through an area of (in my case) Europe. Apart from having a well deserved break over summer, I always used those as an opportunity to learn the languages of the countries I visited, mainly French, Spanish and Italian.
Once I decided I would move to Spain that summer I went all in. I travelled through Spain and I fully immersed myself in the language. I tried to speak it with as many people as I could, I constantly noted down words I kept forgetting, I bought local newspapers and forced myself to read as much as I could (not understanding much yet).
After a few weeks of constant deliberate practice – we also covered this on the podcast here and here – I reached a tipping point: I earned conversational level in a language I could not even say “Hola, que tal?” in a few months prior.
Flash forward 10 or so years and I got on a similar mammoth mission to learn to program, because when I saw the power of automating the boring stuff I knew this would be a game changer for my career. It would be a fun one too, coder’s delight is real!
But… learning to program almost from scratch (at most I was fluent in Excel) is hard.
But I applied the same techniques as the language learning journey: constant deliberate practice, failing my way forward. The tutorials got boring pretty fast (paralysis by tutorial anyone?!), and things were not clicking, and many times I wanted to give up!
Until I started to build projects and apply what I learned. It was still hard, I got stuck constantly, but it was my way of fully emerging into the learning.
And over time, I had great success. The first code I wrote was really ugly, but I got a tool working, that ultimately would become a staple in the organization I worked for. “Done > perfect” is so true and something you seriously need to embrace if you want to succeed as a developer.
More importantly building bigger projects I cared about, started doing 3 things for me:
- I inevitably ran into a lot of design issues, things you usually are not confronted with in the “safe” zone of tutorials.
- I got completely hooked because building software was so much more tangible, I was creating a product, something that could help people and I could talk about later when I had to qualify as a programmer (specially because I did not have a formal CS degree).
- Building complete solutions without being a developer by training, over time I dared to call myself a developer. Without relevant projects shipped this would never have happened. Tutorial purgatory is insidious, you not only waste a lot of time, there is actually a “0 to 1” issue, you won’t bridge the chasm!
So if you take one thing away from this email, let it be that if you want to become a developer and want to get there relatively fast, find one or more projects you can fully immerse yourself in.
Don’t worry about design patterns or overly planning everything when you start. Doing so will keep you stuck.
Have people look at your code. Live and breathe code in the context of your apps. Build in public, ship fast.
Have people use your code, there is no better learning like your code hitting the real world (as Tyson said: “everybody has a plan till they get hit in the face”), it changes everything and you will grow.
I hope my story inspired you and made you realize you might be “two inches removed from gold”, it is possible, using the right approach.
– Bob
What took us years to learn we have distilled in our PDM program which will significantly shortcut the time it takes to become a proficient developer.
Check out how it works and what people achieve working with us here. For more information about our uniquely practical approach check out this training.
Real Python: What's a Python Namespace Package, and What's It For?
Python namespace packages are an advanced Python feature. You may have heard them mentioned in relation to the __init__.py file. Specifically, if you don’t include at least an empty __init__.py file in your package, then your package becomes a namespace package.
For the most part, namespace packages and regular packages won’t behave differently when it comes to using them in your project. In fact, you’ve probably accidentally forgotten to include an __init__.py file in at least one package but didn’t notice any side effects. While namespace packages are generally a bit slower to import than regular packages, you won’t usually run into many issues.
In this tutorial, you’ll dive into what Python namespace packages are, where they come from, why they’re created when you don’t include an __init__.py file, and when they might be useful.
Along the way, you’ll get the chance to make your own namespace package structures, and you’ll install and extend an experimental namespace package from PyPI.
Python namespace packages are probably going to be useful for people who manage or architect collections of related packages. That said, you’ll get to experiment with a project that can make namespace packages more accessible to the average user.
This is an advanced tutorial, and to get the most out of it, you should be very familiar with the basics of Python and comfortable with the import system, as well as having some exposure to packaging.
So, what’s a Python namespace package, and what’s it for?
Free Source Code: Click here to download the free source code that you’ll use to explore Python namespace packages.
In Short: Python Namespace Packages Are a Way to Organize Multiple PackagesBy way of a quick recap, you’ll first examine the general concept of a Python namespace before tackling namespace packages. A namespace is a way to group objects under a specific name. You can group values, functions, and other objects.
For example, when you import math, you gain access to the math namespace. Inside the math namespace, you can select from a whole host of different objects.
You can also think of a Python dictionary as a namespace. With a Python dictionary, you can take two variables that started out as completely separate, independent variables and include them within the same dictionary namespace:
>>>>>> real_python = {} >>> home_page = "https://realpython.com" >>> import_tutorial = "python-import" >>> real_python["home_page"] = home_page >>> real_python["import_tutorial"] = import_tutorialNow, you can reference both the home_page and import_tutorial values from the real_python namespace. Namespace packages work in a similar way, but they combine whole packages instead of values or other Python objects.
This allows you to have two independent packages on PyPI, for example, that still share the same namespace. One single namespace package doesn’t make a whole lot of sense. To really see the advantage of namespace packages, you need at least two packages.
Namespace packages are typically used by businesses that may have extensive libraries of packages that they want to keep under a company namespace. For example, the Microsoft Azure packages are all accessible, once installed, through the azure namespace.
That’s why you’ll create your own company namespace in the next section.
What Does a Namespace Package Look Like?Imagine you work for the Snake Corporation, and your team wants to make sure that all the packages in its library are always accessible from the snake_corp namespace. So, no matter what package you’re using, as long as it was made by the Snake Corporation, you’ll import from snake_corp.
Without namespace packages, you’d have two options:
- Create a monorepo, which would be a single package called snake_corp with hundreds of modules for all the different libraries and utilities that you might need.
- Create various packages, but prefix them all with snake_corp. For example, you might have snake_corp_dateutil as a package.
The trouble with creating a monorepo is that everyone has to download all the code even if they only use a tiny fraction of it. It also complicates matters in terms of version management and other packaging workflows, especially if different teams are in charge of different subpackages.
The issue with creating various packages with a common prefix is that it can be quite verbose, messy, and inelegant. And, at the end of the day, the Snake Corporation CEO has said that they don’t like that solution. They’d prefer using a monorepo over prefixing all the packages. Besides, using common prefixes is just a convention that doesn’t technically create a common namespace.
What you have in this situation is a perfect use case for namespace packages! Namespace packages allow you to have multiple separate packages with their own packaging workflow, but they can all live in the same snake_corp namespace.
Note: In the following examples, you’ll be creating a bunch of packages. You’ll note that the containing folders and the package names vary depending on whether you’re referring to the name for when you pip install or import, for example. Check out this table for a rough breakdown of what’s generally used:
Purpose Typical Casing Example PyPI and pip Kebab charset-normalizer Import Snake charset_normalizer Prose Pascal or none Charset NormalizerIn the following examples, you’ll be using these conventions. But be aware that these conventions aren’t universal.
Read the full article at https://realpython.com/python-namespace-package/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Python for Beginners: Pandas Insert Columns into a DataFrame in Python
We use a pandas dataframe to store and manipulate tabular data in python. In this article, we will discuss how to insert a new column into the pandas dataframe using the insert() method.
Table of Contents- The Pandas insert() Method
- Pandas Insert a Column at The Beginning of a DataFrame
- Insert Column at The End of a DataFrame in Python
- Pandas Insert Column at a Specific Index in a DataFrame
- Conclusion
The insert() method is used to insert a column into a dataframe at a specific position. It has the following syntax.
DataFrame.insert(loc, column, value, allow_duplicates=_NoDefault.no_default)Here,
- The loc parameter takes the index at which the new column is inserted as its input argument.
- The column parameter takes the column name as its input.
- The value parameter takes a list or pandas series as values for the specified column.
- The allow_duplicates parameter is used to decide if we can insert duplicate column names into the dataframe. By default, the insert() method raises a ValueError exception if the dataframe contains a column with the same name that we are trying to insert. If you want to insert duplicate column names into the pandas dataframe, you can set the allow_duplicates parameter to True.
To insert a column at the beginning of a dataframe, we can use the insert() method. Here, we will set the loc parameter to 0 so that the new column is inserted at the beginning. You can observe this in the following example.
import pandas as pd myDicts=[{"Roll":1,"Maths":100, "Physics":80, "Chemistry": 90}, {"Roll":2,"Maths":80, "Physics":100, "Chemistry": 90}, {"Roll":3,"Maths":90, "Physics":80, "Chemistry": 70}, {"Roll":4,"Maths":100, "Physics":100, "Chemistry": 90}, {"Roll":5,"Maths":90, "Physics":90, "Chemistry": 80}, {"Roll":6,"Maths":80, "Physics":70, "Chemistry": 70}] df=pd.DataFrame(myDicts) print("The original dataframe is:") print(df) df.insert(0,"Name", ["Aditya","Joel", "Sam", "Chris", "Riya", "Anne"]) print("The mofified dataframe is:") print(df)Output:
The original dataframe is: Roll Maths Physics Chemistry 0 1 100 80 90 1 2 80 100 90 2 3 90 80 70 3 4 100 100 90 4 5 90 90 80 5 6 80 70 70 The mofified dataframe is: Name Roll Maths Physics Chemistry 0 Aditya 1 100 80 90 1 Joel 2 80 100 90 2 Sam 3 90 80 70 3 Chris 4 100 100 90 4 Riya 5 90 90 80 5 Anne 6 80 70 70In this example, we first converted a list of dictionaries to a dataframe using the DataFrame() function. Then, we inserted the Name column in the created dataframe at index 0 using the insert() method. For this, we passed the value 0 as the first input argument, the string "Name" as the second input argument and the list of values as the third input argument to the insert() method.
Insert Column at The End of a DataFrame in PythonTo insert a column at the end of the dataframe, we can directly assign the column values to the column name in the dataframe as shown below.
import pandas as pd myDicts=[{"Roll":1,"Maths":100, "Physics":80, "Chemistry": 90}, {"Roll":2,"Maths":80, "Physics":100, "Chemistry": 90}, {"Roll":3,"Maths":90, "Physics":80, "Chemistry": 70}, {"Roll":4,"Maths":100, "Physics":100, "Chemistry": 90}, {"Roll":5,"Maths":90, "Physics":90, "Chemistry": 80}, {"Roll":6,"Maths":80, "Physics":70, "Chemistry": 70}] df=pd.DataFrame(myDicts) print("The original dataframe is:") print(df) df["Name"]= ["Aditya","Joel", "Sam", "Chris", "Riya", "Anne"] print("The mofified dataframe is:") print(df)Output:
The original dataframe is: Roll Maths Physics Chemistry 0 1 100 80 90 1 2 80 100 90 2 3 90 80 70 3 4 100 100 90 4 5 90 90 80 5 6 80 70 70 The mofified dataframe is: Roll Maths Physics Chemistry Name 0 1 100 80 90 Aditya 1 2 80 100 90 Joel 2 3 90 80 70 Sam 3 4 100 100 90 Chris 4 5 90 90 80 Riya 5 6 80 70 70 AnneIn the above example, we have used the indexing operator to insert a new column at the end of a dataframe.
Instead of the above approach, we can also use the insert() method to insert a column at the end. For this, we will use the following steps.
- First, will obtain the list of column names using the columns attribute of the dataframe. The columns attribute contains a list of column names.
- Next, we will use the len() function to find the total number of columns in the dataframe. Let it be numCol.
- Once, we get the number of columns in the dataframe, we know that the current columns exist at the positions 0 to numCol-1. Hence, we will insert the new column to the dataframe at the index numCol using the insert() method.
After execution of the above steps, we can insert a column at the end of the dataframe as shown in the following example.
import pandas as pd myDicts=[{"Roll":1,"Maths":100, "Physics":80, "Chemistry": 90}, {"Roll":2,"Maths":80, "Physics":100, "Chemistry": 90}, {"Roll":3,"Maths":90, "Physics":80, "Chemistry": 70}, {"Roll":4,"Maths":100, "Physics":100, "Chemistry": 90}, {"Roll":5,"Maths":90, "Physics":90, "Chemistry": 80}, {"Roll":6,"Maths":80, "Physics":70, "Chemistry": 70}] df=pd.DataFrame(myDicts) print("The original dataframe is:") print(df) numCol=len(df.columns) df.insert(numCol,"Name", ["Aditya","Joel", "Sam", "Chris", "Riya", "Anne"]) print("The mofified dataframe is:") print(df)Output:
The original dataframe is: Roll Maths Physics Chemistry 0 1 100 80 90 1 2 80 100 90 2 3 90 80 70 3 4 100 100 90 4 5 90 90 80 5 6 80 70 70 The mofified dataframe is: Roll Maths Physics Chemistry Name 0 1 100 80 90 Aditya 1 2 80 100 90 Joel 2 3 90 80 70 Sam 3 4 100 100 90 Chris 4 5 90 90 80 Riya 5 6 80 70 70 Anne Pandas Insert Column at a Specific Index in a DataFrameTo insert a column at a specific position in the dataframe, you can use the insert() method as shown below.
import pandas as pd myDicts=[{"Roll":1,"Maths":100, "Physics":80, "Chemistry": 90}, {"Roll":2,"Maths":80, "Physics":100, "Chemistry": 90}, {"Roll":3,"Maths":90, "Physics":80, "Chemistry": 70}, {"Roll":4,"Maths":100, "Physics":100, "Chemistry": 90}, {"Roll":5,"Maths":90, "Physics":90, "Chemistry": 80}, {"Roll":6,"Maths":80, "Physics":70, "Chemistry": 70}] df=pd.DataFrame(myDicts) print("The original dataframe is:") print(df) df.insert(2,"Name", ["Aditya","Joel", "Sam", "Chris", "Riya", "Anne"]) print("The mofified dataframe is:") print(df)Output:
The original dataframe is: Roll Maths Physics Chemistry 0 1 100 80 90 1 2 80 100 90 2 3 90 80 70 3 4 100 100 90 4 5 90 90 80 5 6 80 70 70 The mofified dataframe is: Roll Maths Name Physics Chemistry 0 1 100 Aditya 80 90 1 2 80 Joel 100 90 2 3 90 Sam 80 70 3 4 100 Chris 100 90 4 5 90 Riya 90 80 5 6 80 Anne 70 70In this example, we have inserted the Name column at index 2 of the input dataframe using the insert() method.
ConclusionIn this article, we discussed different ways to insert a column in a pandas dataframe. To learn more about python programming, you can read this article on how to create an empty dataframe in python. You might also like this article on working with XML files in Python.
I hope you enjoyed reading this article. Stay tuned for more informative articles.
Happy Learning!
The post Pandas Insert Columns into a DataFrame in Python appeared first on PythonForBeginners.com.
Mike Driscoll: PyDev of the Week: Janos Gabler
This week we welcome Janos Gabler (@JanosGabler) as our PyDev of the Week! Janos is the creator of estimagic, a Python package for nonlinear optimization. You can catch up with Janos on his website or by checking out Janos’ GitHub Profile.
Let’s spend some time getting to know Janos better!
Can you tell us a little about yourself (hobbies, education, etc.)I am Jano?. I live in Bonn, Germany, where I did a PhD in economics. I am now a postdoc at the University of Bonn and teach “Effective Programming Practices for Economists” and “Scientific computing”.
My contract runs until October. I am currently deciding what I want to do next. Most likely, I will be looking for Jobs in AI or the scientific Python ecosystem, but there is a slight chance of founding a startup.
While I try to avoid yak-shaving at work, I fully embrace it in my hobbies. For example, I like baking, which eventually led me to build my own wood-fired brick oven. I also enjoy woodworking, and the bookshelf I am currently building required me to learn to weld, so I could construct a big bandsaw out of scrap metal which I needed to resaw the boards for the shelf.
Why did you start using Python?I started using Python in 2015 for empirical research (using pandas and statsmodels). I had no previous experience in any other programming language and did not expect programming to be something I would enjoy. This changed very quickly!
I was lucky to attend “Effective programming practices for Economists” (the class I am teaching now) right at the beginning of my programming journey. This introduced me to git, unit testing and best practices.
The projects quickly became more challenging. There was a short period when I regretted picking a “slow language”, but I quickly learned how to get around that. First with Cython, then Numba and nowadays JAX.
What other programming languages do you know and which is your favorite?It speaks for Python that I don’t know any other programming language well. I have some experience in Fortran, Matlab, C, and R, but I did all my computational projects during my PhD in Python.
I guess this also answers the question of which language is my favorite?
Having said that, I enjoy reading code in other languages and would like to learn a functional language like Haskell when I find the time.
What projects are you working on now?My main focus is deep learning and natural language processing, and I am interested in how AI can make us more productive. In a few years, scientists and programmers will use very different tools than now and will be vastly more effective. Things like GitHub copilot are just the start. I want to be part of that process, either by working on better language models or by integrating language models into next-generation tools.
On the side, I continue working on estimagic together with amazing contributors. The goal of estimagic is to enable scientists who are not experts in numerical optimization to solve challenging optimization problems in practice. They should not have to care too much about selecting algorithms or setting their tuning parameters. We are therefore developing new algorithms that are more adaptive and automatically adjust some tuning parameters that previously had to be specified by a user.
Which Python libraries are your favorite (core or 3rd party)?You mean besides estimagic? There are so many libraries I really like and use a lot:
One of my absolute favorites is JAX. First, it gives you automatic differentiation, Jit compilation, and GPU acceleration almost for free if you know numpy. But it does not stop there. Vmap lets you vectorize functions, and you can thus write simple functions that are easy to test and vectorize later. And due to pytrees (think of them as nested dictionaries if you haven’t heard the term), you can use quite flexible data structures in places where the math (and most libraries) expect one-dimensional numpy arrays.
Pytask is a workflow management system for reproducible research inspired by pytest. It is really easy to use, especially if you already know pytest, and I have used it for all my research projects in my PhD.
One of my favorite core libraries is inspect. It lets you check the signatures of functions at runtime. So if you are wrapping functions, you can look at their signature and call them with the correct arguments.
I am also continuously amazed by the foundational libraries numpy and scipy. None of the things I do would be possible in Python without them. I first really appreciated this at last year’s scipy conference. BTW: I’ll be at the scipy conference in Austin again and happy to chat!
What’s the origin story of estimagic?In computational economics, we encounter a lot of challenging optimization problems. Either to solve economic models or to fit their parameters to data. There were many good open-source optimizers, but they were scattered across different libraries. Most of them forced me to put start parameters into an unlabeled array, making it hard to see which parameter was which. Few provided error handling, logging, and other convenience features.
Estimagic is based on the insight that all of these features can be added to existing optimizers by wrapping them, i.e., without modifying their source code. I wrote a very rudimentary prototype in 2019 where parameters could be provided as a pandas DataFrame (so the index provided names), constraints could be implemented via reparametrization, and the optimization could be monitored in real-time in a dashboard. It was horrible but good enough to excite some people about the idea. Together we built estimagic into something better than I ever would have imagined.
What are some of the most unusual scientific models you have seen estimagic used for?The most unusual application I heard of was not a scientific model. While giving a tutorial on estimagic, I met an aerospace engineer who works on flying taxis that can take off and land vertically. I love the idea there might be a flying taxi that contains a part optimized with estimagic!
Is there anything else you’d like to say?I encourage everyone who uses a small open-source library to contact the authors and provide feedback. As a user, you often get a feature you want or a fix for free. As a maintainer, you get a chance to make your library better. And while you are at it, give them a star on GitHub.
Thanks so much for doing the interview, Janos!
The post PyDev of the Week: Janos Gabler appeared first on Mouse Vs Python.
Python Does What?!: Annotation Inheritance
Type annotations in Python are mostly a static declaration to a type-checker like mypy or pyrightabout the expected types. However, they are also a dynamic data structure which a growing number of libraries such as the original attrsand dataclasses in the standard library, and even sqlalchemyuse at runtime. >>> from dataclasses import dataclass
>>>
>>> @dataclass
... class C:
... a: int
... b: str
...
>>> C(1, "a")
C(a=1, b='a') These libraries inspect the annotations of a class to generate__init__ and __eq__, saving a lot of boilerplate code. You could call this type of API named tuple without the tuple. (To get meta, the typing module has added dataclass_transformwhich libraries can use to properly annotate new class decorators with this API.)
These libraries support inheritance of fields. >>> @dataclass
... class D(C):
... e: int
...
>>> D(1, "a", 2)
D(a=1, b='a', e=2) Type checkers also consider class annotations to be inherited. For example, mypy considers this to be correct: class A:
a: int
class B(A): pass
B().a That code fails at runtime, because nothing is actually setting a on the B instance. But, what if B was a dataclass? >>> class A:
... a: int
...
>>> @dataclass
... class B:
... pass
...
>>> B(1)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: __init__() takes 1 positional argument but 2 were given It doesn't work, because annotations are not inherited. >>> A.__annotations__
{'a': <class 'int'>}
>>> B.__annotations__
{} It's up to the library to look up it's inheritance tree and decide to include the annotations of parents or not when generating code. As it happens, dataclasses has made the design decision to only inherit annotations from other dataclasses.
As an aside, class variables which are used to represent default values are inherited. >>> class A:
... a = 1
...
>>> @dataclass
... class B(A):
... a: int
...
>>> B()
B(a=1)
We can write another decorator which grabs annotations from parents and adds them in method resolution order, as if they were inherited. def inherit_annotations(cls):
annotations = {}
for parent in cls.__mro__[::-1]:
# reverse order so children override parents
annotations.update(getattr(parent, "__annotations__", {}))
# use getattr(): not everything has __annotations__ (e.g. object)
cls.__annotations__.update(annotations)
return cls Since all dataclasses sees is the __annotations__ dict at runtime, any modifications made before the class decorator runs will be reflected in the generated fields. >>> @dataclass
... @inherit_annotations
... class B(A): pass
...
>>> B(1)
B(a=1) Here's a robustified version of the function.
I know what you're thinking though: why not just use multiple class decorators? Sure, all but one of the generated __init__s will be overwritten, but that's fine because they all have the same behavior anyway. import attr
from dataclasses import dataclass
@dataclass
@attr.define
class DualCitizen:
a: int
@dataclass
class Dataclassified(DualCitizen):
pass
@attr.define
class Attrodofined(DualCitizen):
pass Looks like perfectly normal class definitions. >>> DualCitizen(1)
DualCitizen(a=1)
>>> Dataclassified(1)
Dataclassified(a=1)
>>> Attrodofined(1)
Attrodofined(1) And it works.
So, type-checkers consider annotations to be inherited, but class decorators which use annotations at runtime only inherit annotations from ancestors with the same decorator. We can work around this either by multiply decorating the ancestors, or by pulling annotations from ancestors into __annotations__.
Python GUIs: Working With Classes in Python and PyQt
Python supports object-oriented programming (OOP) through classes, which allow you to bundle data and behavior in a single entity. Python classes allow you to quickly model concepts by creating representations of real objects that you can then use to organize your code.
Most of the currently available GUI frameworks for Python developers, such as PyQt, PySide, and Tkinter, rely on classes to provide apps, windows, widgets, and more. This means that you'll be actively using classes for designing and developing your GUI apps.
In this tutorial, you'll learn how OOP and classes work in Python. This knowledge will allow you to quickly grasp how GUI frameworks are internally organized, how they work, and how you can use their classes and APIs to create robust GUI applications.
Defining Classes in PythonPython classes are templates or blueprints that allow us to create objects through instantiation. These objects will contain data representing the object's state, and methods that will act on the data providing the object's behavior.
Instantiation is the process of creating instances of a class by calling the class constructor with appropriate arguments.
Attributes and methods make up what is known as the class interface or API. This interface allows us to operate on the objects without needing to understand their internal implementation and structure.
Alright, it is time to start creating our own classes. We'll start by defining a Color class with minimal functionality. To do that in Python, you'll use the class keyword followed by the class name. Then you provide the class body in the next indentation level:
python >>> class Color: ... pass ... >>> red = Color() >>> type(red) <class '__main__.Color'>In this example, we defined our Color class using the class keyword. This class is empty. It doesn't have attributes or methods. Its body only contains a pass statement, which is Python's way to do nothing.
Even though the class is minimal, it allows us to create instances by calling its constructor, Colo(). So, red is an instance of Color. Now let's make our Color class more fun by adding some attributes.
Adding Class and Instance &AcyttributesPython classes allow you to add two types of attributes. You can have class and instance attributes. A class attribute belongs to its containing class. Its data is common to the class and all its instances. To access a class attribute, we can use either the class or any of its instances.
Let's now add a class attribute to our Color class. For example, let's say we need to keep note of how many instance of Color your code creates. Then you can have a color_count attribute:
python >>> class Color: ... color_count = 0 ... def __init__(self): ... Color.color_count += 1 ... >>> red = Color() >>> green = Color() >>> Color.color_count 2 >>> red.color_count 2Now Color has a class attribute called color_count that gets incremented every time we create a new instance. We can quickly access that attribute using either the class directly or one of its instances, like red.
To follow up with this example, say that we want to represent our Color objects using red, green, and blue attributes as part of the RGB color model. These attributes should have specific values for specific instances of the class. So, they should be instance attributes.
To add an instance attribute to a Python class, you must use the .__init__() special method, which we introduced in the previous code but didn't explain. This method works as the instance initializer because it allows you to provide initial values for instance attributes:
python >>> class Color: ... color_count = 0 ... def __init__(self, red, green, blue): ... Color.color_count += 1 ... self.red = red ... self.green = green ... self.blue = blue ... >>> red = Color(255, 0, 0) >>> red.red 255 >>> red.green 0 >>> red.blue 0 >>> Color.red Traceback (most recent call last): ... AttributeError: type object 'Color' has no attribute 'red'Cool! Now our Color class looks more useful. It has the usual class attributes and also three new instance attributes. Note that, unlike class attributes, instance attributes can't be accessed through the class itself. They're specific to a concrete instance.
There's something that jumps into sight in this new version of Color. What is the self argument in the definition of .__init__()? This attribute holds a reference to the current instance. Using the name self to identify the current instance is a strong convention in Python.
We'll use self as the first or even the only argument to instance methods like .__init__(). Inside an instance method, we'll use self to access other methods and attributes defined in the class. To do that, we must prepend self to the name of the target attribute or method instance of the class.
For example, our class has an attribute .red that we can access using the syntax self.red inside the class. This will return the number stored under that name. From outside the class, you need to use a concrete instance instead of self.
Providing Behavior With MethodsA class bundles data (attributes) and behavior (methods) together in an object. You'll use the data to set the object's state and the methods to operate on that data or state.
Methods are just functions that we define inside a class. Like functions, methods can take arguments, return values, and perform different computations on an object's attributes. They allow us to make our objects usable.
In Python, we can define three types of methods in our classes:
- Instance methods, which need the instance (self) as their first argument
- Class methods, which take the class (cls) as their first argument
- Static methods, which take neither the class nor the instance
Let's now talk about instance methods. Say that we need to get the attributes of our Color class as a tuple of numbers. In this case, we can add an .as_tuple() method like the following:
python class Color: representation = "RGB" def __init__(self, red, green, blue): self.red = red self.green = green self.blue = blue def as_tuple(self): return self.red, self.green, self.blueThis new method is pretty straightforward. Since it's an instance method, it takes self as its first argument. Then it returns a tuple containing the attributes .red, .green, and .blue. Note how you need to use self to access the attributes of the current instance inside the class.
This method may be useful if you need to iterate over the RGB components of your color objects:
python >>> red = Color(255, 0, 0) >>> red.as_tuple() (255, 0, 0) >>> for level in red.as_tuple(): ... print(level) ... 255 0 0Our as_tuple() method works great! It returns a tuple containing the RGB components of our color objects.
We can also add class methods to our Python classes. To do this, we need to use the @classmethod decorator as follows:
python class Color: representation = "RGB" def __init__(self, red, green, blue): self.red = red self.green = green self.blue = blue def as_tuple(self): return self.red, self.green, self.blue @classmethod def from_tuple(cls, rbg): return cls(*rbg)The from_tuple() method takes a tuple object containing the RGB components of a desired color as an argument, creates a valid color object from it, and returns the object back to the caller:
python >>> blue = Color.from_tuple((0, 0, 255)) >>> blue.as_tuple() (0, 0, 255)In this example, we use the Color class to access the class method from_tuple(). We can also access the method using a concrete instance of this class. However, in both cases, we'll get a completely new object.
Finally, Python classes can also have static methods that we can define with the @staticmethod decorator:
python class Color: representation = "RGB" def __init__(self, red, green, blue): self.red = red self.green = green self.blue = blue def as_tuple(self): return self.red, self.green, self.blue @classmethod def from_tuple(cls, rbg): return cls(*rbg) @staticmethod def color_says(message): print(message)Static methods don't operate either on the current instance self or the current class cls. These methods can work as independent functions. However, we typically put them inside a class when they are related to the class, and we need to have them accessible from the class and its instances.
Here's how the method works:
python >>> Color.color_says("Hello from the Color class!") Hello from the Color class! >>> red = Color(255, 0, 0) >>> red.color_says("Hello from the red instance!") Hello from the red instance!This method accepts a message and prints it on your screen. It works independently from the class or instance attributes. Note that you can call the method using the class or any of its instances.
Writing Getter & Setter MethodsProgramming languages like Java and C++ rely heavily on setter and getter methods to retrieve and update the attributes of a class and its instances. These methods encapsulate an attribute allowing us to get and change its value without directly accessing the attribute itself.
For example, say that we have a Label class with a text attribute. We can make text a non-public attribute and provide getter and setter methods to manipulate the attributes according to our needs:
python class Label: def __init__(self, text): self.set_text(text) def text(self): return self._text def set_text(self, value): self._text = str(value)In this class, the text() method is the getter associated with the ._text attribute, while the set_text() method is the setter for ._text. Note how ._text is a non-public attribute. We know this because it has a leading underscore on its name.
The setter method calls str() to convert any input value into a string. Therefore, we can call this method with any type of object. It will convert any input argument into a string, as you will see in a moment.
If you come from programming languages like Java or C++, you need to know Python doesn't have the notion of private, protected, and public attributes. In Python, you'll use a naming convention to signal that an attribute is non-public. This convention consists of adding a leading underscore to the attribute's name. Note that this naming pattern only indicates that the attribute isn't intended to be used directly. It doesn't prevent direct access, though.
This class works as follows:
python >>> label = Label("Python!") >>> label.text() 'Python!' >>> label.set_text("PyQt!") >>> label.text() 'PyQt!' >>> label.set_text(123) >>> label.text() '123'In this example, we create an instance of Label. The original text is passed to the class constructor, Label(), which automatically calls __init__() to set the value of ._text by calling the setter method text(). You can use text() to access the label's text and set_text() to update it. Remember that any input will be converted into a string, as we can see in the final example above.
Note that the Label class above is just a toy example, don't confuse this class with similarly named classes from GUI frameworks like PyQt, PySide, and Tkinter.
The getter and setter pattern is pretty common in languages like Java and C++. Because PyQt and PySide are Python bindings to the Qt library, which is written in C++, you'll be using this pattern a lot in your Qt-based GUI apps. However, this pattern is less popular among Python developers. Instead, they use the @property decorator to hide attributes behind properties.
Here's how most Python developer will write their Label class:
python class Label: def __init__(self, text): self.text = text @property def text(self): return self._text @text.setter def text(self, value): self._text = str(value)This class defines .text as a property. This property has getter and setter methods. Python calls them automatically when we access the attribute or update its value in an assignment:
python >>> label = Label("Python!") >>> label.text 'Python!' >>> label.text = "PyQt" >>> label.text 'PyQt' >>> label.text = 123 >>> label.text '123'Python properties allow you to add function behavior to your attributes while permitting you to use them as normal attributes instead of as methods.
Writing Special MethodsPython supports many special methods, also known as dunder or magic methods, that are part of its class mechanism. We can identify these methods because their names start and end with a double underscore, which is the origin of their other name: dunder methods.
These methods accomplish different tasks in Python's class mechanism. They all have a common feature: Python calls them automatically depending on the operation we run.
For example, all Python objects are printable. We can print them to the screen using the print() function. Calling print() internally falls back to calling the target object's __str__() special method:
python >>> label = Label("Python!") >>> print(label) <__main__.Label object at 0x10354efd0>In this example, we've printed our label object. This action provides some information about the object and the memory address where it lives. However, the actual output is not very useful from the user's perspective.
Fortunately, we can improve this by providing our Label class with an appropriate __str__() method:
python class Label: def __init__(self, text): self.text = text @property def text(self): return self._text @text.setter def text(self, value): self._text = str(value) def __str__(self): return self.textThe __str__() method must return a user-friendly string representation for our objects. In this case, when we print an instance of Label to the screen, the label's text will be displayed:
python >>> label = Label("Python!") >>> print(label) Python!As you can see, Python takes care of calling __str__() automatically when we use the print() function to display our instances of Label.
Another special method that belongs to Python's class mechanism is __repr__(). This method returns a developer-friendly string representation of a given object. Here, developer-friendly implies that the representation should allow a developer to recreate the object itself.
python class Label: def __init__(self, text): self.text = text @property def text(self): return self._text @text.setter def text(self, value): self._text = str(value) def __str__(self): return self.text def __repr__(self): return f"{type(self).__name__}(text='{self.text}')"The __repr__() method returns a string representation of the current objects. This string differs from what __str__() returns:
python >>> label = Label("Python!") >>> label Label(text='Python!')Now when you access the instance on your REPL session, you get a string representation of the current object. You can copy and paste this representation to recreate the object in an appropriate environment.
Reusing Code With InheritanceInheritance is an advanced topic in object-oriented programming. It allows you to create hierarchies of classes where each subclass inherits all the attributes and behaviors from its parent class or classes. Arguably, code reuse is the primary use case of inheritance.
Yes, we code a base class with a given functionality and make that functionality available to its subclass through inheritance. This way, we implement the functionality only once and reuse it in every subclass.
Python classes support single and multiple inheritance. For example, let's say we need to create a button class. This class needs .width and .height attributes that define its rectangular shape. The class also needs a label for displaying some informative text.
We can code this class from scratch, or we can use inheritance and reuse the code of our current Label class. Here's how to do this:
python class Button(Label): def __init__(self, text, width, height): super().__init__(text) self.width = width self.height = height def __repr__(self): return ( f"{type(self).__name__}" f"(text='{self.text}', " f"width={self.width}, " f"height={self.height})" )To inherit from a parent class in Python, we need to list the parent class or classes in the subclass definition. To do this, we use a pair of parentheses and a comma-separated list of parent classes. If we use several parent classes, then we're using multiple inheritance, which can be challenging to reason about.
The first line in __init__() calls the __init__() method on the parent class to properly initialize its .text attribute. To do this, we use the built-in super() function. Then we define the .width and .height attributes, which are specific to our Button class. Finally, we provide a custom implementation of __repr__().
Here's how our Button class works:
python >>> button = Button("Ok", 10, 5) >>> button.text 'Ok' >>> button.text = "Click Me!" >>> button.text 'Click Me!' >>> button.width 10 >>> button.height 5 >>> button Button(text='Ok', width=10, height=5) >>> print(button) Click Me!As you can conclude from this code, Button has inherited the .text attribute from Label. This attribute is completely functional. Our class has also inherited the __str__() method from Label. That's why we get the button's text when we print the instance.
Using Classes in PyQt GUI AppsEverything we've learned so far about Python classes is the basis of our future work in GUI development. When it comes to working with PyQt, PySide, Tkinter, or any other GUI framework, we'll heavily rely on our knowledge of classes and OOP because most of them are based on classes and class hierarchies.
We'll now look at how to use inheritance to create some GUI-related classes. For example, when we create an application with PyQt or PySide, we usually have a main window. To create this window, we typically inherit from QMainWindow:
- PyQt6
- PySide6
In the definition of our Window class, we use the QMainWindow class as the parent class. This tells Python that we want to define a class that inherits all the functionalities that QMainWindow provides.
We can continue adding attributes and methods to our Window class. Some of these attributes can be GUI widgets, such as labels, buttons, comboboxes, checkboxes, line edits, and many others. In PyQt, we can create all these GUI components using classes such as QLabel, QPushButton, QComboBox, QCheckBox, and QLineEdit.
All of them have their own sets of attributes and methods that we can use according to our specific needs when designing the GUI of a given application.
Wrapping Up Classes-Related ConceptsAs we've seen, Python allows us to write classes that work as templates that you can use to create concrete objects that bundle together data and behavior. The building blocks of Python classes are:
- Attributes, which hold the data in a class
- Methods, which provide the behaviors of a class
The attributes of a class define the class's data, while the methods provide the class's behaviors, which typically act on that data.
To better understand OOP and classes in Python, we should first discuss some terms that are commonly used in this aspect of Python development:
-
Classes are blueprints or templates for creating objects -- just like a blueprint for creating a car, plane, house, or anything else. In programming, this blueprint will define the data (attributes) and behavior (methods) of the object and will allow us to create multiple objects of the same kind.
-
Objects or Instances are the realizations of a class. We can create objects from the blueprint provided by the class. For example, you can create John's car from a Car class.
-
Methods are functions defined within a class. They provide the behavior of an object of that class. For example, our Car class can have methods to start the engine, turn right and left, stop, and so on.
-
Attributes are properties of an object or class. We can think of attributes as variables defined in a class or object. Therefore, we can have:
- class attributes, which are specific to a concrete class and common to all the instances of that class. You can access them either through the class or an object of that class. For example, if we're dealing with a single car manufacturer, then our Car class can have a manufacturer attribute that identifies it.
- instance attributes, which are specific to a concrete instance. You can access them through the specific instance. For example, our Car class can have attributes to store properties such as the maximum speed, the number of passengers, the car's weight, and so on.
-
Instantiation is the process of creating an individual instance from a class. For example, we can create John's car, Jane's car, and Linda's car from our Car class through instantiation. In Python, this process runs through two steps:
- Instance creation: Creates a new object and allocates memory for storing it.
- Instance initialization: Initializes all the attributes of the current object with appropriate values.
-
Inheritance is a mechanism of code reuse that allows us to inherit attributes and methods from one or multiple existing classes. In this context, we'll hear terms like:
- Parent class: The class we're inheriting from. This class is also known as the superclass or base class. If we have one parent class, then we're using single inheritance. If we have more than one parent class, then we're using multiple inheritance.
- Child class: The class that inherits from a given parent. This class is also known as the subclass.
Don't feel frustrated or bad if you don't understand all these terms immediately. They'll become more familiar with use as you use them in your own Python code. Many of our GUI tutorials make use of some or all of these concepts.
ConclusionNow you know the basics of Python classes. You also learned fundamental concepts of object-oriented programming, such as inheritance. You also learned that most GUI frameworks are heavily based on classes. Therefore knowing about classes will open the door to begin building your own GUI app using PyQt, PySide, Tkinter, or any other GUI framework for Python.
For an in-depth guide to building GUIs with Python see my PyQt6 book.
Codementor: Make your own Library in C Programming Language
Test and Code: 194: Test & Code Returns
A brief discussion of why Test & Code has been off the air for a bit, and what to expect in upcoming episodes.
Links:
<p>A brief discussion of why Test & Code has been off the air for a bit, and what to expect in upcoming episodes.</p><p>Links:</p><ul><li><a href="https://pythontest.com/pytest-book/" title="Python Testing with pytest, 2nd Edition" rel="nofollow">Python Testing with pytest, 2nd Edition</a></li><li><a href="https://training.talkpython.fm/courses/getting-started-with-testing-in-python-using-pytest" title="Getting started with pytest Online Course" rel="nofollow">Getting started with pytest Online Course</a></li><li><a href="https://pythontest.com/training/" title="Software Testing with pytest Training" rel="nofollow">Software Testing with pytest Training</a></li><li><a href="https://pythonbytes.fm/" title="Python Bytes Podcast" rel="nofollow">Python Bytes Podcast</a></li></ul>Brian Okken: Test & Code Returns
Pythonicity: Decorator overuse
Decorators are a beloved feature of Python, but like any good thing can be overused. The key is acknowledging that decorators are just functions.
A function returning another function, usually applied as a function transformation using the @wrapper syntax. Common examples for decorators are classmethod() and staticmethod().
The decorator syntax is merely syntactic sugar, the following two function definitions are semantically equivalent:
def f(arg): ... f = staticmethod(f) @staticmethod def f(arg): ... RenamedSo the critical feature of the @ syntax is to retain the defined object’s name; otherwise it is just a function call. Which leads to the first example of overuse: defining a new object just to change the name. Consider this example adapted from a popular project.
class Response: def __bool__(self): return self.ok @property def ok(self): ...Since a property wraps a function, it is natural to make the function have the implementation instead. Then it becomes clear that the property does not share the same name, so why bother with @.
class Response: def __bool__(self): ... ok = property(__bool__)A related scenario is where the local name of the function is irrelevant, which is typical in wrapped functions:
@functools.wraps(wrapped, assigned=WRAPPER_ASSIGNMENTS, updated=WRAPPER_UPDATES)
This is a convenience function for invoking update_wrapper() as a function decorator when defining a wrapper function. It is equivalent to partial(update_wrapper, wrapped=wrapped, assigned=assigned, updated=updated). For example:
>>> from functools import wraps >>> def my_decorator(f): ... @wraps(f) ... def wrapper(*args, **kwds): ... print('Calling decorated function') ... return f(*args, **kwds) ... return wrapperThe “convenience function” is useless indirection when the wrapper is immediately returned. Even the documentation points out that wraps is just a partial function. The example could be simply:
def my_decorator(f): def wrapper(*args, **kwds): print('Calling decorated function') return f(*args, **kwds) return update_wrapper(wrapper, f)Giving partial(update_wrapper, wrapped=f) a short name does not make it any clearer conceptually.
With blocksAnother sign is if the decorator’s functionality only executes code before or after the wrapped function. Context managers are inherently more flexible by providing the same functionality for any code block. In some cases a function boundary is natural to bookend, e.g., logging or timing. The question is whether the function block is too broad a context to manage.
Decorators were introduced in version 2.4; context managers in 2.5. All ancient history now, but decorators had a ~2 year head start. For example, transactions are a seminal use case for context managers, but Django pre-dates 2.5, so it had a transaction decorator first. This is how transactions are currently presented:
atomic is usable both as a decorator:
from django.db import transaction @transaction.atomic def viewfunc(request): # This code executes inside a transaction. do_stuff()and as a context manager:
from django.db import transaction def viewfunc(request): # This code executes in autocommit mode (Django's default). do_stuff() with transaction.atomic(): # This code executes inside a transaction. do_more_stuff()So it has both, but the decorator is presented first, and is it a good example? Seems likely that a full request would have setup and teardown work that is unrelated to a database transaction. It is uncontroversial to want try blocks to be as narrow as possible. Surely there is no benefit to a request operation rolling back a vacuous transaction, nor a response operation rolling back a transaction that was committable.
Any context manager can be trivially transformed into a decorator; the converse is not true. And even if the function block is coincidentally perfect, a with block has negligible impact on readability. It is just indentation.
Partial functionsNext is a lack of appreciation of partially bound functions. Many decorator examples go out of their way to write an unnecessary def statement, in order to make using a decorator look natural. The below example is common in Python tutorials.
import functools def repeat(num_times): def decorator_repeat(func): @functools.wraps(func) def wrapper_repeat(*args, **kwargs): for _ in range(num_times): value = func(*args, **kwargs) return value return wrapper_repeat return decorator_repeat @repeat(num_times=4) def greet(name): print(f"Hello {name}") greet("World") Hello World Hello World Hello World Hello WorldFirst the obligatory observation that abstracting a for loop in Python is not necessarily a good idea. But assuming that is the goal, it is still worth questioning why repeating 4 times is coupled to the name greet. Is print supposed to represent the “real” function in this example, or should the wrapped function be named greet_4x? It is much simpler to start with the basic functionality and postpone how to wrap it.
def repeat(num_times, func, *args, **kwargs): for _ in range(num_times): value = func(*args, **kwargs) return value def greet(name): print(f"Hello {name}") repeat(4, greet, "World") Hello World Hello World Hello World Hello WorldWe can stop there really. But even assuming that the goal is to bind the repetition, using partial functions is still simpler.
from functools import partial greet_4x = partial(repeat, 4, greet) greet_4x("World") Hello World Hello World Hello World Hello WorldNot exactly the same without wraps, but that would be trivial to add. Futhermore it is less useful because partial objects can be easily introspected. Now onto the next - and dubious - assumption: that we really want it used as a decorator. This requires assuming the body of greet is not a simple call to an underlying wrapped function, and yet for some reason the repetition is supposed to be coupled to the wrapper function’s name anyway. Still simpler:
repeats = partial(partial, repeat, 4) @repeats def greet(name): print(f"Hello {name}") greet("World") Hello World Hello World Hello World Hello WorldNested partials may appear a little too clever, but they are just the flatter version of the original nested repeat functions. And again, none of this indirection is necessary.
For loopsA real-world example of repeat is retrying functions until success, optionally with delays. A popular one uses examples like:
@backoff.on_exception(backoff.expo, requests.exceptions.RequestException) def get_url(url): return requests.get(url)The same pattern (ahem) repeats. The decorated function is a trivial wrapper around the “real” function. Why not:
get_url = backoff.on_exception(backoff.expo, requests.exceptions.RequestException)(requests.get)Furthermore, for loops can be customized via the __iter__ protocol, just as with blocks are customizable. The author’s waiter package demonstrates the same functionality with for loops and undecorated functions.
AdvocacySo before assuming a decorator is the right abstraction, start with whether a def function is the right abstraction. Building out functionality in this progression works well:
- code blocks: with and for and customizable
- flat functions
- nested functions: using partial
- decorated functions
Go Deh: Function purity and idempotence
Someone mentioned idempotence at work. I looked it up and noted that it too is a property of functions, like function purity.
I decided to see if I could write functions with combinations of those properties and embedded tests for those properties.
Running my resultant program produces this result:
Created on Fri Mar 3 18:04:09 2023@author: Paddy3118
pure_idempotent.py
Explores Purity and idempotence with Python examples
Definitions: Pure: * Answer relies solely on inputs. Same out for same in. I.E: `f(x) == f(x) == f(x) == ...` * No side-effects.
Idempotent: * The first answer from any input, if used as input to subsequent runs of the function, will all yield the same answer. I.E: `f(x) == f(f(x)) == f(f(f(x))) == ...` * Any side effect of a first function execution is *preserved* on subsequent runs of the function using the previous answer.
Side effect: * A function is said to have side effects if it relies apon or modifies state outside of that *given* by its arguments. Modifying mutable arguments is also a side effect.
#--------
def remove_twos(arg: list[int]) -> list[int]: "Returns a copy of the list with all twos removed." return [x for x in arg if x != 2]
Function is: Pure Idempotent
#--------
def return_first_int(arg: int) -> int: "Return the int given in its first call" global external_state
if external_state is None: external_state = arg return external_state
Function is: Impure! External state changed Idempotent
#--------
def plus_one(arg: int) -> int: "Add one to arg" return arg + 1
Function is: Pure Non-idempotent! Output changes for nested calls
#--------
def epoc_plus_seconds(secs: float) -> float: "Return time since epoch + seconds" time.sleep(0.1) return time.time() + secs
Function is: Impure! Output changes for same input Non-idempotent! Output changes for nested calls
Code
The code that produces the above (but not its arbitrary colourising), is the following:
# -*- coding: utf-8 -*-"""Created on Fri Mar 3 18:04:09 2023@author: Paddy3118
pure_idempotent.py
Explores Purity and idempotence with Python examples
Definitions: Pure: * Answer relies solely on inputs. Same out for same in. I.E: `f(x) == f(x) == f(x) == ...` * No side-effects.
Idempotent: * The first answer from any input, if used as input to subsequent runs of the function, will all yield the same answer. I.E: `f(x) == f(f(x)) == f(f(f(x))) == ...` * Any side effect of a first function execution is *preserved* on subsequent runs of the function using the previous answer.
Side effect: * A function is said to have side effects if it relies apon or modifies state outside of that *given* by its arguments. Modifying mutable arguments is also a side effect."""
import inspect
print(__doc__)
# %% Pure, idempotent.print('\n#--------')
def remove_twos(arg: list[int]) -> list[int]: "Returns a copy of the list with all twos removed." return [x for x in arg if x != 2]
print(f"\n{inspect.getsource(remove_twos)}")arg0 = [1, 2, 3, 2, 4, 5, 2]print('Function is:')print(' Pure' if remove_twos(arg0.copy()) == remove_twos(arg0.copy()) else ' Impure')print(' Idempotent' if (answer1:=remove_twos(arg0)) == remove_twos(answer1) else 'Non-idempotent')
# %% Impure, idempotent.print('\n#--------')
def return_first_int(arg: int) -> int: "Return the int given in its first call" global external_state
if external_state is None: external_state = arg return external_state
print(f"\n{inspect.getsource(return_first_int)}")# Purityexternal_state = initial_state = Nonearg0 = 1same_output = (return_first_int(arg0)) == return_first_int(arg0)same_state = external_state == initial_stateprint('Function is:')if same_output and same_state: print(' Pure')else: if not same_output: print(' Impure! Output changes for same input') if not same_state: print(' Impure! External state changed')# Idempotenceexternal_state = Noneanswer1, state1 = return_first_int(arg0), external_stateanswer2, state2 = return_first_int(answer1), external_statesame_output = answer1 == answer2same_state = state1 == state2if same_output and same_state: print(' Idempotent')else: if not same_output: print(' Non-idempotent! Output changes for nested calls') if not same_state: print(' Non-idempotent! External state changes for nested calls')
# %% Pure, non-idempotent.print('\n#--------')
def plus_one(arg: int) -> int: "Add one to arg" return arg + 1
print(f"\n{inspect.getsource(plus_one)}")# Purityarg0 = 1same_output = (plus_one(arg0)) == plus_one(arg0)print('Function is:')if same_output: print(' Pure')else: print(' Impure! Output changes for same input')# Idempotenceanswer1 = plus_one(arg0)answer2 = plus_one(answer1)same_output = answer1 == answer2if same_output: print(' Idempotent')else: print(' Non-idempotent! Output changes for nested calls')
# %% Impure, non-idempotent.print('\n#--------')
import time
def epoc_plus_seconds(secs: float) -> float: "Return time since epoch + seconds" time.sleep(0.1) return time.time() + secs
print(f"\n{inspect.getsource(epoc_plus_seconds)}")# Purityarg0 = 1same_output = (epoc_plus_seconds(arg0)) == epoc_plus_seconds(arg0)print('Function is:')if same_output: print(' Pure')else: print(' Impure! Output changes for same input')# Idempotenceanswer1 = epoc_plus_seconds(arg0)answer2 = epoc_plus_seconds(answer1)same_output = answer1 == answer2if same_output: print(' Idempotent')else: print(' Non-idempotent! Output changes for nested calls')
END.
PyBites: Jim Hodapp on coaching software engineers and the power of Rust
Watch here:
Or listen here:
This week we have Jim Hodapp on our podcast.
We talk about his career journey going from software engineer + manager to full-time developer coach, some of the tactics he uses with his clients, and why coaching is a powerful tool for software engineers.
Then we pivot to a more technical discussion about Rust, his passion for the language, why it’s an interesting language to consider, also for Python developers, and to his developer community Rust Never Sleeps.
We hope you enjoy this interview and that it inspires (and challenges) you to keep learning new things and expand your horizons.
Links:
– Embedded Rust WiFi crate for RP2040 microcontrollers
– Rust web application to monitor home ambient air conditions
– Jim’s coaching/Rust community
– Connect with Jim: Pybites / Twitter / LinkedIn
– Mentioned books: The Staff Engineer’s Path / Thich Nhat Hanh Essential Writings / The School of Life
Pages
Recent Publications
- Managing Hidden Dependencies in OO Software: a Study Based on Open Source Projects
- Open Source Communities as Liminal Ecosystems
- Investigating developers' email discussions during decision-making in Python language evolution
- Developers, Quality Control and Download Volume in Open Source Software (OSS) Projects
More...