Planet Python
Talk Python to Me: #468: Python Trends Episode 2024
eGenix.com: eGenix PyRun - One file Python Runtime 2.5.0 GA
eGenix PyRun™ is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run Python based scripts and applications to Unix based systems simple and efficient.
eGenix PyRun's executable only needs 4-6MB on disk, but still supports most Python applications and scripts.
Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to containers, VMs, clusters, client installations, customers or end-users.
It makes "installing" Python on a Unix based system as simple as copying a single file.
eGenix has been using eGenix PyRun as run-time for the Linux version of mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product.
We provide the source archive to build your own eGenix PyRun on Github, as well as a few binary distributions to get you started on Linux x86_64. In the future, we will set up automated builds for several other platforms.Please see the product page for more details:
>>> eGenix PyRun - One file Python Runtime
NewsThis major release of eGenix PyRun comes with the following enhancements:
Enhancements / Changes- Added support for Python 3.8 - 3.11
- Removed support for Python 3.5-3.7
- Modernized the directory setup and build
- Changed the license to the Apache2 license
- Extracted the code from our internal mono-repo to put on Github
- Relaunched the project on Github
Please visit the eGenix PyRun product page for downloads, instructions on installation and documentation of the product.
Commercial support for this product is available directly from eGenix.com.
Please see the support section of our website for details.
For more information on eGenix PyRun, licensing and download instructions, please write to sales@egenix.com.
Enjoy !
Marc-Andre Lemburg, eGenix.com
EuroPython: How to Maximize Your Experience at EuroPython 2024 🐍✨
EuroPython 2024 is set to be an electrifying event for Python enthusiasts, taking place in the vibrant city of Prague from July 8th to 14th. Whether you&aposre a first-timer or a seasoned attendee, here are some tips to help you squeeze every bit of goodness out of the conference.
Plan Your Schedule in Advance 🗓️Review the EuroPython schedule to spot sessions, workshops, and events that spark your interest. The week is loaded with activities: Monday and Tuesday are for tutorials and free workshops (C API Summit, HumbleData, DjangoGirls and WASM Summit); Wednesday to Friday are full of keynotes and talks, social events, PyLadies events and panel discussions. Finally, the weekend is reserved for the Open Source Sprints.
Mix and match your experience by attending a variety of sessions! From technical talks and tutorials to keynote speeches, there&aposs a smorgasbord of content for all levels. We have talks for beginners and advanced Pythonistas including tracks like Python internals, LLMs, Ethics & Philosophy, Web Technologies, Education & Community, DevOps, and so much more.
Pro tip: Keynote speakers always drop some serious knowledge bombs 😉Engage in Workshops & Summits 🛠️Dive into workshops for a hands-on learning adventure. Whether you&aposre a newbie learning to build websites with Django Girls or exploring data science with HumbleData, there&aposs something for everyone. Discuss the state of the C API at the C API Summit or get the latest on WebAssembly at the WASM Summit.
Some of these events have limited slots, while others are first-come, first-served. Make sure to register on the website https://ep2024.europython.eu/programme and arrive early on the day of the event!
Network Actively 🤝Sessions are packed with information, but they might not cover your particular problem. Don&apost hesitate to ask the speakers about anything they didn&apost cover. Make sure to use Discord to reach out if you cannot find them in the corridors.
All participants should join our Discord server as this is the main communication platform during the conference. The link invite will be sent via email on the following days.
Take advantage of breaks to mingle and tap into the wealth of knowledge around you. And don&apost forget the Pac-Man rule: always leave a spot open so others can easily join the conversation.
Engage in the Hallway & Harness the power of networking!Chat with fellow attendees, speakers, and sponsors during breaks, social events, and open spaces. Networking can lead to new collaborations, job opportunities, and friendships within the Python community. Here are some tips to help you start a conversation
- Look for lanyards with the "speaker" tag and ask them about their topic.
- Approach someone with a simple "Hi, what do you think of the event so far?" or "What sessions have you found interesting?"
- Pay attention to the stickers on people&aposs badges indicating their preferred level of contact: "no contact," "happy to hug," or "handshake only." Always respect these preferences.
Remember it&aposs normal to feel shy, many people do, but the Python community is known for being extra nice :) Don&apost miss out on the chance to connect with others, this is the perfect place to try and reach out.
Visit the Sponsors Hall 🛍️Stroll through the Sponsor&aposs Hall to check out the latest products and services from exhibitors. It&aposs swag central and a goldmine for discovering new tools and technologies to boost your Python projects. A lot of our sponsors are also actively hiring in case you are on the lookout.
If you are attending remotely, Sponsors will have channels where you can chat and find out more. Remember to also keep an eye on the Virtual Swag and Job board on our website for extra gifts.
Attend Social Events 🎉Don&apost miss the social events like the PyLadies lunch and Social Event, Prague Boat Trip, Speakers’ Dinner, and the big Social Event where you are invited to bring your board games and instruments!
These events are perfect for connecting & sharing ideas with like-minded individuals in a relaxed setting. Plus, they are a whole lot of fun! 🎉
Take Care of Yourself 🧘Conferences are a marathon, not a sprint. Stay hydrated, eat well, and get plenty of rest. We will help by serving light lunches, snacks and coffee during the breaks throughout the day. Take breaks to recharge and keep your energy levels high.
Feeling overwhelmed? We have a Quiet Room and a low-stimulation room at the venue so you can wind down.
Explore Prague 🏰Seize the chance to explore the enchanting city of Prague. Dive into the local culture, savour the cuisine, and visit the landmarks. The Prague Congress Centre is conveniently located, making it easy to venture out.
Our volunteers wrote a page with tips and things to do: https://ep2024.europython.eu/explore
We hope it helps you and your companions to have a great time in Prague 😃
Share Your Experience 📢After the conference, share your newfound wisdom with your community. Write a blog post, give a presentation, or organize a meetup to spread the knowledge. It reinforces what you&aposve learned and helps those who couldn&apost attend.
Make sure to tag us on socials @EuroPython or use #EuroPython2024. We will be reposting the coolest community posts during the event!
ConclusionEuroPython 2024 promises to be an enriching experience brimming with learning, networking, and fun. By planning ahead, engaging actively, and taking care of yourself, you&aposll make the most of this incredible event.
See you in Prague! 🌟
For more details, visit the EuroPython 2024 official website.
TestDriven.io: Reusable Components in Django with Stimulus and Tailwind CSS - Part 1
Sebastian Pölsterl: scikit-survival 0.23.0 released
I am pleased to announce the release of scikit-survival 0.23.0.
This release adds support for scikit-learn 1.4 and 1.5, which includes missing value support for RandomSurvivalForest. For more details on missing values support, see the section in the release announcement for 0.23.0.
Moreover, this release fixes critical bugs. When fitting SurvivalTree, the sample_weight is now correctly considered when computing the log-rank statistic for each split. This change also affects RandomSurvivalForest and ExtraSurvivalTrees which pass sample_weight to the individual trees in the ensemble. Therefore, the outputs produced by SurvivalTree, RandomSurvivalForest, and ExtraSurvivalTrees will differ from previous releases.
This release fixes a bug in ComponentwiseGradientBoostingSurvivalAnalysis and GradientBoostingSurvivalAnalysis when dropout is used. Previously, dropout was only applied starting with the third iteration, now dropout is applied in the second iteration too.
Finally, this release adds compatibility with numpy 2.0 and drops support for Python 3.8.
Installscikit-survival is available for Linux, macOS, and Windows and can be installed either
via pip:
pip install scikit-survivalor via conda
conda install -c conda-forge scikit-survivalPython GUIs: PyQt6, PySide6, PyQt5 and PySide2 Books -- updated for 2024! — Extended and updated with new examples, demos including Model View Controller architecture
Hello! Today I have released new digital updates to my PyQt5, PyQt6, PySide2 and PySide6 book Create GUI Applications with Python & Qt.
This update brings all versions up to date with the latest developments in Qt, As well as corrections and additions to existing chapters, there are new sections dealing with form layouts, built-in dialogs and developing Qt applications using a Model View Controller (MVC) architecture.
As always, if you've previously bought a copy of the book you get these updates for free! Just go to the downloads page and enter the email you used for the purchase.
You can buy the latest editions below --
- PyQt6 Book 5th Edition, Create GUI Applications with Python & Qt6
- PySide6 Book, 5th Edition, Create GUI Applications with Python & Qt6
- PyQt5 Book 5th Edition, Create GUI Applications with Python & Qt5
- PySide2 Book 5th Edition, Create GUI Applications with Python & Qt5
If you bought the book elsewhere (in paperback or digital) you can register to get these updates too. Email your receipt to register@pythonguis.com
Enjoy!
Zero to Mastery: Python Monthly Newsletter 💻🐍
Awesome Python Applications: aider
aider: Console-based LLM pair programming tool, to edit code in your local git repository.
Links:
Awesome Python Applications: napar
napar: A fast, interactive, multi-dimensional image viewer for annotation and analysis of large images.
Links:
Awesome Python Applications: Plane
Plane: Modern, self-hostable issue and product roadmap tracker. An alternative to JIRA, Linear, and Asana.
Links:
Real Python: The Real Python Podcast – Episode #210: Creating a Guitar Synthesizer & Generating WAV Files With Python
What techniques go into synthesizing a guitar sound in Python? What higher-level programming and Python concepts can you practice while building advanced projects? This week on the show, we talk with Real Python author and core team member Bartosz Zaczyński about his recent step-by-step project, Build a Guitar Synthesizer: Play Musical Tablature in Python.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Robin Wilson: A load of links…
For months now I’ve been collecting a load of links saying that I’ll get round to blogging them "soon". Well, I’m currently babysitting for a friend’s daughter (who is sleeping peacefully upstairs), so I’ve finally found time to write them up.
So, here are a load of links – a lot of them are geospatial- or data-related, but I hope you might find something here you like even if those aren’t your specialisms:
- QGIS Visual Changelog for v3.36 – QGIS is great GIS software that I use a lot in my work. The reason this link is here though is because of their ‘visual changelog’ that they write for every new version – it is a really good way of showing what has changed in a GUI application, and does a great job of advertising the new features.
- List of unusual units of measurement – a fascinating list from Wikipedia. I think my favourite are the barn (and outhouse/shed) and the shake – you’ll have to read the article to see what these are!
- List of humourous units of measurement – linked from the above is this list of humourous units of measurement, including the ‘beard-second’, and the Smoot.
- lonboard – a relatively new package for very efficiently creating interactive webmaps of geospatial data from Python. It is so much faster than geopandas.explore(), and is developing at a rapid pace. I’m using it a lot these days.
- A new metric for measuring learning – an interesting post from Greg Wilson about the time it takes learners to realise that something was worth learning. I wonder what the values are for some things that I do a lot of – remote sensing, GIS, Python, cloud computing, cloud-native geospatial etc.
- The first global 30-m land-cover dynamic monitoring product with fine classification system from 1985 to 2022 – an interesting dataset that I haven’t had chance to investigate in detail yet, but purports to give 30m global land cover every 5 years from 1985 and every year from 2000.
- CyberChef – from GCHQ (yes, the UK signals intelligence agency) comes this very useful web-based tool for converting data formats, extracting data, doing basic image processing, extracting files and more. You can even build multiple tools into a ‘recipe’ or pipeline.
- envreport – a simple Python script to report lots of information about the Python environment in a diffable form – I can see this being very useful
- Antarctic glaciers named after satellites – seven Antarctic glaciers have been named after satellites, reflecting (ha!) the importance of satellites in monitoring Antarctica
- MoMAColors and MetBrewer – these are color palettes derived from artwork at the Museum of Modern Art and the Metropolitan Museum of Art in New York. There are some beautiful sets of colours in here (see below) which I want to use in some future data visualisations.
- geospatial-cli – a list of geospatial command-line tools. Lots of them I knew, but there were some gems I hadn’t come across before here too.
- map-gl-tools – a nice Javascript package to add a load of syntactic sugar to the MapLibreJS library. I’ve recently started using this library and found things quite clunky (coming from a Leaflet background) so this really helped
- CoolWalks – a nice research paper looking at creating walking routes on the shaded side of the street for various cities
- Writing efficient code for GeoPandas and Shapely in 2023 – a very useful notebook showing how to do things efficiently in GeoPandas in 2023. There are a load of old ways of doing things which are no longer the most efficient!
- Inside PostGIS: Calculating Distance – a post explaining how PostGIS’s distance calculation algorithms work
- quackosm – a Python and CLI tool for reading OpenStreetMap data using DuckDB – either for future analysis in DuckDB or for export to GeoParquet
- Comparing odc.stac.load and stackstac for raster composite workflow – fairly self-explanatory, but it’s interesting to see the differences between two widely-used tools in the STAC ecosystem
- Towards Flexible Data Schemas – this article shows how the flexibility of data schemes in specifications like STAC really help their adoption and use for diverse purposes
Luke Plant: Keeping things in sync: derive vs test
An extremely common problem in programming is that multiple parts of a program need to be kept in sync – they need to do exactly the same thing or behave in a consistent way. It is in response to this problem that we have mantras like “DRY” (Don’t Repeat Yourself), or, as I prefer it, OAOO, “Each and every declaration of behaviour should appear Once And Only Once”.
For both of these mantras, if you are faced with possible duplication of any kind, the answer is simply “just say no”. However, since programming mantras are to be understood as proverbs, not absolute laws, there are times that obeying this mantra can hurt more than it helps, so in this post I’m going to discuss other approaches.
Most of what I say is fairly language agnostic I think, but I’ve got specific tips for Python and web development.
Contents
To step back for a second, the essential problem that we are addressing here is that if making a change to a certain behaviour requires changing more than one place in the code, we have the risk than one will be forgotten. This results in bugs, which can be of various degrees of seriousness depending on the code in question.
To pick a concrete example, suppose we have a rule that says that items in a deleted folder get stored for 30 days, then expunged. We’re going to need some code that does the actual expunging after 30 days, but we’re also going to need to tell the user about the limit somewhere in the user interface. “Once And Only Once” says that the 30 days limit needs to be defined in a single place somewhere, and then reused.
There is a second kind of motivating example, which I think often crops up when people quote “Don’t Repeat Yourself”, and it’s really about avoiding tedious things from a developer perspective. Suppose you need to add an item to a menu, and you find out that first you’ve got to edit the MENU_ITEMS file to add an entry, then you’ve got to edit the MAIN_MENU constant to refer to the new entry, then you’ve got to define a keyboard shortcut in the MENU_SHORTCUTS file, then a menu icon somewhere else etc. All of these different places are in some way repeating things about how menus work. I think this is less important in general, but it is certainly life-draining as a developer if code is structured in this way, especially if it is difficult to discover or remember all the things that have to be done.
The ideal solution: deriveOAOO and DRY say that we aim to have a single place that defines the rule or logic, and any other place should be derived from this.
Regarding the simple example of a time limit displayed in the UI and used in the backend, this might be as simple as defining a constant e.g. in Python:
from datetime import timedelta EXPUNGE_TIME_LIMIT = timedelta(days=30)We then import and use this constant in both our UI and backend.
An important part of this approach is that the “deriving” process should be entirely automatic, not something that you can forget to do. In the case of a Python import statement, that is very easy to achieve, and relatively hard to get wrong – if you change the constant where it is defined in one module, any other code that uses it will pick up the change the next time the Python process is restarted.
Alternative solution: testBy “test”, I mean ideally an automated test, but manual tests may also work if they are properly scripted. The idea is that you write a test that checks the behaviour or code is synced. Often, it may be that for one (or more) instances that need the behaviour will define it using some constant as above, let’s say the “backend” code. Then, for one instance, e.g. the UI, you would hard code “30 days” without using the constant, but have a test that uses the backend constant to build a string, and checks the UI for that string.
ExamplesIn the example above, it might be hard to see why you want to use the fundamentally less reliable, less automatic method I’m suggesting. So I now have to show some motivating examples where the “derive” method ends up losing to the cruder, simpler alternative of “test”.
Example 1 - external data sourcesMy first example comes from the project I’m currently working on, which involves creating CAM files from input data. Most of the logic for that is driven using code, but there are some dimensions that are specified as data tables by the engineers of the physical product.
These data tables look something like below. The details here aren’t important, and I’ve changed them – it’s enough to know that we’ve are creating some physical “widgets” which need to have specific dimensions specified:
Widgets have length 150mm unless specified below
Widget id
Location
Length (mm)
A
start
100
A
end
120
F
start
105
F
end
110
These tables are supplied at design-time rather than run-time i.e. they are bundled with the software and can’t be changed after the code is shipped. But it is still convenient to read them in automatically rather than simply duplicate the tables in my code by some process. So, for the body of the table, that’s exactly what my code does on startup – it reads the bundled XLSX/CSV files.
So we are obeying “derive” here — there is a single, canonical source of data, and anywhere that needs it derives it by an entirely automatic process.
But what about that “150mm” default value specified in the header of that table?
It would be possible to “derive” it by having a parser. Writing such a parser is not hard to do – for this kind of thing in Python I like parsy, and it is as simple as:
import parsy as P default_length_parser = ( P.string("Widgets have length ") >> P.regex(r"\d+").map(int) << P.string("mm unless specified below") )In fact I do something similar in some cases. But in reality, the “parser” here is pretty simplistic – it can’t deal with the real variety of English text that might be put into the sentence, and to claim I’m “deriving” it from the table is a bit of a stretch – I’m just matching a specific, known pattern. In addition, it’s probably not the case that any value for the default length would work – most likely if it was 10 times larger, there would be some other problem, and I’d want to do some manual checking.
So, let’s admit that we are really just checking for something expected, using the “test” approach. You can still define a constant that you use in most of the code:
DEFAULT_LENGTH_MM = 150And then you test it is what you expect when you load the data file:
assert worksheets[0].cell(1, 1).value == f"Widgets have length {DEFAULT_LENGTH_MM}mm unless specified below"So, I’ve achieved my aim: a guard against the original problem of having multiple sources of information that could potentially be out of sync. But I’ve done it using a simple test, rather than a more complex and fragile “derive” that wouldn’t have worked well anyway.
By the way, for this specific project – we’re looking for another contract developer! It’s a very worthwhile project, and one I’m really enjoying – a small flexible team, with plenty of problem solving and fun challenges, so if you’re a talented developer and interested give me a shout.
Example 2 - defining UI behaviour for domain objectsSuppose you have a database that stores information about some kind of entity, like customers say, and you have different types of customer, represented using an enum of some kind, perhaps a string enum like this in Python:
from enum import StrEnum class CustomerType(StrEnum): ENTERPRISE = "Enterprise" SMALL_FRY = "Small fry" # Let’s be honest! Try not to let the name leak… LEGACY = "Legacy"We need to a way edit the different customer types, and they are sufficiently different that we want quite different interfaces. So, we might have a dictionary mapping the customer type to a function or class that defines the UI. If this were a Django project, it might be a different Form class for each type:
CUSTOMER_EDIT_FORMS = { CustomerType.ENTERPRISE: EnterpriseCustomerForm, CustomerType.SMALL_FRY: SmallFryCustomerForm, CustomerType.LEGACY: LegacyCustomerForm, }Now, the DRY instinct kicks in and we notice that we now have two things we have to remember to keep in sync — any addition to the customer enum requires a corresponding addition to the UI definition dictionary. Maybe there are multiple dictionaries like this.
We could attempt to solve this by “deriving”, or some “correct by construction” mechanism that puts the creation of a new customer type all in one place.
For example, maybe we’ll have a base Customer class with get_edit_form_class() as an abstractmethod, which means it is required to be implemented. If I fail to implement it in a subclass, I can’t even construct an instance of the new customer subclass – it will throw an error.
from abc import abstractmethod class Customer: @abstractmethod def get_edit_form_class(self): pass class EnterpriseCustomer(Customer): def get_edit_form_class(self): return EnterpriseCustomerForm class LegacyCustomer(Customer): ... # etc.I still need my enum value, or at least a list of valid values that I can use for my database field. Maybe I could derive that automatically by looking at all the sublclasses?
CUSTOMER_TYPES = [ cls.__name__.upper().replace("CUSTOMER", "") for cls in Customer.__subclasses__() ]Or maybe an __init_subclass__ trick, and I can perhaps also set up the various mappings I’ll need that way?
It’s at this point you should stop and think. In addition to requiring you to mix UI concerns into the Customer class definitions, it’s getting complex and magical.
The alternative I’m suggesting is this: require manual syncing of the two parts of the code base, but add a test to ensure that you did it. All you need is a few lines after your CUSTOMER_EDIT_FORMS definition:
CUSTOMER_EDIT_FORMS = { # etc as before } for c_type in CustomerType: assert ( c_type in CUSTOMER_EDIT_FORMS ), f"You've defined a new customer type {c_type}, you need to add an entry in CUSTOMER_EDIT_FORMS"You could do this as a more traditional unit test in a separate file, but for simple things like this, I think an assertion right next to the code works much better. It really helps local reasoning to be able to look and immediately conclude “yes, I can see that this dictionary must be exhaustive because the assertion tells me so.” Plus you get really early failure – as soon as you import the code.
This kind of thing crops up a lot – if you create a class here, you’ve got to create another one over there, or add a dictionary entry etc. In these cases, I’m finding simple tests and assertions have a ton of advantages when compared to clever architectural contortions (or other things like advanced static typing gymnastics):
they are massively simpler to create and understand.
you can write your own error message in the assertion. If you make a habit of using really clear error messages, like the one above, your code base will literally tell you how to maintain it.
you can easily add things like exceptions. “Every Customer type needs an edit UI defined, except Legacy because they are read only” is an easy, small change to the above.
This contrasts with cleverer mechanisms, which might require relaxing other constraints to the point where you defeat the whole point of the mechanism, or create more difficulties for yourself.
the rule about how the code works is very explicit, rather than implicit in some complicated code structure, and typically needs no comment other than what you write in the assertion message.
you express and enforce the rule, with any complexities it gains, in just one place. Ironically, if you try to enforce this kind of constraint using type systems or hierarchies to eliminate repetition or the need for any kind of code syncing, you may find that when you come to change the constraint it actually requires touching far more places.
temporarily silencing the assertion while developing is easy and doesn’t have far reaching consequences.
Of course, there are many times when being able to automatically derive things at the code level, including some complex relationships between parts of the code, can be a win, and it’s the kind of thing you can do in Python with its many powerful techniques.
But my point is that you should remember the alternative: “synchronise manually, and have a test to check you did it.” Being able to add any kind of executable code at module level – the same level as class/function/constant definitions – is a Python super-power that you should use.
Example 3 - external polymorphism and static typingA variant of the above problem is when, instead of an enum defining different types, I’ve got a set of classes that all need some behaviour defined.
Often we just use polymorphism where a base class defines the methods or interfaces needed and sub-classes provide the implementation. However, as in the previous case, this can involve mixing concerns e.g. user interface code, possibly of several types, is mixed up with the base domain objects. It also imposes constraints on class hierarchies.
Recently for these kind of cases, I’m more likely to prefer external polymorphism to avoid these problems. To give an example, in my current project I’m using the Command pattern or plan-execute pattern extensively, and it involves manipulating CAM objects using a series of command objects that look something like this:
@dataclass class DeleteFeature: feature_name: str @dataclass class SetParameter: param_name: str value: float @dataclass class SetTextSegment: text_name: str segment: int value: str Command: TypeAlias = DeleteFeature | SetParameter | SetTextSegmentNote that none of them share a base class, but I do have a union type that gives me the complete set.
It’s much more convenient to define the behaviour associated with these separately from these definitions, and so I have multiple other places that deal with Command, such as the place that executes these commands and several others. One example that requires very little code to show is where I’m generating user-presentable tables that show groups of commands. I convert each of these Command objects into key-value pairs that are used for column headings and values:
def get_command_display(command: Command) -> tuple[str, str | float | bool]: match command: case DeleteFeature(feature_name=feature_name): return (f"Delete {feature_name}", True) case SetParameter(param_name=param_name, value=value): return (param_name, value) case SetTextSegment(text_name=text_name, segment=segment, value=value): return (f"{text_name}[{segment}]", value)This is giving me a similar problem to the one I had before I had before: if I add a new Command, I have to remember to add the new branch to get_command_display.
I could split out get_command_display into a dictionary of functions, and apply the same technique as in the previous example, but it’s more work, a less natural fit for the problem and potentially less flexible.
Instead, all I need to do is add exhaustiveness checking with one more branch:
match command: ... # etc case _: assert_never(command)Now, pyright will check that I didn’t forget to add branches here for any new Command. The error message is not controllable, in contrast to hand-written asserts, but it is clear enough.
The theme here is that additions in one part of the code require synchronised additions in other parts of the code, rather than being automatically correct “by construction”, but you have something that tests you didn’t forget.
Example 4 - generated codeIn web development, ensuring consistent design and keeping different things in sync is a significant problem. There are many approaches, but let’s start with the simple case of using a single CSS stylesheet to define all the styles.
We may want a bunch of components to have a consistent border colour, and a first attempt might look like this (ignoring the many issues of naming conventions here):
.card-component, .bordered-heading { border-color: #800; }This often becomes impractical when we want to organise by component, rather than by property, which introduces duplication:
.card-component { border-color: #800; } /* somewhere far away … */ .bordered-heading { border-color: #800; }Thankfully, CSS has variables, so the first application of “derive” is straightforward – we define a variable which we can use in multiple places:
:root { --primary-border-color: #800; } /* elsewhere */ .bordered-heading { border-bottom: 1px solid var(--primary-border-color); }However, as the project grows, we may find that we want to use the same variables in different contexts where CSS isn’t applicable. So the next step at this point is typically to move to Design Tokens.
Practically speaking, this might mean that we now have our variables defined in a separate JSON file. Maybe something like this (using a W3C draft spec):
{ "primary-border-color": { "$value": "#800000", "$type": "color" } "primary-hightlight-color": { "$value": "#FBC100", "$type": "color" } }From this, we can automatically generate CSS fragments that contain the same variables quite easily – for simple cases, this isn’t more than a 50 line Python script.
However, we’ve got some choices when it comes to how we put everything together. I think the general assumption in web development world is that a fully automatic “derive” is the only acceptable answer. This typically means you have to put your own CSS in a separate file, and then you have a build tool that watches for changes, and compiles your CSS plus the generated CSS into the final output that gets sent to the browser.
In addition, once you’ve bought into these kind of tools you’ll find they want to do extensive changes to the output, and define more and more extensions to the underlying languages. For example, postcss-design-tokens wants you to write things like:
.foo { color: design-token('color.background.primary'); }And instead of using CSS variables in the output, it puts the value of the token right in to every place in your code that uses it.
This approach has various problems, in particular that you become more and more dependent on the build process, and the output gets further from your input. You can no longer use the Dev Tools built in to your browser to do editing – the flow of using Dev Tools to experiment with changing a single spacing or colour CSS variable for global changes is broken, you need your build tool. And you can’t easily copy changes from Dev Tools back into the source, because of the transformation step, and debugging can be similarly difficult. And then, you’ll probably want special IDE support for the special CSS extensions, rather than being able to lean on your editor simply understanding CSS, and any other tools that want to look at your CSS now need support etc.
It’s also a lot of extra infrastructure and complexity to solve this one problem, especially when our design tokens JSON file is probably not going to change that often, or is going to have long periods of high stability. There are good reasons to want to be essentially build free. The current state of the art in this space is that to get your build tool to compile your CSS you add import './styles.css' in your entry point Javascript file! What if I don’t even have a Javascript file? I think I understand how this sort of thing came about, but don’t try to tell me that it’s anything less than completely bonkers.
Do we have an alternative to the fully automatic derive?
Using the “test” approach, we do. We can even stick with our single CSS file – we just write it like this:
/* DESIGN TOKENS START */ /* auto-created block - do not edit */ :root { --primary-border-color: #800000; --primary-highlight-color: #FBC100; } /* DESIGN TOKENS END */ /* the rest of our CSS here */The contents of this block will be almost certainly auto-generated. We won’t have a process that fully automatically updates it, however, because this is the same file where we are putting our custom CSS, and we don’t want any possibility of lost work due to the file being overwritten as we are editing it.
On the other hand we don’t want things to get out of sync, so we’ll add a test that checks whether the current styles.css contains the block of design tokens that we expect to be there, based on the JSON. For actually updating the block, we’ll need some kind of manual step – maybe a script that can find and update the DESIGN TOKEN START block, maybe cog – which is a perfect little tool for this use case — or we could just copy-paste.
There are also slightly simpler solutions in this case, like using a CSS import if you don’t mind having multiple CSS files.
ConclusionFor all the examples above, the solutions I’ve presented might not work perfectly for your context. You might also want to draw the line at different place to me. But my main point is that we don’t have to go all the way with a fully automatic derive solution to eliminate any manual syncing. Having some manual work plus a mechanism to test that two things are in sync is a perfectly legitimate solution, and it can avoid some of the large costs that come with structuring everything around “derive”.
Python Insider: Python 3.13.0 beta 3 released
I'm pleased to announce the release of Python 3.13 beta 3.
https://www.python.org/downloads/release/python-3130b3/
This is a beta preview of Python 3.13
Python 3.13 is still in development. This release, 3.13.0b3, is the third of four beta release previews of 3.13.
Beta release previews are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release.
We strongly encourage maintainers of third-party Python projects to test with 3.13 during the beta phase and report issues found to the Python bug tracker as soon as possible. While the release is planned to be feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (Tuesday 2024-07-30). Our goal is to have no ABI changes after beta 4 and as few code changes as possible after 3.13.0rc1, the first release candidate. To achieve that, it will be extremely important to get as much exposure for 3.13 as possible during the beta phase.
Please keep in mind that this is a preview release and its use is not recommended for production environments.
Major new features of the 3.13 series, compared to 3.12Some of the new major new features and changes in Python 3.13 are:
New features- A new and improved interactive interpreter , based on PyPy’s, featuring multi-line editing and color support, as well as colorized exception tracebacks .
- An experimental free-threaded build mode , which disables the Global Interpreter Lock, allowing threads to run more concurrently. The build mode is available as an experimental feature in the Windows and macOS installers as well.
- A preliminary, experimental JIT , providing the ground work for significant performance improvements.
- The (cyclic) garbage collector is now incremental , which should mean shorter pauses for collection in programs with a lot of objects.
- A modified version of mimalloc is now included, optional but enabled by default if supported by the platform, and required for the free-threaded build mode.
- Docstrings now have their leading indentation stripped, reducing memory use and the size of .pyc files. (Most tools handling docstrings already strip leading indentation.)
- The dbm module has a new dbm.sqlite3 backend that is used by default when creating new files.
- The minimum supported macOS version was changed from 10.9 to 10.13 (High Sierra). Older macOS versions will not be supported going forward.
- Support for type defaults in type parameters.
- A new type narrowing annotation , typing.TypeIs.
- A new annotation for read-only items in TypeDicts .
- PEP 594 (Removing dead batteries from the standard library) scheduled removals of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3.
- Many other removals of deprecated classes, functions and methods in various standard library modules.
- C API removals and deprecations. (Some removals present in alpha 1 were reverted in alpha 2, as the removals were deemed too disruptive at this time.)
- New deprecations, most of which are scheduled for removal from Python 3.15 or 3.16.
(Hey, fellow core developer, if a feature you find important is missing from this list, let Thomas know.)
For more details on the changes to Python 3.13, see What’s new in Python 3.13 . The next pre-release of Python 3.13 will be 3.13.0b4, currently scheduled for 2024-07-16.
More resources- Online Documentation
- PEP 719 , 3.13 Release Schedule
- Report bugs at Issues · python/cpython · GitHub .
- Help fund Python directly (or via GitHub Sponsors), and support the Python community.
Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.
Your release team,
Thomas Wouters
Łukasz Langa
Ned Deily
Steve Dower
Python Software Foundation: Announcing the PSF Board Candidates for 2024!
What an exciting list! Please take a look at who is running for the PSF Board this year on the PSF Board Election 2024 Nominees page. This year there are 3 seats open on the PSF board. You can see who is currently on the board on the PSF Officers & Directors page. (Débora Azevedo, Kwon-Han Bae, and Tania Allard are at the end of their current terms.)
- Nominations open: Tuesday, June 11th, 2:00 pm UTC
- Nomination cut-off: Tuesday, June 25th, 2:00 pm UTC
- Voter application/affirmation cut-off date: Tuesday, June 25th, 2:00 pm UTC
- Announce candidates: Thursday, June 27th
- Voting start date: Tuesday, July 2nd, 2:00 pm UTC
- Voting end date: Tuesday, July 16th, 2:00 pm UTC
Not sure what UTC is for you locally? Check using this timezone converter!
VotingVoting opens on Tuesday, July 2nd at 2:00 pm UTC, through Tuesday, July 16th, 2024 2:00 pm UTC. Check the Elections page to see how much time you have left to vote.
If you are a voting member of the PSF that affirmed your intention to participate in this year’s election, you will receive an email from “OpaVote Voting Link <noreply@opavote.com>” with your ballot, the subject line will read “Python Software Foundation Board of Directors Election 2024”. If you haven’t seen your ballot by Wednesday, please first check your spam folder for a message from “noreply@opavote.com”. If you don’t see anything get in touch by emailing psf-elections@python.org so we can look into your account and make sure we have the most up-to-date email for you.
If you have questions about your membership status or the election, please email psf-elections@python.org. You are welcome to join the discussion about the PSF Board election on the PSF Discuss forum.
Mark Dufour: Shed Skin restricted-Python-to-C++ compiler 0.9.9
I have just released version 0.9.9 of Shed Skin, a restricted-Python-to-C++ compiler. It comes with many changes under the hood to improve the code base. For example, Shakeeb has started adding type annotations to Shed Skin itself, whereas I have fixed many C++ compiler warnings. We have also replaced the old CPython-based dict and set implementations with STL unordered_map and unordered_set. While this may cost some performance for especially small toy programs, it really helps with maintainability.
There was also a type inference improvement, which hasn't happened in a while since it wasn't needed. It was however needed to add a cool new example called "pycsg", which shows how one can perform CSG (constructive solid geometry) using binary space partitioning (BSP). After some investigation I decided to make the type inference slightly less "optimistic", which should make it scale a lot better, at the cost of being slower overall. In the end this new example runs about 15 times faster after compilation on my system (not 15%.. 15 times!).
Most notably on the outside, Shed Skin now comes with --floatXX/--intXXX options, which allow you to specify the desired float/int precision. Unfortunately under windows, --int128 does not yet work, since C++ has not yet standardized 128-bit integers.. The relatively new othello2 and collatz examples now require --int64 and --int128, respectively, to function properly.
For the full list of changes, please see the release notes.
I would like to take this opportunity to (once again) invite others to help out in Shed Skin development. There is always enough to do, also low-hanging fruit, on both the Python and the C++ side of things. I at least think it is rather fun to work on.. :) Let me know if you'd like to contribute but aren't sure what you could bring to the table.
Of course just sending in feedback or new example programs (especially if they fail!) can be very motivating as well to keep improving things.
Django Weblog: Django 5.1 beta 1 released
Django 5.1 beta 1 is now available. It represents the second stage in the 5.1 release cycle and is an opportunity for you to try out the changes coming in Django 5.1.
Django 5.1 brings a kaleidoscope of improvements which you can read about in the in-development 5.1 release notes.
Only bugs in new features and regressions from earlier versions of Django will be fixed between now and the 5.1 final release. Translations will be updated following the "string freeze", which occurs when the release candidate is issued. The current release schedule calls for a release candidate in a month from now, and a final release to follow about two weeks after that, scheduled for August 7th.
Early and frequent testing from the community will help minimize the number of bugs in the release. Updates on the release schedule are available on the Django forum.
As with all alpha and beta packages, this is not for production use. But if you'd like to take some of the new features for a spin, or to help find and fix bugs (which should be reported to the issue tracker), you can grab a copy of the beta package from our downloads page or on PyPI.
The PGP key ID used for this release is Natalia Bidart: 2EE82A8D9470983E.
Real Python: Understanding the Python Mock Object Library
Python’s unittest.mock library is a tool that helps you create mock objects to simulate complex logic and unpredictable dependencies. This helps you write valuable tests that verify your application logic is correct, reliable, and efficient.
By the end of this tutorial, you’ll be able to:
- Create Python mock objects using Mock
- Assert that you’re using objects as you intended
- Inspect usage data stored on your Python mocks
- Configure certain aspects of your Python mock objects
- Substitute your mocks for real objects using patch()
- Avoid common problems inherent in Python mocking
You’ll begin by learning about what mocking is and how you can use it to improve your tests.
Get Your Code: Click here to download the free sample code that you’ll use to learn about Python’s mock object library.
Take the Quiz: Test your knowledge with our interactive “Understanding the Python Mock Object Library” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
Understanding the Python Mock Object LibraryIn this quiz, you'll test your understanding of Python's unittest.mock library. With this knowledge, you'll be able to write robust tests, create mock objects, and ensure your code is reliable and efficient.
What Is Mocking?A mock object substitutes and imitates a real object within a testing environment. Using mock objects is a versatile and powerful way to improve the quality of your tests. This is because by using Python mock objects, you can control your code’s behavior during testing.
For example, if your code makes HTTP requests to external services, then your tests execute predictably only so far as the services are behaving as you expected. Sometimes, a temporary change in the behavior of these external services can cause intermittent failures within your test suite.
Because of this, it would be better for you to test your code in a controlled environment. Replacing the actual request with a mock object would allow you to simulate external service outages and successful responses in a predictable way.
Sometimes, it’s difficult to test certain areas of your codebase. Such areas include except blocks and if statements that are hard to satisfy. Using Python mock objects can help you control the execution path of your code to reach these areas and improve your code coverage.
Another reason to use mock objects is to better understand how you’re using their real counterparts in your code. A Python mock object contains data about its usage that you can inspect, such as:
- If you called a method
- How you called the method
- How often you called the method
Understanding what a mock object does is the first step to learning how to use one. Next, you’ll explore the Python mock object library to see how to use Python mock objects.
The Python Mock LibraryPython’s built-in mock object library is unittest.mock. It provides an easy way to introduce mocks into your tests.
Note: The standard library includes unittest.mock starting from Python 3.3 and in all newer versions. If you’re using an older version of Python, then you’ll need to install the official backport of the library.
To do so, install mock from the Python Package Index (PyPI) using pip:
Shell $ python -m pip install mock Copied!You may want to create and activate a virtual environment before installing the package.
unittest.mock provides a class called Mock, which you’ll use to imitate real objects in your codebase. Mock, along with its subclasses, offers incredible flexibility and insightful data that will meet most of your Python mocking needs.
The library also provides a function called patch(), which replaces the real objects in your code with Mock instances. You can use patch() as either a decorator or a context manager, giving you control over the scope in which the object will be mocked. Once the designated scope exits, patch() will clean up your code by replacing the mocked objects with their original counterparts.
Finally, unittest.mock provides solutions for some of the issues inherent in mocking objects, which you’ll explore later in this tutorial.
Now that you have a better understanding of what mocking is and the library you’ll be using, it’s time to dive in and explore the features and functionalities unittest.mock has to offer.
The Mock Objectunittest.mock offers a base class for mocking objects called Mock. The use cases for Mock are practically limitless because Mock is so flexible.
Begin by instantiating a new Mock instance:
Python >>> from unittest.mock import Mock >>> mock = Mock() >>> mock <Mock id='4561344720'> Copied! Read the full article at https://realpython.com/python-mock-library/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Armin Ronacher: What is Self Hosted? What is a Stack?
My colleague Ben Vingar wrote a tool called Counterscale which I would describe as “deploy your own analytics”. Except there is a catch: it needs Cloudflare to run. Is it really self hosted if your only way to deploy it is some proprietary cloud vendor?
What's a Stack?Many years ago we talked about software stacks. A common one happened to be “LAMP”. Short for: Linux, Apache, MySQL and typically PHP, though Python and Perl were choices for the P just as well. LAMP lends itself very well for self hosting because all of it is Open Source software you can run and operate yourself free of charge. There was however also a second stack which was not entirely unpopular: “WAMP“ (The W meaning Microsoft Windows). You would not necessarily run it yourself if you had a choice, but I deployed more than one of these. Why? Because some SMEs were already running Windows. If you wrote some software in PHP, having people run the software on their already existing Windows servers was preferable to also running some Linux thing they did not know how to operate.
What makes LAMP, WAMP and whatever work are a few basic technological choices. Originally one of those abstractions was a protocol called CGI which allowed you to marry a programming language to the web server. Later also things like FastCGI appeared to deal with some of the performance challenges that CGI brought and there were also attempts to move PHP right into the web server as embedded language with mod_php. For the database the abstraction in many cases was a dialect of SQL. I built a tool a long time ago that a company ended up running on Microsoft's SQL server with rather minimal changes. So in some sense what made this work was that one was targeting some form of abstraction.
What's Self Hosted?Counterscale targets something that the open source ecosystem does not really have abstracted today: an analytics engine and some serverless runtime. What was CGI and SQL in case of Counterscale is some serverless runtime environment and a column store. All these things do exist in the Open Source ecosystem. All the pieces are there to build your own serverless runtime and all the things are there to build an analytics store on top of ClickHouse, DuckDB or similar databases and Kafka. But we did not agree on protocols and we definitely did not really have that stuff today in a neatly packaged and reusable thing.
Now of course you can build software that runs entirely on Open Source software. In case of Counterscale you don't even have to look very far: Plausible exists. It's also Open Source, it's also an analytics tool, but rather than being like a “CGI script” in spirit, it's a pretty heavy thing. You gotta run docker containers, run a rather beefy ClickHouse installation, I believe it needs Kafka etc. Running Plausible yourself is definitely not neatly as easy as setting up Counterscale. You do however, have the benefit of not relying on Cloudflare.
Level up the ProtocolsSo what does that leave us with? I'm not sure but I'm starting to think that the web needs new primitives. We now run some things commonly but the abstractions over them are not ideal. So people target (proprietary) systems directly. The modern web needs the CGI type protocols for queues, for authentication, for columns stores, for caches etc. Why does it need that? I think it needs it to lower the cost of building small scale open source software.
The reason it's so easy and appealing to build something like Counterscale directly against Cloudflare or similar services is that they give you higher level abstractions than you would find otherwise. You don't have to think about scaling workers, you don't have to think about scaling databases. The downside of course is that it locks you against that platform.
But what would be necessary to have your “own Cloudflare” thing you can run once and then run all your cool mini CGI like scripts above? We miss some necessary protocols. Yet building these protocols is tricky because you target often the least common denominator. Technology also here is hardly the problem. Don't need any new innovative technology here, but you need the social contract and the mindset. Those are hard things, they require dedication and marketing. I have not yet seen that, but I'm somewhat confident that we might see it.
We probably want these protocols and systems built on top of it because it makes a lot of things easier. Sometimes of the cost of doing something drops low enough, it enables a whole new range of things to exist.
Many times when you start building abstractions over these things, you simplify. Even CGI was an incredibly high level abstraction over HTTP if you think about it. CGI in many ways is the original serverless. It abstracts over both HTTP and how a process spawns and its lifecycle. Serverless is bringing back a bit of that, but so far not in a way where this is actually portable between different clouds.
Abstract over Great IdeasIf you have ever chucked up an OG CGI app you might remember the magic. You write a small script, throw it into a specific folder and you are off to the races. No libraries, no complex stuff. CGI at its core was a great idea: make a web server dynamic via a super trivial protocol anyone can implement. There are more ideas like that. Submitting tasks to a worker queue is a great idea, batch writing a lot of data into a system is a great idea, kafka like topics are a great idea, caches are a great idea, so are SQL databases, column stores and much more.
Laravel Forge does a tiny bit of that I feel. Forge goes a bit in to that direction in the sense that it says quite clearly that some components are useful: databases, caches, SSL, crons etc. However it's ambition stops at the boundary of the Laravel ecosystem which is understandable.
Yet maybe over time we can see more of a “SaaS in a box” kind of experience. A thing you run, that you can plug your newfangled, serverless mini tools in, that can leverage auth and all the needs of a modern web application like queues, column stores, caches etc.
Pages
Recent Publications
- Managing Hidden Dependencies in OO Software: a Study Based on Open Source Projects
- Open Source Communities as Liminal Ecosystems
- Investigating developers' email discussions during decision-making in Python language evolution
- Developers, Quality Control and Download Volume in Open Source Software (OSS) Projects
More...FLOSS Project Planets
- Kushal Das: Multi-factor authentication in django
- The Drop Times: Brad Jones on Modernizing Drupal's Data Management with JSON Integration
- Real Python: The Real Python Podcast – Episode #214: Build Captivating Display Tables in Python With Great Tables
- Python Software Foundation: Notice of Python Software Foundation Bylaws change, effective 10 August 2024
- 1xINTERNET blog: How to optimise team efficiency by reducing "Work in Progress"
FLOSS Research
- Hailey Schoelkopf: Voices of the Open Source AI Definition
- OSI at the United Nations OSPOs for Good
- Better identifying conda packages with ClearlyDefined
- Cailean Osborne: voices of the Open Source AI Definition
- The Open Source Initiative joins CMU in launching Open Forum for AI: A human-centered approach to AI development