Planet Python
PyCon: PyCon US 2024: Call for Volunteers and Hatchery Registration now Open!
Looking to make a meaningful contribution to the Python community? Look no further than PyCon US 2024! Whether you're a seasoned Python pro or a newcomer to the community and looking to get involved, there's a volunteer opportunity that's perfect for you.
Sign-up for volunteer roles is done directly through the PyCon US website. This way, you can view and manage shifts you sign up for through your personal dashboard! You can read up on the different roles to volunteer for and how to sign up on the PyCon US website.
PyCon US is largely organized and run by volunteers. Every year, we ask to fill over 300 onsite volunteer hours to ensure everything runs smoothly at the event. And the best part? You don't need to commit a lot of time to make a difference– some shifts are as short as one hour long! You can sign up for as many or as few shifts as you’d like. Even a couple of hours of your time can go a long way in helping us create an amazing experience for attendees.
Keep in mind that you need to be registered for the event to sign up for a volunteer role.One important way to get involved is to sign up as a Session Chair or Session Runner. This is an excellent opportunity to meet and interact with speakers while helping to ensure that sessions run smoothly. And who knows, you might just learn something new along the way! You can sign up for these roles directly on the Talks schedule.
Volunteer your time at PyCon US 2024 and you’ll be part of a fantastic community that's passionate about Python programming and help us make this year's conference a huge success. Sign up today for the shifts that call to you and join the fun!
Hatchery ProgramFirst introduced in 2018, the Hatchery program offers the pathways for PyCon US attendees to introduce new tracks, activities, summits, demos, etc., at the conference—activities that all share and fulfill the Python Software Foundation’s mission within the PyCon US schedule.
Since its introduction, this program has “hatched” several new tracks that are now staples of our conference, including PyCon US Charlas, Mentored Sprints, and the Maintainer’s Summit. This year, we’ve received eight very compelling proposals. After careful consideration, we have selected four new programs, each of them unique and focus on different aspects of the Python community.
FlaskCon - Friday, May 17, 2024Join us in a mini conference dedicated to Flask, its community and ecosystem, as well as related web technologies. Meet maintainers and community members, learn about how to get involved, and join us during the sprint days to contribute. Submit your talk proposal today!
Organized by David Lord, Phil Jones, Adam Englander, David Carmichael, Abdur-Rahmaan Janhangeer
Community Organizers Summit - Saturday, May 18, 2024Do you organize a Conference, Meetup, User Group, Hackathon, or other community event in your area? Are you trying to start a group but don't know where to start? Whether you have 30 years of experience or are looking to create a new event, this summit is for you.
Join us for a summit of Presentations, Panels, and Breakout Sessions about organizing community events.
Organized by Mason Egger, Kevin Horn, and Heather White
Sign-up is required. Register to secure your spot.
Humble Data - Saturday, May 18, 2024Are you eager to embark on a tech career but unsure where to start? Are you curious about data science? Taking the first steps in this area is hard, but you don’t have to do it alone. Join our workshop for complete beginners and get started in Python data science - even if you’ve never written a single line of code!
We invite those from underrepresented groups to apply to join us for a fun, supportive workshop that will give you the confidence to get started in this exciting area. You can expect plenty of exercises, as well as inspiring talks from those who were once in your shoes. You’ll cover the basics of programming in Python, as well as useful libraries and tools such as Jupyter notebooks, pandas, and Matplotlib.
In this hands-on workshop, you’ll work through a series of beginner-friendly materials at your own pace. You’ll work within small groups, each with an assigned mentor, who will be there to help you with any questions or whenever you get stuck. All you’ll need to bring is a laptop that can connect to the internet and a willingness to learn!
Organized by Cheuk Ting Ho and Jodie Burchell
Sign-up is required. Register to secure your spot.Documentation Summit - Sunday May 19, 2024
A full-day summit including talks and panel sessions inviting leaders in documentation to share their experience in how to make good documentation, discussion about documentation tools such as sphinx, mkdocs, themes etc, what are the common mistakes and how to avoid them. Accessibility of documentation is also an important topic so we will also cover talks or discussions regarding accessibility of documentation.
This summit is aimed at anyone who cares about or is involved in any aspect of open source documentation, such as, but not limited to, technical writers, developers, developer advocates, project maintainers and contributors, accessibility experts, documentation tooling developers, and documentation end-users.Organized by Cheuk Ting Ho
Sign-up is required. Register to secure your spot.Register Now
Registration for PyCon US is now open, and all of the Hatchery programs are included as part of your PyCon US registration (no additional cost). Some of the programs require advanced sign up, in which case walk-ins will only be accepted if space is available. Please check each Hatchery program carefully to determine whether a registration is required or not.
Head over to your PyCon US Dashboard to add any of the above Hatchery programs to your PyCon US registration. Don't worry, you can always change your mind and cancel later to open up the space for someone else!
Congratulations to all the accepted program organizers! Thank you for bringing forward your fresh ideas to PyCon US. We look forward to seeing you in Pittsburgh.Real Python: Python Deep Learning: PyTorch vs Tensorflow
PyTorch vs TensorFlow: What’s the difference? Both are open source Python libraries that use graphs to perform numerical computation on data. Both are used extensively in academic research and commercial code. Both are extended by a variety of APIs, cloud computing platforms, and model repositories.
If they’re so similar, then which one is best for your project?
In this video course, you’ll learn:
- What the differences are between PyTorch and TensorFlow
- What tools and resources are available for each
- How to choose the best option for your specific use case
You’ll start by taking a close look at both platforms, beginning with the slightly older TensorFlow, before exploring some considerations that can help you determine which choice is best for your project. Let’s get started!
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Anwesha Das: Opening up Ansible release to the community
Transparency, collaboration, inclusivity, and openness lay the foundation of the Open Source community. As the project&aposs maintainers, few of our tasks make the entry bar of contribution low, collaboration easy, and the governance model fair. Ansible Community Engineering Team always thrives on these purposes through our different endeavors.
Ansible has historically been released by Red Hat employees. We planned to open up the release to the community. And I was asked about that. My primary goal was releasing Ansible, which should be dull and possible for the community. This was my first time dealing with Github actions. There is still a lot to learn. But we are there now.
The Release Management working group started releasing the Ansible Community package using GitHub Actions workflow from Ansible version 9.3.0 . The recent 9.4.0 release has also been released following the same workflow.
Thank you Felix Fontein, Maxwell G, Sviatoslav Sydorenko and Toshio for helping out in shaping the workflow with you valuable feedback, doing the actual release and giving answers to my enumerable queries.
EuroPython: EuroPython 2024: Ticket sales now open! 🐍
Hey hey, everyone,
We are thrilled to announce that EuroPython is back and better than ever! ✨
EuroPython 2024 will be held 8-14 July at the Prague Congress Centre (PCC), Czech Republic. Details of how to participate remotely will be published soon.
The conference will follow the same structure as the previous editions:
- Two Workshop/Tutorial Days (8-9 July, Mon-Tue)
- Three Conference Days (10-12 July, Wed-Fri)
- Sprint Weekend (13-14 July, Sat-Sun)
Secure your spot at EuroPython 2024 by purchasing your tickets today. For more information and to grab your tickets, visit https://ep2024.europython.eu/tickets before they sell out!
Get your tickets fast before the late-bird prices kick in. 🏃
Looking forward to welcoming you to EuroPython 2024 in Prague! 🇨🇿
Cheers,
The EuroPython 2024 Organisers
Python Bytes: #377 A Dramatic Episode
Python Software Foundation: New Open Initiative for Cybersecurity Standards
The Python Software Foundation is pleased to announce our participation in co-starting a new Open Initiative for Cybersecurity Standards collaboration with the Apache Software Foundation, the Eclipse Foundation, other code-hosting open source foundations, SMEs, industry players, and researchers. This collaboration is focused on meeting the real challenges of cybersecurity in the open source ecosystem, and demonstrating full cooperation with and supporting the implementation of the European Union’s Cyber Resilience Act (CRA). With our combined efforts, we are optimistic that we will reach our goal of establishing common specifications for secure open source development based on existing open source best practices.
New regulations, such as those in the CRA, highlight the need for secure by design and strong supply chain security standards. The CRA will lead to standard requests from the Commission to the European Standards Organisations and we foresee requirements from the United States and other regions in the future. As open source foundations, we want to respond to these requests proactively by establishing common specifications for secure software development and meet the expectations of the newly defined term Open Source Steward.
Open source communities and foundations, including the Python community, have long been practicing and documenting secure software development processes. The starting points for creating common specifications around security are already there, thanks to millions of contributions to hundreds of open source projects. In the true spirit of open source, we plan to learn from, adapt, and build upon what already exists for the collective betterment of our greater software ecosystem.
The PSF’s Executive Director Deb Nicholson will attend and participate in the initial Open Initiative for Cybersecurity Standards meetings. Later on, various PSF staff members will join in relevant parts of the conversation to help guide the initiative alongside their peers. The PSF looks forward to more investment in cybersecurity best practices by Python and the industry overall.
This community-driven initiative will have a lasting impact on the future of cybersecurity and our shared open source communities. We welcome you to join this collaborative effort to develop secure open source development specifications. Participate by sharing your knowledge, input, and raising up existing community contributions. Sign up for the Open Initiative for Process Specifications mailing list to get involved and stay updated on this initiative. Check out the press release's from the Eclipse Foundation’s and the Apache Software Foundation for more information.
Hynek Schlawack: Python Project-Local Virtualenv Management Redux
One of my first TIL entries was about how you can imitate Node’s node_modules semantics in Python on UNIX-like operating systems. A lot has happened since then (to the better!) and it’s time for an update. direnv still rocks, though.
Luke Plant: Enforcing conventions in Django projects with introspection
Naming conventions can make a big difference to the maintenance issues in software projects. This post is about how we can use the great introspection capabilities in Python to help enforce naming conventions in Django projects.
Contents
Let’s start with an example problem and the naming convention we’re going to use to solve it. There are many other applications of the techniques here, but it helps to have something concrete.
The problem: DateTime and DateTimeField confusionOver several projects I’ve found that inconsistent or bad naming of DateField and DateTimeField fields can cause various problems.
First, poor naming means that you can confuse them for each other, and this can easily trip you up. In Python, datetime is a subclass of date, so if you use a field called created_date assuming it holds a date when it actually holds a datetime, it might be not obvious initially that you are mishandling the value, but you’ll often have subtle problems down the line.
Second, sometimes you have a field named like expired which is actually the timestamp of when the record expired, but it could easily be confused for a boolean field.
Third, not having a strong convention, or having multiple conventions, leads to unnecessary time wasted on decisions that could have been made once.
Finally, inconsistency in naming is just confusing and ugly for developers, and often for users further down the line, because names tend to leak.
Even if you do have an established convention, it’s possible for people not to know. It’s also very easy for people to change a field’s type between date and datetime without also changing the name. So merely having the convention is not enough, it needs to be enforced.
Note
If you want to change the name and type of a field (or any other atribute), and want the data to preserve data as much as possible, you usually need to do it in two stages or more depending on your needs, and always check the migrations created – otherwise Django’s migration framework will just see one field removed and a completely different one added, and generate migrations that will destroy your data.
For this specific example, the convention I quite like is:
field names should end with _at for timestamp fields that use DateTimeField, like expires_at or deleted_at.
field names should end with _on or _date for fields that use DateField, like issued_on or birth_date.
This is based on the English grammar rule that we use “on” for dates but “at” for times – “on the 25th March”, but “at 7:00 pm” – and conveniently it also needs very few letters and tends to read well in code. The _date suffix is also helpful in various contexts where _on seems very unnatural. You might want different conventions, of course.
To get our convention to be enforced with automated checks we need a few tools.
The tools IntrospectionIntrospection means the ability to use code to inspect code, and typically we’re talking about doing this when our code is already running, from within the same program and using the same programming language.
In Python, this starts from simple things like isintance() and type() to check the type of an object, to things like hasattr() to check for the presence of attributes and many other more advanced techniques, including the inspect module and many of the metaprogramming dunder methods.
Django app and model introspectionDjango is just Python, so you can use all normal Python introspection techniques. In addition, there is a formally documented and supported set of functions and methods for introspecting Django apps and models, such as the apps module and the Model _meta API.
Django checks frameworkThe third main tool we’re going to use in this solution is Django’s system checks framework, which allows us to run certain kinds of checks, at both “warning” and “error” level. This is the least important tool, and we could in fact switch it out for something else like a unit test.
The solutionIt’s easiest to present the code, and then discuss it:
from django.apps import apps from django.conf import settings from django.core.checks import Tags, Warning, register @register() def check_date_fields(app_configs, **kwargs): exceptions = [ # This field is provided by Django's AbstractBaseUser, we don't control it # and we’ll break things if we change it: "accounts.User.last_login", ] from django.db.models import DateField, DateTimeField errors = [] for field in get_first_party_fields(): field_name = field.name model = field.model if f"{model._meta.app_label}.{model.__name__}.{field_name}" in exceptions: continue # Order of checks here is important, because DateTimeField inherits from DateField if isinstance(field, DateTimeField): if not field_name.endswith("_at"): errors.append( Warning( f"{model.__name__}.{field_name} field expected to end with `_at`, " + "or be added to the exceptions in this check.", obj=field, id="conventions.E001", ) ) elif isinstance(field, DateField): if not (field_name.endswith("_date") or field_name.endswith("_on")): errors.append( Warning( f"{model.__name__}.{field_name} field expected to end with `_date` or `_on`, " + "or be added to the exceptions in this check.", obj=field, id="conventions.E002", ) ) return errors def get_first_party_fields(): for app_config in get_first_party_apps(): for model in app_config.get_models(): yield from model._meta.get_fields() def get_first_party_apps() -> list[AppConfig]: return [app_config for app_config in apps.get_app_configs() if is_first_party_app(app_config)] def is_first_party_app(app_config: AppConfig) -> bool: if app_config.module.__name__ in settings.FIRST_PARTY_APPS: return True app_config_class = app_config.__class__ if f"{app_config_class.__module__}.{app_config_class.__name__}" in settings.FIRST_PARTY_APPS: return True return FalseWe start here with some imports and registration, as documented in the “System checks” docs. You’ll need to place this code somewhere that will be loaded when your application is loaded.
Our checking function defines some allowed exceptions, because there are some things out of our control, or there might be other reasons. It also mentioned the exceptions mechanism in the warning message. You might want a different mechanism for exceptions here, but I think having some mechanism like this, and advertising its existence in the warnings, is often pretty important. Otherwise, you can end up with worse consequences when people just slavishly follow rules. Notice how in the exception list above I’ve given a comment detailing why the exception is there though – this helps to establish a precedent that exceptions should be justified, and the justification should be there in the code.
We then loop through all “first party” model fields, looking for DateTimeField and DateField instances. This is done using our get_first_party_fields() utility, which is defined in terms of get_first_party_apps(), which in turn depends on:
a custom setting FIRST_PARTY_APPS which I’ve created in my settings.py like this:
FIRST_PARTY_APPS = ["myapp", "myotherapp"] INSTALLED_APPS = [ "django.contrib.auth", "django.contrib.sessions", ] + FIRSTY_PARTY_APPS + [ ... ]You may have a different way of recognising your own apps.
The id values passed to Warning here are examples – you should change according to your needs. You might also choose to use Error instead of Warning.
OutputWhen you run manage.py check, you’ll then get output like:
System check identified some issues: WARNINGS: myapp.MyModel.created: (conventions.E001) MyModel.created field expected to end with `_at`, or be added to the exceptions in this check. System check identified 1 issue (0 silenced).As mentioned, you might instead want to run this kind of check as a unit test.
ConclusionThere are many variations on this technique that can be used to great effect in Django or other Python projects. Very often you will be able to play around with a REPL to do the introspection you need.
Where it is possible, I find doing this far more effective than attempting to document things and relying on people reading and remembering those docs. Every time I’m tripped up by bad names, or when good names or a strong convention could have helped me, I try to think about how I could push people towards a good convention automatically – while also giving a thought to unintended bad consequences of doing that prematurely or too forcefully.
Zero to Mastery: Python Monthly Newsletter 💻🐍
Tryton News: Newsletter April 2024
During the last month we focused on fixing bugs, improving the behaviour of things, speeding-up performance issues and adding new features for you.
Changes for the User Sales, Purchases and ProjectsWhen processing an exception on an order, the user can ignore the exception and so no more related lines/documents will be re-created. But in case of a mistake it was not possible to cancel the ignore. Now we allow the Sale and Purchase administrator group to edit the list of ignored- lines to be able to remove mistakes. After changes to the list of ignored lines the user needs to manually reprocess the order, using the Process button, to restore it to a coherent state.
Accounting, Invoicing and PaymentsAccount users are now allowed to delete draft account moves.
Stock, Production and ShipmentsWhen creating a stock forecast the warehouse is now filled in automatically.
Now the scheduled task maintains a global order of assignations for shipments and productions. A global order is important because assignations are competing with each other to get the products first.
User InterfaceWe now hide the traceback from an error behind an expander widget, as it may scare some users and it is not helpful for most of them.
System Data and ConfigurationEmployees are now activated based on the start and end date of their employment.
New ModulesThe new stock_product_location_place module allows a specific place to be defined where goods are stored in their location. You can refer to its documentation for more details.
New DocumentationWe reworked parts of the Tryton documentation.
How to enter in an opening balance.
We changed our documentation hub from readthedocs to self hosting.
New ReleasesWe released bug fixes for the currently maintained long term support series
7.0 and 6.0, and for the penultimate series 6.8.
We now make cron and workers exit silently on a keyboard interrupt.
We also introduced a switch on trytond-admin to be able to delay the creation of indexes. This is because the index creation can take a long time to complete when updating modules on big databases. Using this switch the database schema can be quickly created, but will be without the performance gain from the new indexes, which are not available yet. Another run at a more appropriate time without the switch can then be used to create the indexes.
For history records we now display the date time on access errors.
Changes for Implementers and DevelopersWe now use dot notation and binary operators when converting PYSON to a string when it is to be displayed to the user.
1 post - 1 participant
Go Deh: Finding a sub-list within a list, in Python
Existing?
As part of a larger project, I thought I might need to search for a sub-list within a given list, and because I am lazy i did a quick google and did not like the answers I found.I started with the thought that the best algorithm for me would be to start searching from the index of the first item in the sublist and so on, but none of the googled answers used list.index.
I decided then to create my own
My versionWell I want to use list.index. If the item is not in the list then it raises an error, so I'll need a try-except block too.
I look for successive first item from the sub-list in the list and if found, accumulate the index in the answer and move on to search for the next match.
It seemed easy to add flags to:
- Stop after finding a first index of the sub-list in the list.
- Allow overlapping matches or not. [1,0,1] is found twice in [1,0,1,0,1] at indices 0 and 2, but only once if overlapping is not allowed
"""Find instance of a sub-list in a list"""
def index_sublist(lst: list[Any], sublst: list[Any], only_first: bool=False, non_overlapping=False, ) -> list[int]: "Find instance of a (non-empty), sub-list in a list" if not sublst: raise ValueError("Empty sub-list") if not lst: return [] first, ln = sublst[0], len(sublst) ans, i = [], 0 while True: try: i = lst.index(first, i) except ValueError: break if lst[i: i+ln] == sublst: ans.append(i) if only_first: break i += ln if non_overlapping else 1 return ans
#%%def test(): assert index_sublist([], [1], only_first=False) == [] assert index_sublist([1], [1], only_first=False) == [0] assert index_sublist([1,0,1], [1], only_first=False) == [0, 2] assert index_sublist([2,1,0,1], [1], only_first=True) == [1] assert index_sublist([2,1,0,1], [1, 3], only_first=False) == [] assert index_sublist([1,0,1,0,1], [1,0,1], only_first=False, non_overlapping=False) == [0, 2] assert index_sublist([1,0,1,0,1], [1,0,1], only_first=False, non_overlapping=True) == [0]
#%%if __name__ == '__main__': test()
End.
Armin Ronacher: Skin in the Game
There was a bit of a kerfuffle about subverting open source projects recently. That incident made me think about something that's generally on my mind. That thought again was triggered by that incident but is otherwise not in response to it. I want to talk about some of the stresses of being an Open Source contributor and maintainer but specifically about something that have been unsure over the years about: anonymity and pseudonymity.
Over the years it has been pretty clear that some folks are contributing in the Open Source space and don't want to have their name attached to their contributions. I'm not going to judge if they have legitimate reasons for doing so or if pseudonymity a good or bad thing. That it is happening, is simply a fact of life. The consequences of that however are quite interesting and I think worth discussing.
When I talk about names, I primarily think about the ability to associate an online handle and a contribution to a real human being. That does not imply that it should be necessarily trivial for people to find that information, but it should be something that is at least in principle be possible. There is obviously a balance to all of this, but given that there are real consequences to “doing stuff on the internet” there has to be a way to get in contact with the person behind it. So as far as “naming a person” here is concerned it's not so much about a particular name, but as in being able to identify the human being behind it.
While we might get away with believing nothing on the internet matters and laws do not apply, that's not really true. In fact particularly with Open Source we're all leveraging copyright laws and the ability to enforce contracts to work together. And no matter how much we write “THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES” not all legal consequences can be waived.
Which leads me to some development in internet anonymity I have observed over the last 20 years which I find worth reflecting on. When I got started with Open Source, pseudonyms felt much less common. The distance to the legal system at least to me felt much closer than today. I give you a handful examples of this: When I got started doing stuff on the internet and you did something really stupid, someone called your ISP and you had an angry conversation. Because the subscriber of that line was known. A lot of the systems on the earlier internet were based on a lot more trust than would be acceptable today. An angry ISP was not the worst that would happen to you, a lot of people got charged with wire-fraud for things that today are just being ignored because they have become too commonplace (like probably most DDOS attacks these days). When I created my first SourceForge account, the “real name” field was not optional, CLAs talked about names and asked for signatures. When my stuff was packaged up in Debian some of the first things that came my way were folks explaining me some legal stuff about licenses I was unaware before. After I started getting involved with Ubuntu I went to a key signing party where I showed my passport to other human beings to demonstrate that I exist. When I became a Python core contributor I signed a physical paper for the CLA.
A lot of this feels quite untypical today. We no longer do a lot of these things and I believe it mostly just works because people don't go to court much about Open Source projects any more. It probably also works because over time Open Source became more established. If you contribute via GitHub today, even the terms of service probably help resolving copyright issues by being quite explicit about how contributions to public repositories happen (you contribute under the license of the repository).
But sometimes people do go to court. Open Source projects in many ways are an unclear amalgamation of different contributions and we just collectively hope that we all agree that contributions come in under the same licenses as the file in the root of the project. The Linux kernel once did not accept contributions from pseudonymous users. It did so for good reasons. They need to know who the person is that contributes so they know what to do in case of a licensing conflict and there was more than one lawsuit involving Linux. This was true even after the DCO was put in place. Today, pseudonyms are accepted. Not just in Linux, but also in many large projects. An example of that is the CNCF which found a nice middle ground on the name and what you sign off with: “A real name does not require a legal name, nor a birth name, nor any name that appears on an official ID (e.g. a passport). Your real name is the name you convey to people in the community for them to use to identify you as you.”
Most important however is this part: “Your real name should not be an anonymous id or false name that misrepresents who you are.” The need of getting in contact with the person exists and did not go away. It always existed and it quite likely will continue to exist. There are good reasons why you want to know who the person is. Maybe the person contributed code they did not own the copyright of, maybe their employer writes you an angry email. Concerns about licensing are a common reason for why people want to know who the people are that contribute. Maybe sanctions or other legal restrictions prevent to accept contributions from that person. Another reason you might need to get in contact with the author is to change the license. You might remember that a lot of projects tried to move from GPL v2 to GPL v2 or later. A change that required the agreement of every person that contributed before. Reaching out to people sometimes is not the easiest of tasks.
However in addition to pseudonymous contributions, there is also a sharp increase of anonymous contributions. Particularly thanks to GitHub pull requests it's incredibly common that you get commits now from folks whose only identity is a made up user name, no visible email address and some default avatar that GitHub generated.
This is not necessarily a problem, but to me it feels like a trend that I'm not sure how to work with. It creates a somewhat complex form of interaction where one person might be out in the open, the other person might be entirely anonymous. Many of us old timers who went into Open Source in former times have a pretty well established online identity (either via a “real name” or pseudonym). I also think that many of us who are in this for a while feel quite a bit of stress and responsibility for the things we created, at least that is very much true for me. Multiple times over the years did I hear or read online that a person chooses to contribute anonymously is because their employer bans Open Source work. One the one hand it's great that people find a way to avoid these restrictions, on the other hand if that ever gets found out you probably are going to have some unfriendly talks with someone else's legal team. While in practice none of my code is important enough that I think something like this will happen, I can absolutely see this happen to large Open Source projects where a rogue employee contributes on their employer's time or otherwise proprietary code.
I have a heard the sentiment a few times now that one should vet the contributions, not the contributors. That's absolutely true. Yet at the same time many of us are quite frankly assuming good actors and just happy to get contributions. We sometimes merge pull requests not in the best state of mind, sometimes we feel pressured. It can be quite hard to spot back doors and hostile commits, particularly if the other side is sufficiently motivated. But here is the thing: you know who I am, I do not know who a lot of the people are that send pull requests against my libraries. An asymmetry I need to work with.
What motivates me to write this, is that I feel quite a bit of asymmetry in contributions these days. It's a lot easier to contribute to Open Source these days and that's a good thing. But it also comes at a cost. It's impossible to find yourself having become a critical piece of software deployed all over the world by accident. Your users update to the latest version of your code without any vetting on their own. Yet the brunt of the responsibility falls on you, the person associated with the project. A person that might be known. Yet a lot of the contributions are random people, and you might not have a good change to identify them. Sometimes it's not even the contributions, it's already anonymous users on the issue trackers that increase that pressure.
I find that environment at time to be emotionally stressful, much more than it has been. I don't even maintain particularly popular pieces of Open Source libraries these days but I still feel much more stressed about that experience than years ago and a pretty big element of it is that I feel that a lot of the issues and commits are from people who show up once and then leave. Maybe it's because I'm older, or because I also have other things in my life than Open Source, but the situation is what it is.
Which brings me back to the identity thing. It's probably great for a lot of people that their online identity is not clearly linked to the real world identity. What I find less great is that with this loss of real identity many of the real world legal consequences are then stuck with me, a person that can be identified. I don't assume that knowing who the folks are that contribute will solve any problems, mind you. While I do have some probably unrealistic hope that law enforcement agencies would find it a bit easier to get involved if they can better identify a bad actor, I'm not even sure if they find much of an interest to get involved in the first place. To me, it's mostly a piece of mind thing.
Everybody's contribution into ones projects turns into a permanent liability in a way. I take responsibility of someone else's commit with the moment I press the merge button. While many of those contributions are benign no matter what, you do start to trust repeated contributors after a while. A well established identity on the internet creates a form of inner piece, a handing over a project more and more to a person you don't know less so. Yet it can happen absolutely gradually. Maybe verified identities an illusion, but sometimes these illusions is all that's needed to feel more relaxed.
I don't think we should force people to have a real world identity on the internet, but we also have to probably take a step back and look at how we came here and if we like it this way. In a sense this is a generic rant about missing the “good old times” (that probably never were), where people talked to each other eye to eye. Instead more and more, interactions on the internet feel like that they are happening with faceless figures you will probably never ever meet, see, talk or write to.
So what's left? I don't know. Neither do I know if this is a problem that only I feel, nor do I know a solution to it if it was one. All I can say is that I find Open Source stressful more than one way these days.
Glyph Lefkowitz: Software Needs To Be More Expensive
One of the ideas that James Hoffmann — probably the most influential… influencer in the coffee industry — works hard to popularize is that “coffee needs to be more expensive”.
The coffee industry is famously exploitative. Despite relatively thin margins for independent café owners1, there are no shortage of horrific stories about labor exploitation and even slavery in the coffee supply chain.
To summarize a point that Mr. Hoffman has made over a quite long series of videos and interviews2, some of this can be fixed by regulatory efforts. Enforcement of supply chain policies both by manufacturers and governments can help spot and avoid this type of exploitation. Some of it can be fixed by discernment on the part of consumers. You can try to buy fair-trade coffee, avoid brands that you know have problematic supply-chain histories.
Ultimately, though, even if there is perfect, universal, zero-cost enforcement of supply chain integrity… consumers still have to be willing to, you know, pay more for the coffee. It costs more to pay wages than to have slaves.
The Price of SoftwareThe problem with the coffee supply chain deserves your attention in its own right. I don’t mean to claim that the problems of open source maintainers are as severe as those of literal child slaves. But the principle is the same.
Every tech company uses huge amounts of open source software, which they get for free.
I do not want to argue that this is straightforwardly exploitation. There is a complex bargain here for the open source maintainers: if you create open source software, you can get a job more easily. If you create open source infrastructure, you can make choices about the architecture of your projects which are more long-term sustainable from a technology perspective, but would be harder to justify on a shorter-term commercial development schedule. You can collaborate with a wider group across the industry. You can build your personal brand.
But, in light of the recent xz Utils / SSH backdoor scandal, it is clear that while the bargain may not be entirely one-sided, it is not symmetrical, and significant bad consequences may result, both for the maintainers themselves and for society.
To fix this problem, open source software needs to get more expensive.
A big part of the appeal of open source is its implicit permission structure, which derives both from its zero up-front cost and its zero marginal cost.
The zero up-front cost means that you can just get it to try it out. In many companies, individual software developers do not have the authority to write a purchase order, or even a corporate credit card for small expenses.
If you are a software engineer and you need a new development tool or a new library that you want to purchase for work, it can be a maze of bureaucratic confusion in order to get that approved. It might be possible, but you are likely to get strange looks, and someone, probably your manager, is quite likely to say “isn’t there a free option for this?” At worst, it might just be impossible.
This makes sense. Dealing with purchase orders and reimbursement requests is annoying, and it only feels worth the overhead if you’re dealing with a large enough block of functionality that it is worth it for an entire team, or better yet an org, to adopt. This means that most of the purchasing is done by management types or “architects”, who are empowered to make decisions for larger groups.
When individual engineers need to solve a problem, they look at open source libraries and tools specifically because it’s quick and easy to incorporate them in a pull request, where a proprietary solution might be tedious and expensive.
That’s assuming that a proprietary solution to your problem even exists. In the infrastructure sector of the software economy, free options from your operating system provider (Apple, Microsoft, maybe Amazon if you’re in the cloud) and open source developers, small commercial options have been marginalized or outright destroyed by zero-cost options, for this reason.
If the zero up-front cost is a paperwork-reduction benefit, then the zero marginal cost is almost a requirement. One of the perennial complaints of open source maintainers is that companies take our stuff, build it into a product, and then make a zillion dollars and give us nothing. It seems fair that they’d give us some kind of royalty, right? Some tiny fraction of that windfall? But once you realize that individual developers don’t have the authority to put $50 on a corporate card to buy a tool, they super don’t have the authority to make a technical decision that encumbers the intellectual property of their entire core product to give some fraction of the company’s revenue away to a third party. Structurally, there’s no way that this will ever happen.
Despite these impediments, keeping those dependencies maintained does cost money.
Some Solutions Already ExistThere are various official channels developing to help support the maintenance of critical infrastructure. If you work at a big company, you should probably have a corporate Tidelift subscription. Maybe ask your employer about that.
But, as they will readily admit there are a LOT of projects that even Tidelift cannot cover, with no official commercial support, and no practical way to offer it in the short term. Individual maintainers, like yours truly, trying to figure out how to maintain their projects, either by making a living from them or incorporating them into our jobs somehow. People with a Ko-Fi or a Patreon, or maybe just an Amazon wish-list to let you say “thanks” for occasional maintenance work.
Most importantly, there’s no path for them to transition to actually making a living from their maintenance work. For most maintainers, Tidelift pays a sub-hobbyist amount of money, and even setting it up (and GitHub Sponsors, etc) is a huge hassle. So even making the transition from “no income” to “a little bit of side-hustle income” may be prohibitively bureaucratic.
Let’s take myself as an example. If you’re a developer who is nominally profiting from my infrastructure work in your own career, there is a very strong chance that you are also a contributor to the open source commons, and perhaps you’ve even contributed more to that commons than I have, contributed more to my own career success than I have to yours. I can ask you to pay me3, but really you shouldn’t be paying me, your employer should.
What To Do Now: Make It Easy To Just Pay MoneySo if we just need to give open source maintainers more money, and it’s really the employers who ought to be giving it, then what can we do?
Let’s not make it complicated. Employers should just give maintainers money. Let’s call it the “JGMM” benefit.
Specifically, every employer of software engineers should immediately institute the following benefits program: each software engineer should have a monthly discretionary budget of $50 to distribute to whatever open source dependency developers they want, in whatever way they see fit. Venmo, Patreon, PayPal, Kickstarter, GitHub Sponsors, whatever, it doesn’t matter. Put it on a corp card, put the project name on the line item, and forget about it. It’s only for open source maintenance, but it’s a small enough amount that you don’t need intense levels of approval-gating process. You can do it on the honor system.
This preserves zero up-front cost. To start using a dependency, you still just use it4. It also preserves zero marginal cost: your developers choose which dependencies to support based on perceived need and popularity. It’s a fixed overhead which doesn’t scale with revenue or with profit, just headcount.
Because the whole point here is to match the easy, implicit, no-process, no-controls way in which dependencies can be added in most companies. It should be easier to pay these small tips than it is to use the software in the first place.
This sub-1% overhead to your staffing costs will massively de-risk the open source projects you use. By leaving the discretion up to your engineers, you will end up supporting those projects which are really struggling and which your executives won’t even hear about until they end up on the news. Some of it will go to projects that you don’t use, things that your engineers find fascinating and want to use one day but don’t yet depend upon, but that’s fine too. Consider it an extremely cheap, efficient R&D expense.
A lot of the options for developers to support open source infrastructure are already tax-deductible, so if they contribute to something like one of the PSF’s fiscal sponsorees, it’s probably even more tax-advantaged than a regular business expense.
I also strongly suspect that if you’re one of the first employers to do this, you can get a round of really positive PR out of the tech press, and attract engineers, so, the race is on. I don’t really count as the “tech press” but nevertheless drop me a line to let me know if your company ends up doing this so I can shout you out.
AcknowledgmentsThank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics such as “How do I figure out which open source projects to give money to?”.
-
I don’t have time to get into the margins for Starbucks and friends, their relationship with labor, economies of scale, etc. ↩
-
While this is a theme that pervades much of his work, the only place I can easily find where he says it in so many words is on a podcast that sometimes also promotes right-wing weirdos and pseudo-scientific quacks spreading misinformation about autism and ADHD. So, I obviously don’t want to link to them; you’ll have to take my word for it. ↩
-
and I will, since as I just recently wrote about, I need to make sure that people are at least aware of the option ↩
-
Pending whatever legal approval program you have in place to vet the license. You do have a nice streamlined legal approvals process, right? You’re not just putting WTFPL software into production, are you? ↩
Robin Wilson: Explore UK place names on a map – my new web app
As an Easter present for you all, I’ve got a new web app you can play with.
It lets you search for UK place names – things like ‘ends with burgh’ or ‘starts with great’ or ‘contains sea’ – and then plot them on an interactive map. Once you’ve done that, you can even share links to your settings so other people can see your findings.
Have a look at the web app now, and please tweet/toot/email me with any interesting things you find.
A few examples:
- Various different name suffixes
- Names with river names in them
- Names with north in them vs names with south in them
- Names that start and end with the same sequence of letters
The data comes from the Ordnance Survey Open Names dataset, and it was processed on my local machine using Python to create some filtered data that could be used in the webapp, which all runs client-side.
Glyph Lefkowitz: The Hat
This year I will be going to two conferences: PyCon 2024, and North Bay Python 2024.
At PyCon, I will be promoting my open source work and my writing on this blog. As I’m not giving a talk this year, I am looking forward to organizing and participating in some open spaces about topics like software architecture, open source sustainability, framework interoperability, and event-driven programming.
At North Bay Python, though, I will either be:
- doing a lot more of that, or
- looking for new full-time employment, pausing the patreon, and winding down this experiment.
If you’d like to see me doing the former, now would be a great time to sign up to my Patreon to support the continuation of my independent open source work and writing.
The Bad NewsIt has been nearly a year since I first mentioned that I have a Patreon on this blog. That year has been a busy one, with consulting work and personal stuff consuming more than enough time that I have never been full time on independent coding & blogging. As such, I’ve focused more on small infrastructure projects and less on user-facing apps than I’d like, but I have spent the plurality of my time on it.
For that to continue, let alone increase, this work needs to—at the very least—pay for health insurance. At my current consulting rate, a conservative estimate based on some time tracking is that I am currently doing this work at something like a 98.5% discount. I do love doing it! I would be happy to continue doing it at a significant discount! But “significant” and “catastrophic” are different thresholds.
As I have said previously, this is an appeal to support my independent work; not to support me. I will be fine; what you will be voting for with your wallet is not my well-being but a choice about where I spend my time.
Hiding The HatWhen people ask me what I do these days, I sometimes struggle to explain. It is confusing. I might say I have a Patreon, I do a combination of independent work and consulting, or if I’m feeling particularly sarcastic I might say I’m an ✨influencer✨. But recently I saw the very inspiring Matt Ricardo describing the way he thinks about his own Patreon, and I realized what I am actually trying to do, which is software development busking.
Previously, when I’ve been thinking about making this “okay, it’s been a year of Patreon, let’s get serious now” post, I’ve been thinking about adding more reward products to my Patreon, trying to give people better value for their money before asking for anything more, trying to finish more projects to make a better sales pitch, maybe making merch available for sale, and so on. So aside from irregular weekly posts on Friday and acknowledgments sections at the bottom of blog posts, I’ve avoided mentioning this while I think about adding more private rewards.
But busking is a public performance, and if you want to support my work then it is that public aspect that you presumably want to support. And thus, an important part of the busking process is to actually pass the hat at the end. The people who don’t chip in still get to see the performance, but everyone else need to know that they can contribute if they liked it.1
I’m going to try to stop hiding the metaphorical hat. I still don’t want to overdo it, but I will trust that you’ll tell me if these reminders get annoying. For my part today, in addition to this post, I’m opening up a new $10 tier on Patreon for people who want to provide a higher level of support, and officially acknowledging the rewards that I already provide.
What’s The Deal?So, what would you be supporting?
What You Give (The Public Part)- I have tended to focus on my software, and there has been a lot of it! You’d be supporting me writing libraries and applications and build infrastructure to help others do the same with Python, as well as maintaining existing libraries (like the Twisted ecosystem libraries) sometimes. If I can get enough support together to more than bare support for myself, I’d really like to be able to do things like pay people to others to help with aspects of applications that I would struggle to complete myself, like accessibility or security audits.
- I also do quite a bit of writing though, about software and about other things. To make it easier to see what sort of writing I’m talking about, I’ve collected just the stuff that I’ve written during the period where I have had some patrons, under the supported tag.
- Per my earlier sarcastic comment about being an “influencer”, I also do quite a bit of posting on Mastodon about software and the tech industry.
As I said above, I will be keeping member benefits somewhat minimal.
- I will add you to SponCom so that your name will be embedded in commit messages like this one on a cadence appropriate to your support level.
- You will get access to my private Patreon posts where I discuss what I’ve
been working on. As one of my existing patrons put it:
I figure I’m getting pretty good return on it, getting not only targeted project tracking, but also a peek inside your head when it comes to Sores Business Development. Maybe some of that stuff will rub off on me :)
- This is a somewhat vague and noncommittal benefit, but it might be the best one: if you are interested in the various different projects that I am doing, you can help me prioritize! I have a lot of things going on. What would you prefer that I focus on? You can make suggestions in the comments of Patreon posts, which I pay a lot more attention to than other feedback channels.
- In the near future2 I am also planning to start doing some “office hours” type live-streaming, where I will take audience questions and discuss software design topics, or maybe do some live development to showcase my process and the tools I use. When I figure out the mechanics of this, I plan to add some rewards to the existing tiers to select topics or problems for me to work on there.
If that sounds like a good deal to you, please sign up now. If you’re already supporting me, sharing this and giving a brief testimonial of something good I’ve done would be really helpful. Github is not an algorithmic platform like YouTube, despite my occasional jokey “remember to like and subscribe”, nobody is getting recommended DBXS, or Fritter, or Twisted, or Automat, or this blog unless someone goes out and shares it.
-
A year into this, after what feels like endlessly repeating this sales pitch to the point of obnoxiousness, I still routinely interact with people who do not realize that I have a Patreon at all. ↩
-
Not quite comfortable putting this on the official patreon itemized inventory of rewards yet, but I do plan to add it once I’ve managed to stream for a couple of weeks in a row. ↩
Glyph Lefkowitz: DBXS 0.1.0
Yesterday I published a new release of DBXS for you all. It’s still ZeroVer, but it has graduated from double-ZeroVer as this is the first nonzero minor version.
More to the point though, the meaning of that version increment this version introduces some critical features that I think most people would need to give it a spin on a hobby project.
What’s New-
It has support for MySQL and PostgreSQL using native asyncio drivers, which means you don’t need to take a Twisted dependency in production.
-
While Twisted is still used for some of the testing internals, Deferred is no longer exposed anywhere in the public API, either; your tests can happily pretend that they’re doing asyncio, as long as they can run against SQLite.
-
There is a new repository convenience function that automatically wires together multiple accessors and transaction discipline. Have a look at the docstring for a sense of how to use it.
-
Several papercuts, like confusing error messages when messing up query result handling, and lack of proper handling of default arguments in access protocols, are now addressed.
If you’ve been looking for an open source project to try your hand at contributing to, DBXS might be a great opportunity, for a few reasons:
-
The team is quite small (just me, right now!), so it’s easy to get involved.
-
It’s quite generally useful, so there’s a potential for an audience, but right now it doesn’t really have any production users; there’s still time to change things without a lot of ceremony.
-
Unlike many other small starter projects, it’s got a test suite with 100% coverage, so you can contribute with confidence that you’re not breaking anything.
-
There’s not that much code (a bit over 2 thousand SLOC), so it’s not hard to get your head around.
-
There are a few obvious next steps for improvement, which I’ve filed as issues if you want to pick one up.
Share and enjoy, and please let me know if you do something fun with it.
AcknowledgmentsThank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor! I am also available for consulting work if you think your organization could benefit from expertise on topics such as “How do I shot SQL?”.
Real Python: The Real Python Podcast – Episode #198: Build a Video Game With Python Turtle & Visualize Data in Seaborn
Can you build a Space Invaders clone using Python's built-in turtle module? What advantages does the Seaborn data visualization library provide compared to Matplotlib? Christopher Trudeau is back on the show this week, along with special guest Real Python core team member Bartosz Zaczyński. We're sharing another batch of PyCoder's Weekly articles and projects.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Python Software Foundation: DjangoCon Africa Grant Process Retrospective
The PSF received an open letter asking us, amongst other things, to look into some of our recent grant decisions and make recommendations to the PSF Board for improving the Grants Program. We contracted Carol Willing, of Willing Consulting, to do this work in the form of a retrospective. Carol’s scope included reading through mailing lists, examining Board and Grants Working group norms, creating a comprehensive timeline, conducting interviews, documenting findings, and offering recommendations for the future.
In the retrospective Willing contextualizes the PSF Grants Program as part of the work of a non-profit with a charitable mission, incorporating research on best practices and effective governance. The full text of the DjangoCon Africa Grant Process Retrospective is now available. We are eager to explore the suggestions made in the retrospective and respond to community feedback.
This retrospective is just one step in our process to ensure the PSF Grants Program is responsive, transparent, and more approachable. We also recently started hosting PSF Grants Program Office Hours. The office hours are a text-only chat-based session hosted on the Python Software Foundation Discord at 1-2PM UTC (9AM Eastern) on the third Tuesday of the month. (Check what time
that is for you.) We look forward to sharing more of our progress as
we continue to enhance and improve the PSF Grants Program.
Seth Michael Larson: Security Developer-in-Residence Weekly Report #32
Published 2024-03-29 by Seth Larson
Reading time: minutes
Returned from my vacation this week and have gotten things back in order heading into April. This report covers what's happened since the first week of March.
CISA Open Source SummitI attended the Open Source Security summit hosted by CISA in early March. The event was attended by many other open source ecosystems. The summit focused on strengthening the security of open source infrastructure like package repositories.
The Principles for Package Repository Security document was a top point of discussion. This document provides a roadmap for other package repositories to prioritize security work into discrete projects and all examples have prior art that can be learned from other package repositories (such as Trusted Publishers for PyPI).
The summit also discussed the available resources and challenges between the public sector and open source software and a tabletop exercise between package repositories, the public sector, and open source maintainers and users.
Google Summer of Code 2024Google Summer of Code is open now and there are many available ideas for Python including one that I submitted with Dustin Ingram on adopting the OpenSSF Hardened Compiler Options for C/C++ for CPython. The task description is:
- There's already a list of compiler option candidates to adopt, use that as the initial list.
- Do some performance evaluation for how each compiler option affects performance (using CPython's existing performance suite). Report back on the performance impact of enabling each option.
- Implement a small custom tool (proposed in the existing issue) that allows ignoring existing violations of compiler options while preventing future violations.
- At this point we've achieved a lot of value, all future CPython contributions will have these compiler options applied.
- After the tooling is integrated, fill the rest of the project time by remediating known issues.
Applications are due by April 2nd, 2024 so if you're interested in working on this idea act quickly to prepare your application. I've already received some interest and have been providing some guidance to potential applicants.
Speaking and Tabletop Exercise participant at SOSS Community Day NAI'm speaking at the OpenSSF SOSS Community Day in Seattle on April 15th. I'm also a participant in the Tabletop Exercise that caps off SOSS Community Day.
Other items- CPython source and documentation builds moved to GitHub Actions thanks to Developer-in-Residence Łukasz Langa for reviewing and dry-running the GitHub Action during the most recent CPython release.
- Security advisories for CVE-2023-6597 and CVE-2024-0450 were published while I was away by Ee Durbin.
- CPython 3.13.0a5 is released containing some changes to default certificate verification behavior. Please test the latest CPython alpha releases for 3.13!
- Reviewed Brett Cannon's lock file pre-PEP to ensure package URLs and SBOMs can be constructed reliably and for future changes to checksum algorithms.
- I'll be blogging for the Python Language Summit at PyCon US 2024.
That's all for this week! 👋 If you're interested in more you can read last week's report.
Thanks for reading! ♡ Did you find this article helpful and want more content like it?
Get notified of new posts by subscribing to the RSS feed or the email newsletter.
This work is licensed under CC BY-SA 4.0
Matt Layman: Start Polishing - Building SaaS with Python and Django #187
Pages
Recent Publications
- Managing Hidden Dependencies in OO Software: a Study Based on Open Source Projects
- Open Source Communities as Liminal Ecosystems
- Investigating developers' email discussions during decision-making in Python language evolution
- Developers, Quality Control and Download Volume in Open Source Software (OSS) Projects
More...