Planet Python

A Python f-string (formatted string literal) allows you to insert variables or expressions directly into a string by placing them inside curly braces {}. In this tutorial, you will learn about the Pyt

We're thrilled to announce the much-anticipated return of DjangoCon Europe, set to take place in the vibrant city of Dublin, Ireland, in 2025! DjangoCon Europe has been a cornerstone of the Django community, bringing together developers and enthusiasts from all over Europe and beyond to celebrate and advance the Django web framework.

Save the Dates

Mark your calendars for DjangoCon Europe 2025, which will be held from April 23th to 27th. The conference will host a balanced mix of insightful talks, hands-on workshops, and ample opportunities for networking and socialising with fellow Django enthusiasts.

Explore Dublin

With its rich history and vibrant tech scene, Dublin is the perfect backdrop for this year's conference. Dublin's thriving tech community and innovative spirit make it an ideal host for DjangoCon Europe. Plus, the city's lively culture, breathtaking architecture, and friendly locals are sure to provide an unforgettable experience.

Call for Proposals

DjangoCon Europe wouldn't be the same without the insightful and diverse talks contributed by our community. We encourage you to consider submitting a proposal to share your knowledge, experiences, and insights with the Django community. Keep an eye out for the Call for Proposals (CFP) announcement. This is your chance to contribute to the conference program and help make DjangoCon Europe 2025 exceptional.

Get Involved

DjangoCon Europe is a community-driven event, and we rely on the active participation and support of our community members. Here are a few ways you can get involved:

• Attend: Join us in Dublin for a week of learning, networking, and fun.
• Speak: Share your expertise by submitting a talk proposal when the CFP opens.
• Sponsor: Support the conference financially and gain visibility in the Django community (email us at sponsors@djangocon.eu)
• Volunteer: Help us make the conference run smoothly by volunteering your time and skills (https://forms.gle/xmwxssiheMa1oCvPA)

Stay tuned for updates on registration, sponsorship opportunities, and more by following DjangoCon Europe on Twitter and Linkedin.

Stay Informed

To stay up-to-date with the latest DjangoCon Europe 2025 news, visit our website and follow us on Twitter & Linkedin. We will be sharing details about the schedule, speakers, and more in the coming months, so make sure you're on the list!

We can't wait to see you in Dublin for DjangoCon Europe 2025. Get ready for a week of learning, networking, and celebrating all things Django. It's going to be an unforgettable event, and we look forward to sharing this experience with you. Thank you for being a part of our amazing Django community!

See you in Dublin! 🍀

PS: Keep an eye on our social media for special offer we will have during the upcoming holiday season 😉

Kamala Harris lost. Here we are. So it goes.

Are you sad? Are you scared?

I am very sad. I am very scared.

But, like everyone else in this position, most of all, I want to know what to do next.

A Mission For Progress

I believe that we should set up a missionary organization for progressive and liberal values.

In 2017, Kayla Chadwick wrote the now-classic article, “I Don’t Know How To Explain To You That You Should Care About Other People”. It resonated with millions of people, myself included. It expresses an exasperation with a populace that seems ignorant of economics, history, politics, and indeed unable to read the news. It is understandable to be frustrated with people who are exercising their electoral power callously and irresponsibly.

But I think in 2024, we need to reckon with the fact that we do, in fact, need to explain to a large swathe of the population that they should care about other people.

We had better figure out how to explain it soon.

Shared Values — A Basis for Hope

The first question that arises when we start considering outreach to the conservative-leaning or undecided independent population is, “are these people available to be convinced?”.

To that, I must answer an unqualified “yes”.

I know that some of you are already objecting. For those of us with an understanding of history and the mechanics of bigotry in the United States, it might initially seem like the answer is “no”.

As the Nazis came to power in the 1920s, they were campaigning openly on a platform of antisemitic violence. Everyone knew what the debate was. It was hard to claim that you didn’t, in spite of some breathtakingly cowardly contemporaneous journalism, they weren’t fooling anyone.

It feels ridiculous to say this, but Hitler did not have support among Jews.

Yet, after campaigning on a platform of defaming immigrants, and Mexican immigrants specifically for a decade, a large part of what drove his victory is that Trump enjoyed a shockingly huge surge of support among the Hispanic population. Even some undocumented migrants — the ones most likely to be herded into concentration camps starting in January — are supporting him.

I believe that this is possible because, in order to maintain support of the multi-ethnic working-class coalition that Trump has built, the Republicans must maintain plausible deniability. They have to say “we are not racist”, “we are not xenophobic”. Incredibly, his supporters even say “I don’t hate trans people” with startling regularity.

Most voters must continue to believe that hateful policies with devastating impacts are actually race-neutral, and are simply going to get rid of “bad” people. Even the ones motivated by racial resentment are mostly motivated by factually incorrect beliefs about racialized minorities receiving special treatment and resources which they are not in fact receiving.

They are victims of a disinformation machine. One that has rendered reality incomprehensible.

If you listen to conservative messaging, you can hear them referencing this all the time. Remember when JD Vance made that comment about Democrats calling Diet Mountain Dew racist?

Many publications wrote about this joke “bombing”1, but the kernel of truth within it is this: understanding structural bigotry in the United States is difficult. When we progressives talk about it, people who don’t understand it think that our explanations sound ridiculous and incoherent.

There’s a reason that the real version of critical race theory is a graduate-level philosophy-of-law course, and not a couple of catch phrases.

If, without context, someone says that “municipal zoning laws are racist”, this makes about as much sense as “Diet Mountain Dew is racist” to someone who doesn’t already know what “redlining” is.

Conservatives prey upon this confusion to their benefit. But they prey on this because they must do so. They must do so because, despite everything, hate is not actually popular among the American electorate. Even now, they have to be deceived into it.

The good news is that all we need to do is stop the deception.

Politics Matter

If I have sold you on the idea that a substantial plurality of voters are available to be persuaded, the next question is: can we persuade them? Do we, as progressives, have the resources and means to do so? We did lose, after all, and it might seem like nothing we did had much of an impact.

Let’s analyze that assumption.

Across the country, Trump’s margins increased. However, in the swing states, where Harris spent money on campaigning, his margins increased less than elsewhere. At time of writing, we project that the safe-state margin shift will be 3.55% towards trump, and the swing-state margin shift will be 1.69%.

This margin was, sadly, too small for a victory, but it does show that the work mattered. Perhaps given more time, or more resources, it would have mattered just a little bit more, and that would have been decisive.

This is to say, in the places where campaign dollars were spent, even against the similar spending of the Trump campaign, we pushed the margin of support 1.86% higher within 107 days. So yes: campaigning matters. Which parts and how much are not straightforward, but it definitely matters.

This is a bit of a nonsensical comparison for a whole host of reasons2, but just for a ballpark figure, if we kept this pressure up continuously during the next 4 years, we could increase support for a democratic candidate by 25%.

We Can Teach, Not Sell

Political junkies tend to overestimate the knowledge of the average voter. Even when we are trying to compensate for it, we tend to vastly overestimate how much the average voter knows about politics and policy. I suspect that you, dear reader, are a political junkie even if you don’t think of yourself as one.

To give you a sense of what I mean, across the country, on Election day and the day after, there was a huge spike in interest for the Google query, “did Joe Biden drop out”.

Consistently over the last decade, democratic policies are more popular than their opponents. Even deep red states, such as Kansas, often vote for policies supported by democrats and opposed by Republicans.

This confusion about policy is not organic; it is not voters’ fault. It is because Republicans constantly lie.

All this ignorance might seem discouraging, but it presents an opportunity: people will not sign up to be persuaded, but people do like being informed. Rather than proselytizing via a hard sales pitch, it should be possible to offer to explain how policy connects to elections. And this is made so much the easier if so many of these folks already generally like our policies.

The Challenge Is Enormous

I’ve listed some reasons for optimism, but that does not mean that this will be easy.

Republicans have a tremendously powerful, decentralized media apparatus that reinforces their culture-war messaging all the time.

After some of the post-election analysis, “The Left Needs Its Own Joe Rogan” is on track to become a cliché within the week.3 While I am deeply sympathetic to that argument, the right-wing media’s success is not organic; it is funded by petrochemical billionaires.

We cannot compete via billionaire financing, and as such, we have to have a way to introduce voters to progressive and liberal media. Which means more voters need social connections to liberals and progressives.

Good Works

The democratic presidential campaign alone spent a billion and a half dollars. And, as shown above, this can be persuasive, but it’s just the persuasion itself.

Better than spending all this money on telling people what good stuff we would do for them if we were in power, we could just show them, by doing good stuff. We should live our values, not just endlessly reiterate them.

A billion dollars is a significant amount of power in its own right.

For historical precedent, consider the Black Panthers’ Free Breakfast For Children program. This program absolutely scared the shit out of the conservative power structure, to the point that Nixon’s FBI literally raided them for giving out free food to children.

Religious missionaries, who are famously annoying, often offset their annoying-ness by doing charitable work in the communities they are trying to reach. A lot of the country that we need to reach are religious people, and nominally both Christians and leftists share a concern for helping those in need, so we should find some cultural common ground there.

We can leverage that overlap in values by partnering with churches. This immediately makes such work culturally legible to many who we most need to reach.

Jobs Jobs Jobs

When I raised this idea with Philip James, he had been mulling over similar ideas for a long time, but with a slightly different tack: free career skills workshops from folks who are obviously “non-traditional” with respect to the average rural voter’s cultural expectations. Recruit trans folks, black folks, women, and non-white immigrants from our tech networks.

Run the trainings over remote video conferencing to make volunteering more accessible. Run those workshops through churches as a distribution network.

There is good evidence that this sort of prolonged contact and direct exposure to outgroups, to help people see others as human beings, very effective politically.

However, job skills training is by no means the only benefit we could bring. There are lots of other services we could offer remotely, particularly with the skills that we in the tech community could offer. I offer this as an initial suggestion; if you have more ideas I’d love to hear them. I think the best ideas are ones where folks can opt in, things that feel like bettering oneself rather than receiving charity; nobody likes getting handouts, particularly from the outgroup, but getting help to improve your own skills feels more participatory.

I do think that free breakfast for children, specifically, might be something to start with because people are far more willing to accept gifts to benefit others (particularly their children, or the elderly!) rather than themselves.

Take Credit

Doing good works in the community isn’t enough. We need to do visible good works. Attributable good works.

We don’t want to be assholes about it, but we do want to make sure that these benefits are clearly labeled. We do not want to attach an obligation to any charitable project, but we do want to attach something to indicate where it came from.

I don’t know what that “something” should be. The most important thing is that whatever “something” is appeals to set of partially-overlapping cultures that I am not really a part of — Midwestern, rural, southern, exurban, working class, “red state” — and thus, I would want to hear from people from those cultures about what works best.

But it’s got to be something.

Maybe it’s a little sticker, “brought to you by progressives and liberals. we care about you!”. Maybe it’s a subtle piece of consistent branding or graphic design, like a stylized blue stripe. Maybe we need to avoid the word “democrats”, or even “progressive” or “liberal”, and need some independent brand for such a thing, that is clearly tenuously connected but not directly; like the Coalition of Liberal and Leftist Helpful Neighbors or something.

Famously, when Trump sent everybody a check from the government, he put his name on it. Joe Biden did the same thing, and Democrats seem to think it’s a good thing that he didn’t take credit because it “wasn’t about advancing politics”, even though this obviously backfired. Republicans constantly take credit for the benefits of Democratic policies, which is one reason why voters don’t know they’re democratic policies.

Our broad left-liberal coalition is attempting to improve people’s material conditions. Part of that is, and must be, advancing a political agenda. It’s no good if we provide job trainings and free lunches to a community if that community is just going to be reduced to ruin by economically catastrophic tariffs and mass deportations.

We cannot do this work just for the credit, but getting credit is important.

Let’s You And Me — Yes YOU — Get Started

I think this is a good idea, but I am not the right person to lead it.

For one thing, building this type of organization requires a lot of organizational and leadership skills that are not really my forte. Even the idea of filing the paperwork for a new 501(c)3 right now sounds like rolling Sisyphus’s rock up the hill to me.

For another, we need folks who are connected to this culture, in ways that I am not. I would be happy to be involved — I do have some relevant technical skills to help with infrastructure, and I could always participate in some of the job-training stuff, and I can definitely donate a bit of money to a nonprofit, but I don’t think I can be in charge.

You can definitely help too, and we will need a wide variety of skills to begin with, and it will definitely need money. Maybe you can help me figure out who should be in charge.

This project will be weaker without your support. Thus: I need to hear from you.

You can email me, or, if you’d prefer a more secure channel, feel free to reach out over Signal, where my introduction code is glyph.99 . Please start the message with “good works:” so I can easily identify conversations about this.

If I receive any interest at all, I plan to organize some form of meeting within the next 30 days to figure out concrete next steps.

Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more things like it, or you’d like to support my various open-source endeavors, you can support my work as a sponsor! My aspirations for this support are more in the directions of software development than activism, but needs must, when the devil drives. Thanks especially to Philip James for both refining the idea and helping to edit this post, and to Marley Myrianthopoulos for assistance with the data analysis.

1. Personally I think that the perception of it “bombing” had to do with the microphones during his speech not picking up much in the way of crowd noise. It sounded to me like there were plenty of claps and laughs at the time. But even if it didn’t land with most of the audience, it definitely resonated for some of them. ↩

2. A brief, non-exhaustive list of the most obvious ones:

   • This is a huge amount of money raised during a crisis with an historic level of enthusiasm among democrats. There’s no way to sustain that kind of momentum.
   • There are almost certainly diminishing returns at some point; people harbor conservative (and, specifically, bigoted) beliefs to different degrees, and the first million people will be much easier to convince than the second million, etc.
   • Support share is not fungible; different communities will look different, and some will be saturated much more quickly than others. There is no reason to expect the rate over time to be consistent, nor the rate over geography.

   ↩

3. I mostly agree with this take, and in the interest of being the change I want to see in the world, let me just share a brief list of some progressive and liberal sources of media that you might want to have a look at and start paying attention to:

   • If Books Could Kill
   • Some More News
   • Behind The Bastards
   • Crooked Media, the publishers of Pod Save America, but you should check out everything they have on offer
   • Bryan Tyler Cohen
   • Hasan Piker
   • PhilosophyTube
   • Hbomberguy
   • FD Signifier
   • Citation Needed
   • Platformer

   Please note that not all of these are to my taste and not all of them may be to yours. They are all at different places along the left-liberal coalition spectrum, but find some sources that you enjoy and trust, and build from there. ↩

Writing a blog on the internet About • Blog • Cool URLs Writing a blog on the internet

Published 2024-11-11 by Seth Larson
Reading time: minutes

Today is the 5-year anniversary of my first blog post in 2019. Since that time I've written nearly 100 articles for my blog, something that I am quite proud of! Writing has had a huge positive impact on my life and career.

I invite you, dear reader, to start writing about topics you're interested in and sharing those writings on the internet. This article is me putting my finger on the scale by sharing what I would do differently if I were to start over again.

2★2★3★3★4★4★3★3★1★1★1★1★Hired at the PSFHired at t...44441313151513139966662★2★3★3★1★1★Writing about securityWriting about se...2222330055223322Q4Q4201920191★1★1★1★Hired at ElasticHired at E...Q2Q2Q3Q3Maintainer of urllib3Maintainer of ur...2233221144Q1Q1Q4Q420202020Q2Q2Q3Q3Q1Q1Q4Q420212021Q2Q2Q3Q3Q1Q1Q4Q420222022Q2Q2Q3Q3Q1Q1Q4Q420232023Q2Q2Q3Q3Q1Q1Q4Q420242024Q2Q2Q3Q3Q1Q1Text is not SVG - cannot display
Number of blog posts published over time with life events. Blue bars show which posts are my personal favorites. Skip the analytics§

If I were to go back in time and do one thing differently about my blog, analytics would be the one.

When I first started I used Google Analytics and found myself obsessing over the dashboards after publishing an article. This wasn't healthy, as many articles would do fine, but all the time was wasted. I'm apparently not alone in this experience.

Seeing the relatively small numbers of readers for the first few articles (single-digits...) can discourage people from writing more. Building an audience takes a looong time and plenty of persistence. That means you'll need something else to motivate you to keep at it in the mean-time.

If you insist on having analytics: I recommend GoatCounter. GoatCounter supports a mode that removes the visitor numbers and only shows referrers. The service is free for small websites, but don't forget to support them if you can.

Create what you want!

The world is a weird place, and you can't control what becomes popular. Create what you want to create for the sake of creating and enjoy the ride!

My most popular article by an extremely wide margin is one I didn't expect: "Move or recover your Wordle stats". I created this little utility for me and my friends and didn't expect hundreds of thousands of people to use it until the New York Times shared the URL on Twitter.

My most recent viral article I wrote in ~15 minutes about an unexpected behavior in Python regular expressions that caused a bug in some of my code.

Own your work

Publishing on the internet means deciding where you will publish your work. We've seen far too many platforms either die or become completely user-hostile. To prevent this from happening to your hard work:

• Create the original work in a format that is transformable (such as Markdown or HTML).
• Publish that work to publicly accessible URLs that you can share.
• Share your URLs in many ways, like RSS, email newsletter, social media, or elsewhere.

For easy-to-start publishing platforms, I recommend either GitHub Pages or Bear Blog. If you have the savvy and interest: host content on your own website. There are far too many guides to getting started with this, choose one using a technology that you're interested in.

See also: "Publish on your Own Site, Syndicate Elsewhere".

Let your authenticity shine

Please note that I am a cis white male and has not had to justify my existence or expertise in a space. Unfortunately not all of my friends can be their authentic selves online, but knowing them in real life I certainly wish that the world allowed them to be.

I always enjoy when a blog shows off the author, either through writing style, phrases, personal touches, pictures, jokes, or little pieces of life. Don't be afraid to leave in things you think about when you're writing. I try to strike the right balance between how I might speak about an idea if I were to talk in person and writing for a diverse audience.

Don't put yourself in a box

You don't have to write about the same one or two topics, forever. I am guilty of this, and I am working on writing about more than only open source and security. Recently I have started to write about video-game preservation.

Again, write about what you want to write about. Writing about something new, even if it's only once, can be very refreshing. Don't let vague feelings about what your audience "expects" to get in the way of creative expression.

Don't think you only need to write about "professional" topics or topics that have broad appeal. You can write about anything at all from rocket science to what's happening in your local community.

Start at the end

Start with the conclusion! A reader should be able to know your main ideas without a single page scroll (because almost all readers won't make it past the first few paragraphs). Check out how your draft looks on a phone to confirm this is the case.

After that first page scroll you've already pared down to the more dedicated readers so start giving them details. If you like narrative writing like me, this is a good place to start the actual story.

Keep it short

In terms of writing, you should be able to write the main points and details of an article quickly, assuming you've done your research beforehand.

Once all the main points are there, resist the urge to make an in-progress article "more grand" or "comprehensive". Instead, link out to resources that already exist or plan on writing follow-up articles later. Many smaller articles are more easily consumable for readers and more writeable for you (double win!)

Ship early instead of never

I've wasted so much time trying to "finish" blog posts. Endlessly trying to polish something into being perfect is not worth it, because it increases the chances that the work won't ever be published!

Try to be okay publishing something that isn't perfect, because your idea of "perfect" will change over time. You need to go through the "research-write-edit-publish" cycle to improve, not by endlessly editing one piece.

Hang up when you're done

Don't worry about "conclusions" or "wrapping-up" a blog post at the end. Just stop writing as soon there's no more to say. I promise almost no one reads all the way to the end (except your most loyal readers: remember they like you!)

Speaking of stopping: this is it! Thanks to everyone who has read this blog 💜

Have thoughts or questions? Let's chat over email or social:

sethmichaellarson@gmail.com
@sethmlarson@fosstodon.org

Want more articles like this one? Get notified of new posts by subscribing to the RSS feed or the email newsletter. I won't share your email or send spam, only whatever this is!

Want more content now? This blog's archive has ready-to-read articles. I also curate a list of cool URLs I find on the internet.

Find a typo? This blog is open source, pull requests are appreciated.

Thanks for reading! ♡ This work is licensed under CC BY-SA 4.0

︎

A closer look at non-elementary group-by aggregations

Meeting Minutes from Python Docs Editorial Board: November 11, 2024

Have you wanted the flexibility of f-strings but need safety checks in place? What if you could have deferred evaluation for logging or avoiding injection attacks? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

What is this pattern about?

The Pipeline design pattern (also known as Chain of Command pattern) is a flexible way to handle a sequence of actions, where each handler in the chain processes the input data and passes it to the next handler. This pattern is commonly used in scenarios involving data processing, web scraping, or middleware systems.

In this blog post, I’ll walk you through a specific example that leverages Python’s powerful functools.reduce and partial functions, along with the BeautifulSoup library for parsing HTML content. This code showcases the Pipeline pattern applied to HTML table extraction and processing.

What Does the Code Do?

The code defines a pipeline of data parsing steps for extracting and cleaning tables from an HTML file. It follows a functional programming approach to compose several processing functions into a unified flow using the Chain of Command pattern.

Key Concepts

1. Functional Composition: Combining multiple functions into one that executes in a specific order.
2. Data Parsing Pipeline: Sequential processing of HTML content into structured data (a DataFrame).
3. Error Handling: Ensuring the pipeline gracefully handles missing or malformed data.

Let’s break down the code step by step:

1. Function Composition with composefrom functools import reduce, partial from typing import Callable

The pipeline is created by composing multiple parsing functions into a single unified function. The compose function uses reduce to chain these functions together:

def compose(*functions: ParsingPipeline) -> ParsingPipeline: """Composes functions into a single function""" return reduce(lambda f, g: lambda x: g(f(x)), functions, lambda x: x)

This allows you to define an ordered flow of operations that process input data from one function to the next. Each function modifies the input data, which is then passed down the pipeline.

2. Reading HTML Content

The first step in the pipeline is to read the contents of an HTML file. This is done by read_htm_from:

def read_htm_from(filename: T, mode: T = "r", encoding: T = "utf-8") -> T: with open(filename, mode, encoding=encoding) as file: html_content = file.read() return html_content

This function opens an HTML file and returns its content as a string. It supports different file modes and encodings, making it flexible for various file formats.

Note that T is defined here as TypeVar("T"), see the typing docs.

3. Parsing the HTML Table

Next, read_table_from uses BeautifulSoup to find the HTML table within the file:

from bs4 import BeautifulSoup def read_table_from(htm_file: T, parser: str = "html.parser") -> T: soup = BeautifulSoup(htm_file, parser) table = soup.find("table") return table

This function converts the HTML content into a BeautifulSoup object and extracts the first table it finds. The parsed table is passed down the pipeline for further processing.

4. Extracting Rows and Data

Once the table is identified, the pipeline extracts the rows and applies filtering logic based on custom markers:

def extract_row_data_from( table_rows: T, start_markers: T, continue_markers: T, end_markers: T ) -> T: row_data: T = [] start_processing = False for row in table_rows: if any(marker in row.text for marker in start_markers) and not start_processing: start_processing = True continue if start_processing: if any(marker in row.text for marker in continue_markers): continue if any(marker in row.text for marker in end_markers): break row_data.append(row) return row_data[:-1]

This function inspects each row in the table, checking if the row text matches specified start, continue, or end markers. Data extraction begins after encountering the start marker and ends when the end marker is found.

5. Converting Rows to DataFrame

The next steps involve transforming the extracted row data into a structured pandas DataFrame. First, the rows are separated into individual columns using separate_columns_in:

def separate_columns_in(rows: T) -> T: data_rows: T = [] try: for row in rows: columns = row.find_all(["td", "th"]) data = [col.text for col in columns] data_rows.append(data) return data_rows except Exception as e: print(f"An error occurred: {str(e)}") return []

Then, convert_to_dataframe reshapes this data into a pandas DataFrame:

def convert_to_dataframe(data_rows: T) -> T: df = pd.DataFrame(data_rows) df = df.rename(columns=df.iloc[0]).drop(df.index[0]) df.columns = COLUMN_NAMES df.drop(columns=COLUMNS_TO_REMOVE, axis=1, inplace=True) df.set_index(df.columns[0], inplace=True, drop=True) return df

The DataFrame is cleaned up by renaming columns, removing unnecessary columns, and setting the correct index.

6. Assigning Correct Data Types

Finally, assign_correct_data_type_to ensures that the DataFrame columns have the appropriate data types:

def assign_correct_data_type_to( df: T, dict_types: dict[str, str] = COLUMN_TYPES, datetime_columns: list[str] = DATETIME_COLUMN_NAMES, ) -> T: if not isinstance(df, pd.DataFrame): raise ValueError("Input `df` must be a pandas DataFrame.") df = df.copy() for column in datetime_columns: if column in df.columns: df[column] = pd.to_datetime(df[column]) for column, col_type in dict_types.items(): if column in df.columns: try: if col_type == "numeric": df[column] = pd.to_numeric(df[column], errors="coerce") else: df[column].astype(col_type) except Exception as e: print(f"Error converting column {column} to {col_type}: {e}") return df

This function converts columns into numeric or datetime formats as needed, ensuring that the data is properly structured for analysis.

7. Putting It All Together

At the end of the code, the pipeline is composed by chaining all of the above functions together:

parse_gbx_bt: ParsingPipeline = compose( partial(read_htm_from, mode="r", encoding="utf-8"), read_table_from, read_rows_from, partial( extract_row_data_from, start_markers=["Closed Transactions:"], continue_markers=["Genbox", "balance", "Deposit"], end_markers=["Closed P/L:"], ), separate_columns_in, convert_to_dataframe, assign_correct_data_type_to, )

This creates a fully automated pipeline that:

1. Reads an HTML file.
2. Extracts table data.
3. Cleans and converts the data into a pandas DataFrame.
4. Assigns the correct data types.

Conclusion

This implementation of the Chain of Command or Pipeline pattern in Python demonstrates how to apply functional programming principles to data parsing tasks. The use of functools.reduce and partial, and BeautifulSoup provides a flexible, reusable way to process HTML content and structure it into usable data.

If you’re looking to create complex data processing pipelines that need to handle dynamic data from HTML or other sources, this approach is a clean and maintainable solution.

You can find the code in the repo: https://github.com/jjeg1979/pyBacktestAnalyzer.

And if you want to watch the code clinic where I presented the tool, feel free to check it out at https://pybites.circle.so/c/circle-coaching-calls/python-for-the-trader-code-clinic.

If you cannot access…well, what are you waiting for to become a PDM member?

I wrote in the past about how I'm a pessimist that strives for positive outcomes. One of the things that I gradually learned is is wishing others to succeed. That is something that took me a long time to learn. I did not see the value in positive towards other people's success, but there is. There is one thing to be sceptical to a project or initiative, but you can still encourage the other person and wish them well.

I think not wishing others well is a coping mechanism of sorts. For sure it was for me. As you become more successful in life, it becomes easier to be supportive, because you have established yourself in one way or another and you feel more secure about yourself.

That said, there is something I continue to struggle with, and that are morals. What if the thing the other person is doing seems morally wrong to me? I believe that much of this struggle stems from the fear of feeling complicit in another's choices. Supporting someone — even passively — can feel like tacit approval, and that can be unsettling. Perhaps encouragement doesn't need to imply agreement. Another angle to consider is that my discomfort may actually stem from my own insecurities and doubts. When someone's path contradicts my values, it can make me question my own choices. This reaction often makes it hard to wish them well, even when deep down I want to.

What if my tribe is just wrong on something? I grew up with the idea of “never again”. Anything that remotely looks like fascism really triggers me. There is a well known propaganda film from the US Army called “Don't Be a Sucker” which warns Americans about the dangers of prejudice, discrimination, and fascist rhetoric. I watched this a few times over the years and it still makes me wonder how people can fall for that kind of rhetoric.

But is it really all that hard? Isn't that happening today again? I have a very hard time supporting what Trump or Musk are standing for or people that align with them. Trump's rhetoric and plans are counter to everything I stand for and the remind me a lot of that film. It's even harder for me with Musk. His morals are completely off, he seems to a person I would not want to be friends with, yet he's successful and he's pushing humanity forward.

It's challenging to reconcile my strong opposition to their (and other's) rhetoric and policies with the need to maintain a nuanced view of them. Neither are “literal Hitler”. Equating them with the most extreme historical figures oversimplifies the situation and shuts down productive conversation.

Particularly watching comedy shows reducing Trump to a caricature feels wrong to me. Plenty of his supporters have genuine concerns. I find it very hard to engage with these complexities and it's deeply uncomfortable and quite frankly exhausting.

Life becomes simpler when you just pick a side, but it will strip away the deeper understanding and nuance I want to hold onto. I don’t want to fall into the trap of justifying or defending behaviors I fundamentally disagree with, nor do I want to completely shut out the perspectives of those who support him. This means accepting that people I engage with, might see things very differently, and that maintaining those relationships and wishing them well them requires a level of tolerance I'm not sure I possess yet.

The reason it's particularly hard to me that even if I accept that my tribe maybe wrong in parts, I can see the effects that Trump and others already had on individuals. Think of the Muslim travel ban which kept families apart for years, his border family separation policy, the attempted repeal of Section 230. Some of it was not him, but people he aligned with. Things like the overturning of Roe v. Wade and the effects it had on women, the book bans in Florida, etc. Yes, not quite Hitler, but still deeply problematic for personal freedoms. So I can't ignore the harm that some of these policies have caused in the past and even if I take the most favorable view of him, I have that track record to hold against him.

In the end where does that leave me? Listening, understanding, and standing firm in my values. But not kissing the ring. And probably coping by writing more.

I’ve seen this meme a bunch of times recently. I always reply; what is asserted without evidence may be dismissed without consideration.

Current Gen AI is flawed by hallucination issues, mired in copyright controversy, expensive to run and lacking clear use cases. (Although it’s pretty good at code generation). It’s a massive hype train.

Gen AI, as it is now, was made possible by the invention of “Transformer Architecture” by Google in 2017. We’re seeing fast paced change and development, but all built on that technology.

At some point another quantum breakthrough will change things all over again - and make another step towards AGI. Although it will take several such steps, and order of magnitudes larger models (and multi models), to create anything resembling true AI.

So a huge number of disparate individuals, institutions, governments and companies are pursuing the development of AI. There’s no single cohesive agenda behind it. As new technologies arise we adapt to them, find uses for them, and everyone pursues their agendas with them.

Not particularly special to AI I don’t think.

Python metaclasses, considered advanced programming and Python “black magick” (*) explained in eight words:

The type of a class is a class.

Here’s what knowledge of Object Oriented theory and type systems permit you to deduce from this:

Using the word “class”, instead of “the type of a class is type” or even “the type of a class is a type, classes are types”, implies that a user defined class can be a metaclass. This is indeed the case, and the point of metaclasses in Python.

The type is responsible for creating new instances. So if the type of a class is a class then we can write classes that create classes. Indeed this is the primary usecase for metaclasses.

(Deeper knowledge of Python, and the two phase object creation protocol, may lead you to deduce that this is done by overriding the __new__ method. If you’re familiar with “type” as a class factory you can probably even guess the signature and that you must inherit from type.)

If the type of a class is a class then the type system will permit a type check for the class against its class. And indeed isinstance(klass, metaclass) returns true.

(And deeper knowledge of Python will tell you that the magic methods, the protocol methods, are always looked up on the type. So we can implement behaviour for class objects by providing magic methods on the metaclass.)

All of this implies that classes are themselves objects. Which is true in Python for everything is an object in Python (and everything is a reference).

And so on…

• Type and class are synonyms in Python.
• type(type) is type

And to further round out the type system, these are also Python invariants:

• isinstance(object, object) is True # object is an object
• isinstance(object, type) is True # but also a type
• isinstance(type, object) is True # type is an object
• isinstance(type, type) is True # but also a type

(*) Like all black magick it is useful for understanding the world but never for actual use. Well, except perhaps in very rare circumstances if you know what you’re doing.

📘 Written in 2021.

IronPython in Action was published on the 7th April 2009 and we sold a little over 7000 copies.

Royalties for last quarter amounted to $25.

It took me two years to write thirteen chapters and a couple of appendices, and took Christian Muirhead about the same to write two chapters and an appendix. Jonathan Hartley did the diagrams and illustrations and the worst part was compiling the index.

It took so long because IronPython was still in alpha (!) when we started and it changed several times (including a Silverlight version being released) whilst writing!

After leaving Resolver Systems in 2010 I spent a year contracting on Line of Business apps that ran in Silverlight (Django on the server): Python code running in the browser on the client side. It was glorious.

We even had functional tests on unittest built in to the app.

Work on mock accelerated massively once IronPython in Action was complete. MagickMock was born not long afterwards.

I was also helping maintain the python.org website and adding test discovery to unittest at the time, and speaking at every conference I could find.

It felt like the glory days of the Python community. It’s almost time for PyCon (online) and I’m nostalgic once again.

My first PyCon, the second Dallas PyCon and my first time in the US, there were about 600 attendees. You could almost know everyone.

I shaved my beard to enter Dallas and wore my hair in a pony tail. All I knew was they didn’t like hippies there. It was the nicest greeting at a US airport I’ve ever had.

I went on a road trip with Andrzej Krzywda afterwards trying to find mountains. We found the Ouchita mountains in Oaklahoma and drove back through Arkansas to visit friends of mine in Houston. Along the peaks of the mountains, which are hills really, we found a view called Dead Man’s Vista and we I laughed together at Microsoft.

Not long after this the web explosion happened and Django happened, google adopted Python as an official language and the community started to explode and grow.

That was even before Python became huge as a teaching language and before Python exploded in data science too.

I once paired with Jacob Kaplan Moss at a PyCon sprint and fixed some issue by adding a metaclass to the Django codebase. Which he never committed and found a better way.

That’s the closest I’ve come to deploying a metaclass I think, although I’ve removed a few in my time.

I knew Python had “made it” as a language when one bag stuffing pre-PyCon I met someone who didn’t want to be there. He’d been sent by work. Before that Python was obscure, and only people who really loved it went to PyCon. Which I’m convinced is the secret of Python’s success.

It was built by passion not by money. For the sheer love and the joy of building something beautiful with other people.

I was a Mac user then and had a running joke with Jonathan Hartley about Linux and projectors.

One time he plugged his laptop into the projector prior to his PyCon talk (Testing is a Silver Bullet), tried to fix the x-config from the terminal and rendered his laptop unusable. He did the presentation on mine. The next year Mark Shuttleworth did a keynote talk at PyCon and running some bleeding edge version of Ubuntu also couldn’t plug it into the projector system. Hilarity on my part.

The biggest conference I ever spoke at was a Microsoft one in Brighton where they demoed Silverlight and I demoed IronPython on Silverlight. They didn’t tell me I would be on main stage in front of a few thousand Microsoft devs. I was used to talking to a few hundred at a time!

I had a slide deck built from S5 with reStructured Text markup and a Far Side slide mocking static typing. Which went down a bomb to an audience of C# devs. I still managed, by coincidence, to demo almost the same features of Silverlight as Microsoft bigwig Scott Hanselman who did the keynote.

It was an “interesting experience”, evangelising Python and dynamic languages in “the heart of the beast” as it were. Microsoft went on to step up their involvement with Python and sincere Open Source commitments which they’ve maintained since.

Since I first wrote this Python has finally made it, ranked as the most widely used programming language in the world by TIOBE and PyPL. World number one.

I joined Twitter fourteen years ago and have tweeted over fifty-two thousand times. I follow 1,636 accounts, which is too many, and have 8,670 followers. I use Tweetdeck which is run by Twitter and doesn’t show ads or promoted tweets or mess with tweet order and it lets me use two different accounts.

I use twitter a lot less than I did during my social media and community frenzy whilst I delighted to learn Python, but I still enjoy it.

During that time (2006-2011) I “drank from the firehose”. I read all of slashdot (scanned every headline and read relevant articles), read all of comp.lang.python (every message title - read and replied to many), read all of python-dev (similarly) and all of testing-in-python, blogged almost daily and worked full time as a software engineer commuting to London four times a week and developed mock in my spare time and worked on unittest in the Python standard library. And wrote a book and worked part time doing community liaison and service development for a local charity working with the homeless and disadvantaged. I was Microsoft MVP for three years for my work with IronPython, I spoke at countless conferences and received the Python Software Foundation Community Award for my work running Planet Python and helping out with the Python.org website and mailing infrastructure.

Then in 2011 my first child was born and I started working for Canonical. Three years of large Django web applications then three years of Go and MongoDB and then a year with Red Hat testing Ansible Tower and now four years self employed.

During that time I remembered that the primary drive in my life was spiritual and I started meditating again. One hour a day of mindfulness of breathing. That transformed my life all over again.

I once rode in the back of a beaten up station wagon owned and operated by the creator of the Python programming language whilst sat alongside the creator of Bitorrent, which was written in Python.

I also once had a pub lunch in Oxford with the creator of the Erlang programming language and the creator of the Haskell programming language. We were all three speaking at the ACCU conference. I was speaking on IronPython.

It’s been a fun journey.

Building on Part 1 of this PSF Grants Program Update, we are pleased to share updates to the Grants Workgroup (workgroup) Charter. We have outlined all the changes below in a chart, but there are a couple of changes that we’d like to highlight to grant applicants. These updates in particular will change how and when you apply, and hopefully reduce blockers to getting those applications in and ready for review. Because we are just sharing these updates, we are happy to be flexible on these changes but hope to see all applicants adhere to the changes starting around January 2025. 

• Increase overall process time frame to 8 weeks (formerly 6 weeks). We want to be realistic about how long the process takes and we know that going over our projection can cause pain for applicants. We hope to turn around applications in 6 weeks in most cases, but planning for the extra two weeks can make a big difference for everyone involved!
• Our application form requires that you set the event date out to 6 weeks in advance. We will wait to update that to 8 weeks in advance until January 2025.
• It’s important to note that this time frame begins only once all required information has been received, not exactly from the day the application is submitted. Make sure to check the email you provided on the application to see if the workgroup Chair has any questions regarding your request!
• Add a statement of support for accessibility services. In line with the PSF’s mission to support and facilitate the growth of a diverse community, we are explicitly stating in the charter that we will consider funding accessibility services. For established events (have 2 or more events in the past with more than 200 participants at the last event), we are open to considering accessibility-related requests such as live captioning, sign language interpretation, or certified child care.
• To review these types of requests, we will need sufficient documentation such as quotes, certifications, or any other relevant information.
• Add guidelines around program/schedule review. Previously undocumented, we were checking event programs/schedules to ensure a Python focus as well as a diversity of speakers. Because of event organizing time frames, we often received grant requests before the schedule was available. Moving forward we are accepting 1 of 3 options:
1. The program/schedule for the event
2. A tentative schedule or list of accepted speakers/sessions for the event
3. Programs from previous editions of the event if available, a link to the event’s call for proposals, which should state a required Python focus for the event as well as a statement in support of a diverse speaker group, and a description of the efforts that are being made to ensure a diversity of speakers.

Grants Workgroup Charter Updates   Update Summary Former Charter Projected Benefit Observations Establish fast-track grants: Grants that meet pre-approved criteria skip the review period with the workgroup and go straight to a vote Did not exist previously Resolutions reach applicants sooner, reduce load on workgroup Not many events meet the initial criteria we set to qualify for fast-track review, so this is mostly untested Establish workgroup participation criteria: workgroup members must participate in 60% of the votes to remain active Did not exist previously Resolutions reach applicants sooner, set out clear guidelines on the meaning of active participation, reduce load on Chair Reduction of workgroup membership to only active members has resulted in shorter voting periods by removing blockers to meeting quorum Increase $ amount for PSF Board review: Grant requests over 15K are reviewed by PSF Board Grant requests over 10K were reviewed by PSF Board Resolutions reach applicants sooner, reduces load on PSF Board to ensure they are focused on high level efforts Resolutions have reached applicants sooner, some reduction in load for PSF Board as we are still receiving applications over 15K Increase process timeframe: 8 week processing time from when all information has been received 6 week processing time from when all information has been received Improves community satisfaction and sets realistic expectations, reduces stress on workgroup & Chair We are just sharing this update so it has yet to be tested- come to our Grants Office Hour session to discuss it with us! Establish schedule for Grant review process: 10 day review period and 10 day voting period Did not exist previously Improve community satisfaction by ensuring requests are moving through the process promptly This has worked great to keep things moving as the workgroup has a set expectation of how long they have to comment Establish guideline for workgroup process: no discussion after the vote has begun Did not exist previously Improve community satisfaction by ensuring requests are moving hrough the process promptly While untested, this has set an expectation for the workgroup to comment during the review period Update voting mechanics: Votes will last for 10 days, a majority is reached, or when all voting members have voted, whichever comes first. For a proposal to be successful, it must have ayes in the majority totalling 30% of the WG Decisions were made by a majority rule (50%+1), with no time limit Improve community satisfaction by ensuring votes take place promptly, reduce stress on the workgroup and Chair if members are absent or unable to participate This has worked wonderfully! The Chair no longer has to track down votes. Paired with the participation guideline, voting periods no longer present a bottleneck Removed stated set budget: The annual budget is set by the PSF Board and is subject to change The previously documented budget was $120,000 (regularly exceeded) Removes an inaccurate description of the Grants Program budget and the need to update this line yearly A practical update, no observations to note Update workgroup officer roles: one Chair, one Vice Chair, one Appointed Board Director One Chair and two Vice Chairs Correct an unusual and discouraged practice of having two vice chairs and ensures PSF Board participation A practical update, no observations to note Add a statement of support for accessibility services: for mature events, consideration of granting funds for accessibility services Did not exist previously Establishes criteria for the workgroup and Board to consider accessibility-related requests We are just sharing this update so it has yet to be tested- come to our Grants Office Hour session to discuss it with us! Additional guidelines around grant reviews: tentative schedules OR previous schedules, CfP that shows a Python focus, as well as a description of the efforts being made to ensure a diversity of speakers Did not exist previously in documented form, though we checked for a program Improve community satisfaction with the process, remove delays in the grant review process This has been a great addition, and blockers for many applications have been removed! .table { display: block; overflow-y: hidden; overflow-x: auto; scroll-behavior: smooth; } .table thead { display: table-header-group; vertical-align: middle; border-color: inherit; color: inherit; background: darkcyan; } table, th, td { border: 1px solid black; border-collapse: collapse; } tr { display: table-row; vertical-align: inherit; border-color: inherit; } table th { padding: 16px; text-align: inherit; border-bottom: 1px solid black; color:white!important; } tbody { display: table-row-group; vertical-align: middle; border-color: inherit; } table:not(.tr-caption-container) { min-width: 100%; border-radius: 3px; }  What’s next?

Still on our Grants Program refresh to-do list is:

• Mapping Board-mandated priorities for the Grants Program to policy
• Charter adjustments as needed, based on the priority mapping
  
• Main documentation page re-write
• Budget template update
• Application form overhaul
• Transparency report for 2024
• Exploration and development of other resources that our grant applicants would find useful

Our community is ever-changing and growing, and we plan to be there every step of the way and continue crafting the Grants Program to serve Pythonistas worldwide. If you have questions or comments, we welcome and encourage you to join us at our monthly Grants Program Office Hour sessions on the PSF Discord.

Time has flown by since we received the community call last December for greater transparency and better processes around our Grants Program. PSF staff have produced a Grants Program Transparency Report and begun holding monthly Grants Program Office Hours. The PSF Board also invested in a third-party retrospective and launched a major refresh of all areas of our Grants program.

To provide the Grants Program more support, we assigned Marie Nordin, PSF Community Communications Manager, to support the Grants Program alongside Laura Graves, Senior Accountant. Marie has stepped into the Grants Workgroup Chair role to relieve Laura after 3+ years– thank you, Laura! Marie has been leading the initiatives and work related to the Grants Program in collaboration with Laura.

Behind the scenes, PSF staff has been working with the PSF Board and the Grants Workgroup (workgroup) to translate the feedback we’ve received and the analysis we’ve performed into action, starting with the Grants Workgroup Charter. A full breakdown of updates to the charter can be found in Part 2 of this update.

The PSF Board spent time on their recent retreat to explore priorities for the program going forward. We also ran a more thorough workgroup membership renewal process based on the updated charter to support quicker grant reviews and votes through active workgroup engagement. We’re excited to share refresh progress, updates, and plans for the future of the program later on in this post!

Something wonderful, bringing more changes

Meanwhile, the attention our Grants Program has received in the past year has resulted in something wonderful: we’re getting more requests than ever. Our call to historically underrepresented regions to request funds has been answered in some areas- and we are thrilled! For example, in the African region, we granted around 65K in 2023 and over 140K already this year! And, year to date in 2024 we have awarded more grant funding than we did in all of 2023. The other side of this coin presents us with a new issue– the budget for the program.

Up until this year, we’ve been able to grant at least partial funding to the majority of requests we’ve received while staying within our guidelines and maintaining a feasible annual budget. With more eligible requests incoming, every “yes” brings us closer to the ceiling of our grant budget. In addition to the increased quantity of requests, we are receiving requests for higher amounts. Inflation and the tech crunch have been hitting event organizers everywhere (this includes the PSF-produced PyCon US), and we are seeing that reflected in the number and size of the grant requests we are receiving.

Moving forward, with the increased quantity and amount of eligible grant requests, we will need to take steps to ensure we are balancing grant awards with sustainability for our Grants Program, and the Foundation overall. We know that the most important part of any changes to the Grants Program is awareness and two-way communications with the community. We aim to do that as early and transparently as we possibly can. That means we aren’t changing anything about how we award grants today or even next week– but within the next couple of months. Please keep an eye on our blog and social accounts (Mastodon, X, LinkedIn) for news about upcoming changes, and make sure to share this post with your fellow Python event and initiative organizers.

Grants Workgroup Charter update process

The purpose of the PSF Grants Workgroup (workgroup) is to review, approve, and deny grant funding proposals for Python conferences, training workshops, Meetups, development projects, and other related Python initiatives. The workgroup charter outlines processes, guidelines, and membership requirements for the workgroup. Small changes have been made to the charter over the years, but it’s been some time since any significant changes were implemented.
 
During the summer of 2024, Marie, workgroup chair (hi 👋 it’s me writing this!), and Laura worked on updates for the charter. The updates focused on how to make the Grants Program processes and guidelines work better for the workgroup, the PSF Board, and most especially, the community we serve.

After many hours of discussing pain points, running scenarios, exploring possible guidelines, and drafting the actual wording, Marie and Laura introduced proposed updates for the charter to the Board in July. After a month of review and 1:1 meetings with the PSF Board and workgroup members, the updated charter went to a vote with the PSF Board on August 14th and was approved unanimously.

The workgroup has been operating under its new charter for a couple of months. Before we shared broadly with the community, we wanted to make sure the updates didn’t cause unintended consequences, and we were ready to walk back anything that didn’t make sense. Turns out, our hard work paid off, and the updates have been mostly working as we hoped. We will continue to monitor the impact of the changes and make any adjustments in the next Charter update. Read up on the Grants Workgroup Charter updates in Part 2 of this blog post!

In this tutorial, you’ll learn how to reset a pandas DataFrame index, the reasons why you might want to do this, and the problems that could occur if you don’t.

Before you start your learning journey, you should familiarize yourself with how to create a pandas DataFrame. Knowing the difference between a DataFrame and a pandas Series will also prove useful to you.

In addition, you may want to use the data analysis tool Jupyter Notebook as you work through the examples in this tutorial. Alternatively, JupyterLab will give you an enhanced notebook experience, but feel free to use any Python environment you wish.

As a starting point, you’ll need some data. To begin with, you’ll use the band_members.csv file included in the downloadable materials that you can access by clicking the link below:

Get Your Code: Click here to download the free sample code you’ll use to learn how to reset a pandas DataFrame index.

The table below describes the data from band_members.csv that you’ll begin with:

Column Name PyArrow Data Type Description first_name string First name of member last_name string Last name of member instrument string Main instrument played date_of_birth string Member’s date of birth

As you’ll see, the data has details of the members of the rock band The Beach Boys. Each row contains information about its various members both past and present.

Note: In case you’ve never heard of The Beach Boys, they’re an American rock band formed in the early 1960s.

Throughout this tutorial, you’ll be using the pandas library to allow you to work with DataFrames, as well as the newer PyArrow library. The PyArrow library provides pandas with its own optimized data types, which are faster and less memory-intensive than the traditional NumPy types that pandas uses by default.

If you’re working at the command line, you can install both pandas and pyarrow using the single command python -m pip install pandas pyarrow. If you’re working in a Jupyter Notebook, you should use !python -m pip install pandas pyarrow. Regardless, you should do this within a virtual environment to avoid clashes with the libraries you use in your global environment.

Once you have the libraries in place, it’s time to read your data into a DataFrame:

Python >>> import pandas as pd >>> beach_boys = pd.read_csv( ... "band_members.csv" ... ).convert_dtypes(dtype_backend="pyarrow") Copied!

First, you used import pandas to make the library available within your code. To construct the DataFrame and read it into the beach_boys variable, you used pandas’ read_csv() function, passing band_members.csv as the file to read. Finally, by passing dtype_backend="pyarrow" to .convert_dtypes() you convert all columns to pyarrow types.

If you want to verify that pyarrow data types are indeed being used, then beach_boys.dtypes will satisfy your curiosity:

Python >>> beach_boys.dtypes first_name string[pyarrow] last_name string[pyarrow] instrument string[pyarrow] date_of_birth string[pyarrow] dtype: object Copied!

As you can see, each data type contains [pyarrow] in its name.

If you wanted to analyze the date information thoroughly, then you would parse the date_of_birth column to make sure dates are read as a suitable pyarrow date type. This would allow you to analyze by specific days, months or years, and so on, as commonly found in pivot tables.

The date_of_birth column is not analyzed in this tutorial, so the string data type it’s being read as will do. Later on, you’ll get the chance to hone your skills with some exercises. The solutions include the date parsing code if you want to see how it’s done.

Now that the file has been loaded into a DataFrame, you’ll probably want to take a look at it:

Python >>> beach_boys first_name last_name instrument date_of_birth 0 Brian Wilson Bass 20-Jun-1942 1 Mike Love Saxophone 15-Mar-1941 2 Al Jardine Guitar 03-Sep-1942 3 Bruce Johnston Bass 27-Jun-1942 4 Carl Wilson Guitar 21-Dec-1946 5 Dennis Wilson Drums 04-Dec-1944 6 David Marks Guitar 22-Aug-1948 7 Ricky Fataar Drums 05-Sep-1952 8 Blondie Chaplin Guitar 07-Jul-1951 Copied!

DataFrames are two-dimensional data structures similar to spreadsheets or database tables. A pandas DataFrame can be considered a set of columns, with each column being a pandas Series. Each column also has a heading, which is the name property of the Series, and each row has a label, which is referred to as an element of its associated index object.

The DataFrame’s index is shown to the left of the DataFrame. It’s not part of the original band_members.csv source file, but is added as part of the DataFrame creation process. It’s this index object you’re learning to reset.

The index of a DataFrame is an additional column of labels that helps you identify rows. When used in combination with column headings, it allows you to access specific data within your DataFrame. The default index labels are a sequence of integers, but you can use strings to make them more meaningful. You can actually use any hashable type for your index, but integers, strings, and timestamps are the most common.

Note: Although indexes are certainly useful in pandas, an alternative to pandas is the new high-performance Polars library, which eliminates them in favor of row numbers. This may come as a surprise, but aside from being used for selecting rows or columns, indexes aren’t often used when analyzing DataFrames. Also, row numbers always remain sequential when rows are added or removed in a Polars DataFrame. This isn’t the case with indexes in pandas.

Read the full article at https://realpython.com/pandas-reset-index/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Right now, I am on a never ending quest that requires me to think of building a full fledge MVC controller : an anti-jira tracker that would favours HARD CHECKED facts over wishful thinking.

For this to begin, I am not really motivated in beginning with a full fledged MVC (Model View Controller) à la django because there is a lot of boilerplates and actions to do before a result. But, it has a lot of feature I want, including authentication, authorization and handling security.

For prototypes we normally flavours lightweight framework (à la flask), and CRUD.

CRUD approach is a factorisation of all framework in a single dynamic form that adapts itself to the model to generate HTML forms to input data, tabulate, REST endpoints and search them from the python class declaration and generate the database model. One language to rule them all : PYTHON. You can easily generate even the javascript to handle autocompletion on the generated view from python with enough talent.

But before using a CRUD framework, we need a cruder one, ugly, disgusting but useful for a human before building the REST APIs, writing the class in python, the HTML form, and the controlers.

I call this the crudest CRUD of them all.

Think hard at what you want when prototyping ...

• to write no CONTROLLERS ; flask documentation has a very verbose approach to exposing routes and writing them, writing controller for embasing and searching databases is boring
• to write the fewer HTML views possible, one and only onle would be great ;
• to avoid having to fiddle the many files reflecting separation of concerns : the lesser python files and class you touch the better;
• to avoid having to write SQL nor use an ORM (at least a verbose declarative one) ;
• show me your code and you can mesmerize and even fool me, however show me your data structure and I'll know everthing I have to know about your application : data structure should be under your nose in a readable fashion in the code;/
• to have AT LEAST one end point for inserting and searching so that curl can be used to begin automation and testing, preferably in a factorisable fashion;
• only one point of failure is accepted

Once we set these few condition we see whatever we do WE NEED a dynamic http server at the core. Python being the topic here, we are gonna do it in python.

What is the simplest dynamic web server in python ?

The reference implementation of wsgi that is the crudest wsgi server of them all : wsgiref. And you don't need to download it since it's provided in python stdlib.

First thing first, we are gonna had a default view so that we can serve an HTML static page with the list of the minimal HTML we need to interact with data : sets of input and forms.

Here, we stop. And we see that these forms are describing the data model.

Wouldn't it be nice if we could parse the HTML form easily with a tool from the standard library : html.parser and maybe deduce the database model and even more than fields coud add relationship, and well since we are dreaming : what about creating the tables on the fly from the form if they don't exists ?

The encoding of the relationship do require an hijack of convention where when the parser cross a name of the field in the form whatever_id it deduces it is a foreign key to table « whatever », column « id ».
Once this is done, we can parse the html, do some magick to match HTML input types to database types (adapter) and it's almost over. We can even dream of creating the database if it does not exists in a oneliner for sqlite.

We just need to throw away all the frugality of dependencies by the window and spoil our karma of « digital soberty » by adding the almighty sqlalchemy the crudest (but still heavy) ORM when it comes of the field of the introspective features of an ORM to map a database object to a python object in a clear consistent way. With this, just one function is needed in the controller to switch from embasing (POST method) and searching (GET).

Well, if the DOM is passed in the request. So of course I see the critics here :

• we can't pass the DOM in the request because the HTML form ignores the DOM
• You are not scared of error 415 (request too large) in the get method if you pass the DOM ?

That's where we obviously need two important tools : 1) javascript, 2) limitations.

Since we are human we would also like the form to be readable when served, because, well, human don't read the source and can't see the name attributes of the input. A tad of improving the raw html would be nice. It would also give consistency. It will also diminishes the required size of the formular to send. Here, javascript again is the right anwser. Fine, we serve the static page in the top of the controller. Let's use jquery to make it terse enough. Oh, if we have Javascript, wouldn't il be able to clone the part of the invented model tag inside every form so now we can pass the relevant part of the DOM to the controller ?

I think we have everything to write the crudest CRUD server of them all :D

Happy code reading : import multipart from wsgiref.simple_server import make_server from json import dumps from sqlalchemy import create_engine, MetaData, Table, Column from sqlalchemy import Integer, String, Float, Date, DateTime,UnicodeText, ForeignKey from html.parser import HTMLParser from sqlalchemy.ext.automap import automap_base from sqlalchemy.orm import Session from sqlalchemy import select from sqlalchemy import create_engine from sqlalchemy_utils import database_exists, create_database from urllib.parse import parse_qsl, urlparse engine = create_engine("postgresql://jul@192.168.1.32/pdca") if not database_exists(engine.url): create_database(engine.url) tables = dict() class HTMLtoData(HTMLParser): def __init__(self): global engine, tables self.cols = [] self.table = "" self.tables= [] self.engine= engine self.meta = MetaData() super().__init__() def handle_starttag(self, tag, attrs): attrs = dict(attrs) if tag == "input": if attrs.get("name") == "id": self.cols += [ Column('id', Integer, primary_key = True), ] return try: if attrs.get("name").endswith("_id"): table,_=attrs.get("name").split("_") self.cols += [ Column(attrs["name"], Integer, ForeignKey(table + ".id")) ] return except Exception as e: print(e) if attrs["type"] in ("email", "url", "phone", "text"): self.cols += [ Column(attrs["name"], UnicodeText ), ] if attrs["type"] == "number": if attrs["step"] == "any": self.cols+= [ Columns(attrs["name"], Float), ] else: self.cols+= [ Column(attrs["name"], Integer), ] if attrs["type"] == "date": self.cols += [ Column(attrs["name"], Date) ] if attrs["type"] == "datetime": self.cols += [ Column(attrs["name"], DateTime) ] if attrs["type"] == "time": self.cols += [ Column(attrs["name"], Time) ] if tag== "form": self.table = urlparse(attrs["action"]).path[1:] def handle_endtag(self, tag): if tag=="form": self.tables += [ Table(self.table, self.meta, *self.cols), ] tables[self.table] = self.tables[-1] self.table = "" self.cols = [] with engine.connect() as cnx: self.meta.create_all(engine) cnx.commit() html = """ <!doctype html> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script> <script> $(document).ready(function() { $("form").each((i,el) => { $(el).wrap("<fieldset>"+ el.action + "</fieldset>" ); $(el).append("<input type=submit value=insert formmethod=post ><input type=submit value=search formmethod=get />"); }); $("input:not([type=hidden],[type=submit])").each((i,el) => { $(el).before("<label>" + el.name+ "</label><br/>"); $(el).after("<br>"); }); }); </script> </head> <body> <form action=/user > <input type=number name=id /> <input type=text name=name /> <input type=email name=email > </form> <form action=/event > <input type=number name=id /> <input type=date name=date /> <input type=text name=text /> <input type=number name=user_id /> </form> </body> </html> """ router = dict({"" : lambda fo: html,}) def simple_app(environ, start_response): fo,fi=multipart.parse_form_data(environ) fo.update(**{ k: dict( name=fi.filename, content=fi.file.read().decode('utf-8', 'backslashreplace'), content_type=fi.content_type, ) for k,v in fi.items()}) table = route = environ["PATH_INFO"][1:] fo.update(**dict(parse_qsl(environ["QUERY_STRING"]))) start_response('200 OK', [('Content-type', 'text/html; charset=utf-8')]) try: HTMLtoData().feed(html) except KeyError: pass metadata = MetaData() metadata.reflect(bind=engine) Base = automap_base(metadata=metadata) Base.prepare() if route in tables.keys(): with Session(engine) as session: Item = getattr(Base.classes, table) if environ.get("REQUEST_METHOD", "GET") == "POST": new_item = Item(**{ k:v for k,v in fo.items() if v and not k.startswith("_")}) session.add(new_item) ret=session.commit() fo["insert_result"] = new_item.id if environ.get("REQUEST_METHOD") == "GET": result = [] for elt in session.execute( select(Item).filter_by(**{ k : v for k,v in fo.items() if v and not k.startswith("_")})).all(): result += [{ k.name:getattr(elt[0],k.name) for k in tables[table].columns}] fo["search_result"] = result return [ router.get(route,lambda fo:dumps(fo.dict, indent=4, default=str))(fo).encode() ] print("Crudest CRDU of them all on port 5000...") make_server('', 5000, simple_app).serve_forever()

Kamal offers zero-downtime deploys, rolling restarts, asset bridging, remote builds, accessory service management, and everything else you need to deploy and manage your web app in production with Docker. Originally built for Rails apps, Kamal will work with any type of web app that can be containerized. We dig into Kamal, how it works, and how you could use it on your next project.

This article looks at how to avoid the count query in Django's paginator.

#654 – NOVEMBER 5, 2024
View in Browser »

PySheets: Spreadsheets in the Browser Using PyScript

What goes into building a spreadsheet application in Python that runs in the browser? How do you make it launch quickly, and where do you store the cells of data? This week on the show, we speak with Chris Laffra about his project, PySheets, and his book “Communication for Engineers.”
REAL PYTHON podcast

Adding Keyboard Shortcuts to the Python REPL

Python 3.13 included a new version of the REPL which has the ability to define keyboard shortcuts. This article shows you how to create one and warns you about potential hangups.
TREY HUNNER

Tired of Being Paged? Worry Less With Temporal

Say goodbye to managing failures, network outages, flaky endpoints, and long-running processes. Temporal ensures your code never fails. Period. PLUS, you can get started today on Temporal Cloud with $1,000 free credits on us →
TEMPORAL TECHNOLOGIES sponsor

Running a Million Empty Tests

To better understand just where the performance cost of running tests comes from, Anders ran a million empty tests. This post talks about what he did and the final results.
ANDERS HOVMOLLER

Pillow Release 11.0.0

GITHUB.COM/PYTHON-PILLOW

PEP 750: Template Strings (Major Updates)

PSF

PEP 756: Add PyUnicode_Export() and PyUnicode_Import() C Functions (Withdrawn)

PSF

Python 3.8 Reaches End of Life

PYTHON.ORG

Quiz: Single and Double Underscores in Python Names

REAL PYTHON

Quiz: Getting Started With Async Features in Python

REAL PYTHON

Discussions Thinking of Rewriting Our Go / Java API in Python

REDDIT

Best GUI for Local Client App?

REDDIT

Articles & Tutorials Move to Sigstore Complicates Linux Distros

Currently, CPython signs its artifacts with both PGP and Sigstore. Removing the PGP signature has been proposed, but that has implications: Sigstore is still new enough that many Linux distributions don’t support it yet.
JOE BROCKMEIER

Python’s Magic Methods in Classes

In this video course, you’ll learn what magic methods are in Python, how they work, and how to use them in your custom classes to support powerful features in your object-oriented code.
REAL PYTHON course

[Webinar] How to Build Secure, Ethical, and Scalable AI Operations

As GenAI and LLMs rapidly evolve, the impact of data leaks and unsafe AI outputs makes it critical to secure your AI infrastructure. Learn how MLOps and ML Platform teams can use the newly launched Guardrails Pro to secure AI operations — enabling faster, safer adoption of LLMs at scale →
GUARDRAILS sponsor

Make It Ephemeral: Software Should Decay and Lose Data

In the real world, things decay over time. In the digital world things get kept forever, and sometimes that shouldn’t be so. Designing for deletion is hard.
ARMIN RONACHER

Python 3.13, t-Strings, Dep Groups…

Bite code! does their monthly Python news wrap-up. Check out stories on 3.13, proposed template strings, dependency groups in pyproject.toml, and more.
BITE CODE!

Identifying Products From Images

This project uses computer vision solution to automate doing inventory of products in retail, using YOLOv8 and image embeddings for precise detection.
ALBERT FERRÉ • Shared by Albert Ferré

Write More Pythonic Code With Context Managers

Context managers enable you to create “template” code with initialization and clean up to make the code that uses them easier to read and understand.
JUHA-MATTI SANTALA

Django Girls 10th Birthday!

This post celebrating ten years of Django Girls talks about how it got started, what they’re hoping to do, and how you can get involved.
DJANGO GIRLS

pytest Selection Arguments for Failing Tests

This quick TIL post talks about five useful pytest options that let you control what tests to run with respect to failing tests.
RODRIGO GIRÃO SERRÃO

Asyncio gather() Return Values

This post shows you how to return values from coroutines that have been concurrently executed using asyncio.gather().
JASON BROWNLEE

PyBay 2024

This list contains the recorded talks from the PyBay 2024 conference.
YOUTUBE video

Projects & Code wimsey: Data Contract Library

GITHUB.COM/BENRUTTER

libcom: Image Composition Toolbox

GITHUB.COM/BCMI

simplemind: Experimental Client for AI Providers

GITHUB.COM/KENNETHREITZ

PyChrono: Multi-Physics Simulation in Python

CRISTIANOPIZZAMIGLIO.COM • Shared by Cristiano Pizzamiglio

jamesql: In-Memory NoSQL Database in Python

GITHUB.COM/CAPJAMESG

Events Weekly Real Python Office Hours Q&A (Virtual)

November 6, 2024
REALPYTHON.COM

Canberra Python Meetup

November 7, 2024
MEETUP.COM

Sydney Python User Group (SyPy)

November 7, 2024
SYPY.ORG

DFW Pythoneers 2nd Saturday Teaching Meeting

November 9, 2024
MEETUP.COM

PiterPy Meetup

November 12, 2024
PITERPY.COM

PyCon Sweden 2024

November 14 to November 16, 2024
PYCON.SE

PyCon Hong Kong 2024

November 16 to November 17, 2024
PYCON.HK

PyCon Mini Tokai 2024

November 16 to November 17, 2024
PYCON.JP

PyCon Ireland 2024

November 16 to November 18, 2024
PYTHON.IE

Happy Pythoning!
This was PyCoder’s Weekly Issue #654.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Web scraping is the process of collecting and parsing raw data from the Web, and the Python community has come up with some pretty powerful web scraping tools.

The Internet hosts perhaps the greatest source of information on the planet. Many disciplines, such as data science, business intelligence, and investigative reporting, can benefit enormously from collecting and analyzing data from websites.

In this video course, you’ll learn how to:

• Parse website data using string methods and regular expressions
• Parse website data using an HTML parser
• Interact with forms and other website components

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]