Planet Python

Are you curious to discover the latest trends in Django development?

In collaboration with the Django Foundation, PyCharm surveyed more than 4,000 Django developers from around the globe and analyzed the trends in framework usage based on their answers.

In this blog post, we share the following key findings with you:

• Every third Django developer also uses Flask or FastAPI.
• Most developers use Django for both full-stack and API development.
• 61% of Django developers use asynchronous technologies.
• And many more insights!

Dive in to learn about these findings in more detail and discover other trends in Django development, while also benefiting from illustrative infographics.

Backend: Every third Django developer also uses Flask or FastAPI

Django remains the go-to framework for 74% of developers, though this does represent a minor dip from last year when this figure stood at 83%. FastAPI has managed to maintain its popularity, as 25% of respondents reported using it. Meanwhile, Flask saw a slight decline in popularity (29% in 2022 to 26% in 2023).

33% of web developers who work primarily with Django also use Flask or FastAPI, showing diverse backend skills.

Taking into account that the majority of fully employed developers (49%) report working on several projects at the same time, this may indicate that they choose different tools for different purposes:

• Django – for larger, more complex web apps due to its “batteries included” approach.
• Flask – for simpler applications (especially static sites) or microservices.
• FastAPI – to create API endpoints, especially if your application includes a lot of IO calls (especially for real-time web applications).

The fact that only 11% of all Django developers use all three frameworks might mean that most of them use Flask and FastAPI for similar purposes, shifting to FastAPI due to its async capabilities.

Want to learn how Django is different from Flask and FastAPI? Check out our detailed comparisons between Django and Flask, and Django and FastAPI to see which framework will best suit your needs.

Developing APIs: Most developers use Django for both full-stack and API development

This year’s survey revealed Django is popular for both full-stack (74%) and API development (60%), with a trend towards API work among fully employed devs. Fully employed developers are more likely to use Django for REST API development (65% vs. 60% on average), but less likely to use it for full-stack development (68% vs. 74% on average).

With the rising popularity of htmx, the trend might change in favor of using Django more for full-stack development. 

Interestingly, while DRF has retained the pole position among third-party packages, its popularity dipped as Django Ninja, known for its speed and typing capabilities, continues to gain ground. Django Ninja offers high performance and asynchronous capabilities, similar to another very popular choice for creating APIs, FastAPI, but within the Django ecosystem, which makes the learning curve shorter.

Work with APIs? Read this tutorial to learn how to build APIs with the Django REST framework.

Async: 61% of Django developers use async

There’s a clear transition towards using asynchronous technologies among Django developers, with 61% now incorporating async in their projects (up from 53% last year).

FastAPI, which was built with async programming in mind, is now used by 21% of all Django developers who use async technologies. Django async views are also being used by more respondents (14%), though FastAPI is still more popular for async tasks. With more async support planned for upcoming Django 5 releases, the interest in using async in Django might increase even more.

Frontend: a shift in Django developers’ preferences towards htmx, Alpine.js, and Tailwind CSS

When it comes to frontend, JavaScript is still the most popular language, with 68% of developers stating that they use it. That said, it’s gradually conceding its comfortable lead (75% in 2021 and 2022) to TypeScript, which has grown significantly from 19% in 2021 to 28% in 2023. ​​This increase in popularity is probably attributable to its static typing features, which help catch errors early in the development process and thus make code more robust and maintainable.

Professional Django developers still clearly favor JavaScript frameworks over their competitors, with usage rates of 26% for Vue, 35% for jQuery, and 42% for React, though their overall use is declining year over year.

Newer frameworks like htmx (which grew from 16% in 2022 to 23% in 2023) and Alpine.js (which grew from 6% to 10%) are rapidly gaining traction, suggesting a shift towards simpler tools for modern user interfaces. There is a dedicated django-htmx package developed by Adam Johnson.

Dennis Ivy on htmx:

“Happy to see HTMX here. I’m not the biggest fan (I default to React) but I think it’s suitable for many Django projects that need a little flexibility without having to move to a full fledged JS framework/library.”

We continue to see a downward trend for Bootstrap and significant growth for Tailwind CSS, whose popularity has doubled in the last two years. The increasing preference for Tailwind CSS over Bootstrap suggests a desire for more customizable and less prescriptive approaches to styling in web projects. Read this article from package creator Tim Kamanin for a guided introduction to using Tailwind CSS in Django. 

Dennis Ivy on TailwindCSS:

“Love to see the rise of Tailwind-Django usage. Hope to see more native integration and educational content in this area.” Databases: 75% of Django developers favor PostgreSQL, and 50% rely on Redis for caching

In the Django ecosystem, PostgreSQL leads the pack as the primary database choice (76%) among developers, highlighting the preference for robust, SQL-based systems for web applications. The interest in NoSQL databases like MariaDB (10%) and MongoDB (8%) is also notable, reflecting a diversifying database landscape.

The inclusion of the schema-less and scalable MongoDB among the top database choices, despite its lack of official Django support, reflects developers’ willingness to integrate more flexible, document-oriented databases.

Do you work with MongoDB? Read this step-by-step guide on how to connect Django with MongoDB.

Dennis Ivy on MongoDB:

“This number surprises me considering the different approach Mongo takes and the incompatibility with Django. Curious to know the experience level of those 8%. I would suspect that those are newer devs and / or devs running experimental projects. 

After speaking to several MongoDB team members I was able to conclude my thoughts on this integration in an article.”

On the caching front, Redis remains the go-to solution (54%) for enhancing web app responsiveness, with Memcached (20%) also gaining traction.

Orchestration: More than 50% of Django developers use container orchestration 

Among the favored services, Amazon ECS/Fargate (19%) leads by virtue of its ease of use and integration with AWS, making it a natural choice for developers in the AWS ecosystem. 

Self-managed Kubernetes (14%) appeals to those seeking flexibility and control over their infrastructure, as well as the ability to easily migrate and share between private and public clouds. The popularity of Amazon EKS (12%) and Docker Swarm (12%) might be attributable to the balance they offer between manageability and scalability, catering to various deployment needs.

Deal with Kubernetes infrastructure? Read this guide on how to deploy Django apps in Kubernetes.

CI systems: GitHub Actions is leading the industry

GitHub Actions’ growth (45% in 2023 compared to 35% in 2021) in the CI field highlights its convenience for developers who already use GitHub for source code management. Its simplicity, leveraging straightforward YAML files for pipeline management, makes it an accessible and efficient tool for automating software workflows directly within GitHub’s ecosystem. Moreover, it offers the flexibility of using custom hardware configurations that meet your needs with enough processing power or memory to run larger jobs.

IaC: Infrastructure as code is used by 39% of Django developers

The use of infrastructure as code solutions by 39% of respondents underscores a growing trend towards automation and infrastructure management through code. For larger projects, IaC can ensure more reliable, repeatable, and scalable infrastructure setups. Terraform is the most commonly used infrastructure as code provisioning engine, with 20% of all respondents preferring it over the other options.

Interestingly, an open-source solution, Pulumi, was chosen by 5% of the respondents. The reason for this could be the fact that, right from its inception, Pulumi has offered the flexibility to use any programming language for managing infrastructure. This makes Pulumi widely accessible to developers and DevOps engineers from any background. Terraform began offering a similar option via a CDK in 2022.

Insights based on Django developers’ jobs and experience Django learning resources

Fully employed developers watch less YouTube to learn Django (32% vs. 39% on average) and use fewer AI tools for this purpose (22% vs. 25% on average).

Among team leads, the Django News newsletter, HackerNews, Reddit, and even X (formerly Twitter) are more popular methods of staying up to date with Django developments. They even report learning more often from their friends (16% vs. 11% on average). 

Junior professionals use YouTube and StackOverflow much more often for both educational purposes and to keep abreast of developments in the Django ecosystem. When it comes to their preferred learning method, they tend to use new AI tools more often than their senior colleagues (38% vs. 25% on average).

Miscellaneous

• Fully employed Django developers are more likely to use Django only for work (23% vs. 17% on average). 

• The core component that team leads and fully employed developers as a group favor most is the possibility of migration, with their less preferred components including authentication, templates, and even class-based views.

• As their main editor, team leads tend to prefer PyCharm (31% vs. 29% on average) and Vim (12% vs. 7% on average) to VS Code (31% vs. 29% on average).

Start developing Django apps with PyCharm

Do you work with Django? PyCharm is the best-in-class IDE for Django. Code faster with Django-specific code insights, code completion, and highlighting. Navigate across your project easily. Connect to your database in a single click, and work on TypeScript, JavaScript, and other frontend frameworks. PyCharm also supports Flask and FastAPI out of the box.

Try PyCharm for free

Survey demographics

After filtering out duplicate and unreliable responses, the data set includes around 4,000 responses collected between September and October 2023.

Regional distribution

44% of respondents are based in Europe, 19% in North America, and 17% in Asia.

Age distribution

Most of the respondents are in the 21–49 age range. 38% of all respondents are aged 31–39, while 30% are aged 21–29.

Professional coding experience

The majority of respondents have been coding professionally for 11+ years. 24% of the survey participants have 3–5 years of professional coding experience, and 19% have worked as professional developers for 6–10 years. Those who have been developing professionally for less than two years represent another 25% of the survey respondents.

Job roles

79% of respondents stated that their job roles include development/programming or software engineering. 16% of respondents are team leads. 10% of the respondents stated that their job includes data analysis, data engineering, or data science. 

The data set includes responses only from official Django Software Foundation channels. The responses were collected through the promotion of the survey on official Django channels, such as djangoproject.com and the DSF’s X (formerly Twitter) account, without the involvement of any JetBrains channels. In order to prevent the survey from being slanted in favor of any specific tool or technology, no product-, service-, or vendor-related channels were used to collect responses.

Want to learn more? Explore the Django Developers Survey 2023 to see the full survey data.

Test your understanding of Python’s tools for string interpolation, including f-strings, the .format() method, and the modulo operator.

Take this quiz after reading our String Interpolation in Python: Exploring Available Tools tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

I realised recently that I’d never actually blogged about my pyAURN package – so it’s about time that I did.

When doing some freelance work on air quality a while back, I wanted an easy way to access UK air quality from the Automatic Urban and Rural Network (AURN). Unfortunately, there isn’t a nice API for accessing the data. Strangely, though, they do provide the data in a series of RData files for use by the openair R package. I wanted to use the data from Python though – but conveniently there is a Python package for reading RData files.

So, I put all these together into a simple Python package called pyAURN. It is strongly based upon openair, but is a lot more limited in functionality – it basically only covers importing the data, and doesn’t have many plotting or analysis functions.

Here’s an example of how to use it:

from pyaurn import importAURN, importMeta, timeAverage # Download metadata of site IDs, names, locations etc metadata = importMeta() # Download 4 years of data for the Marylebone Road site # (MY1 is the site ID for this site) # Note: range(2016, 2022) will produce a list of six years: 2016, 2017, 2018, 2019, 2020, and 2021. # Alternatively define a list of years to use eg. [2016,2017,2018,2019,2020,2021] data = importAURN("MY1", range(2016, 2022)) # Group the DataFrame by a frequency of monthly, and the statistic mean(). data_monthly = timeAverage(data,avg_time="month",statistic="mean")

I found this really useful for my air quality analysis work – and I hope you do too. The package is on PyPI, so you can run pip install pyaurn or view the project on Github.

New API Integration Tutorial in Python 2024-06-02, by Dariusz Suchojad

Do you know what airports, telecom operators, defense forces and health care organizations have in common?

They all rely heavily on deep-backend software systems which are integrated and automated using principled methodologies, innovative techniques and well-defined implementation frameworks.

If you'd like to learn how to integrate and automate such complex systems correctly, head over to the new API integration tutorial that will show you how to do it in Python too.

# -*- coding: utf-8 -*- # Zato from zato.server.service import Service # ############################################################################## class MyService(Service): """ Returns user details by the person's name. """ name = 'api.my-service' # I/O definition input = '-name' output = 'user_type', 'account_no', 'account_balance' def handle(self): # For later use name = self.request.input.name or 'partner' # REST connections crm_conn = self.out.rest['CRM'].conn billing_conn = self.out.rest['Billing'].conn # Prepare requests crm_request = {'UserName':name} billing_params = {'USER':name} # Get data from CRM crm_data = crm_conn.get(self.cid, crm_request).data # Get data from Billing billing_data = billing_conn.post(self.cid, params=billing_params).data # Extract the business information from both systems user_type = crm_data['UserType'] account_no = crm_data['AccountNumber'] account_balance = billing_data['ACC_BALANCE'] self.logger.info(f'cid:{self.cid} Returning user details for {name}') # Now, produce the response for our caller self.response.payload = { 'user_type': user_type, 'account_no': account_no, 'account_balance': account_balance, } # ##############################################################################

➤ API programming screenshots
➤ Here's the API integration tutorial again
➤ More API programming examples in Python

More blog posts➤

During the last month we focused on fixing bugs, improving the behaviour of things, speeding-up performance issues - building on the changes from our last release. We also added some new features which we would like to introduce to you in this newsletter.

For an in depth overview of the Tryton issues please take a look at our issue tracker or see the issues and merge requests filtered by label.

Changes for the User CRM, Sales, Purchases and Projects

We’ve moved the language field on the party form to the header.

Accounting, Invoicing and Payments

Tryton now ensures the payment amount is greater than zero.

Stock, Production and Shipments

We’ve added a code with a sequence to the production bill of materials (BOM) to clearly distinguish similarly named BOMs.

We’ve also added a shipped state to the customer shipments.
The shipped-state becomes handy when the delivery process takes a long time.
After being packed now the delivery can become shipped or done (similar to internal and drop-shipments). The transition packed → shipped will delete the staging outgoing moves.

As stock lots are not associated with a company, we now ensure that sequences used to generate their numbers are not be linked to a specific company to avoid access right errors.

User Interface

We updated the TinyMCE in Sao to version 7 (see external changelog).

In all of our Tryton clients the user can now close tabs by clicking on them using the middle mouse button.

To avoid accidentally changing configuration values for external services, we now raise a confirmation warning when changing the credential settings in the following modules:

• account_payment_braintree
• account_payment_stripe

more… (click for more details) System Data and Configuration

We’ve standardized the format of pictures used in Tryton. Product and avatar images are converted to RGB with a resolution of 300 DPI, so we can rely on a consistent printed size.

We added a description field to the product image which can be used as the alt property on websites.

Now reports in Tryton can be printed with a company logo in PNG-format (for transparency support).

New Documentation

We added section anchors to each option of the configuration documentation. To refer to the docs of an option, you can now use e.g. Email settings <config-email>.

New Releases

We released bug fixes for the currently maintained long term support series
7.0 and 6.0, and for the penultimate series 6.8.

Changes for the System Administrator

The logging messages generated by modules and the bus have been unified.

The CSV import now handles the :lang= suffix when it is appended to a translatable field name.

Changes for Implementers and Developers

We’ve added partial support for the Unique and Exclude constraints in the SQLite backend with UNIQUE INDEX as long as they do not use parameters.

A python-format flag is now added to the PO-files.

Authors: @dave @pokoli @udono

1 post - 1 participant

Read full topic

How do you start adding unit tests to your Python code? Can the built-in unittest framework cover most or all of your needs? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

54th issue of Andrei Neagoie's must-read monthly Python Newsletter: Python Security Best Practices, State of Python 2024, Prompt Engineering, and much more. Read the full newsletter to get up-to-date with everything you need to know from last month.

This has been terribly quiet since July 2022, oops. It turns out that having an infant totally sucks your time! In the meantime I’ve continued to build up:

• Training courses – I’ve just listed my new Fast Pandas course plus the existing Successful Data Science Projects and Software Engineering for Data Scientists with runs of all three for July and September
• My NotANumber newsletter, it goes out every month or so, carries Python data science jobs and talks on my strategic work, RebelAI leadership community and Higher Performance Python book updates
• RebelAI – my private data science leadership community (there’s no web presence, just get in touch) for “excellent data scientists turned leaders” – this is having a set of very nice impacts for members
• High Performance Python O’Reilly book – we’re working on the 3rd edition
• PyDataLondon 2024 has a great schedule and if you’re coming – do find me and say hi!

Ian is a Chief Interim Data Scientist via his Mor Consulting. Sign-up for Data Science tutorials in London and to hear about his data science thoughts and jobs. He lives in London, is walked by his high energy Springer Spaniel and is a consumer of fine coffees.

With the number of Python-related conferences around the world, many people might wonder how the selection process is configured and performed. For the largest and oldest European Python conference, EuroPython, we wanted to share how this process works.

The Programme team for each EuroPython conference changes every year. There are mechanisms in place to carry on some of the processes and traditions when dealing with proposals for the next event, although you can still find some differences each year. These differences are not large enough to make a significant impact.

The 2024 Process

In this post, we highlight how the 2024 process was conducted and your role in it, whether as a submitter or potential contributor in future versions.

Opening the Call for Proposals

This year, the Call for Proposals (CfP) configuration was based on the 2023 version, with minor modifications. For example, two new tracks were added to enable more people to categorise their proposals:

• PyData: Research & Applications
• PyData: LLMs

This change was motivated by the popularity of these topics at other global conferences. In addition, other tracks were merged or removed to keep the number manageable.

For many people, having a configuration with both an Abstract and a Description is confusing. Not everyone knows what to write in each field. To address this, we decided to be clearer: we dropped the Description field and asked explicitly for an Outline section. The intention was that to submit a proposal, one would require an Abstract and an Outline for their session.

Reviewers from the Community

We opened a form to get help from the community for reviewing the talks and decided to accept most (if not all) of them to nurture our review results as much as possible. We had more than 20 people helping with reviewing proposals on Pretalx.

We created a few “track groups” containing related categories that the reviewers could choose from. This way, there was no pressure to have an opinion on a topic one might not be familiar with.

We had an average of six reviews per proposal, which greatly helped us make a final decision.

Community Voting

Another way to receive input is through Community Voting, which allows participants who have attended any of the EuroPythons since 2012 to vote on the upcoming programme.

Using a separate simple web application, people participated by voting for the proposals they wanted to see at EuroPython 2024. We were fortunate to have enough people voting to get a good estimate of preferences.

Fun fact: Around 11 people were able to review all of the nearly 640 proposals we received this year.

We are very grateful to everyone who participated!

My reaction when I saw a good proposal to be voted at EP 2024.Programme Committee

This year the programme committee was mostly formed by a group of new people to the conference, helped by a few people familiar with the process from last year. In total, around 11 people were actively participating.

Like most Programme teams, we did our best to get people from different areas to have a more diverse general mindset, including skills from Data, Core, DevOps, and Web technologies.

It was important for us to have local people on the team, and we are very happy to have had two members from the local Czech community helping, while the rest were spread across Europe.

Selection Process

Based on the reviewers&apos results from Pretalx and Community Voting, we generated a master sheet that was used to perform the selection process.

Track by track, the Programme team went through each proposal that had good Pretalx and Community Voting results and voted (again) for the talks they believed were good material for the conference.

During the selection process, we felt that we did not have enough expertise in a specific area. Therefore, we are very thankful that we could add four more members to the selection team to remedy that.

After three calls, each lasting around 2 hours, the Programme team had the first batch of accepted proposals. The speakers for these proposals were notified as soon as the decision was made. Following a similar process, we did the same for the second (and final) batch of accepted and rejected proposals.

To ensure the acceptance of proposals from most tracks and topics, many plots and statistical analyses were created to visualise the ratio of accepted proposals to submitted ones, the variety of topics, and the diversity of speakers.

Plots from pretalx visualising Proposals by Submission date, Session type, Track & State

Even though it sounds cliché, there were many good proposals we couldn&apost accept immediately since the high volume and quality of proposals made it challenging to make instant decisions. We kept debating whether to place them on the waiting list.

Ultimately, we created another category for proposals that "could be accepted" allowing us to manage and organise high-quality proposals that required further deliberation.

Programme team trying to figure which talk to choose from the waiting listWhat about sponsored talks?

Each year, the conference offers sponsors with certain packages the perk of hosting a sponsored talk, meaning that some of the talk slots had to be saved for that purpose. Slots not taken were filled by proposals on the waiting list.

Is selecting the talks the end of the story?

No. After proposals are accepted/confirmed, special requirements emerge, mainly about "I’m sorry, I cannot be at the conference, can I do it online?" Which, in our opinion, is unfortunate news—not because we don’t like it, but because we have learned that remote talks are not as popular with attendees.

Even though there are some special cases that we fully understand, we noticed a few cases not being convincing enough. In those cases, we had to encourage people to give up their slot for other in-person proposals. This is a tricky process as we are limited in the total amount of remote talks possible, the specific reasons for the change, and the overall scenario for the conference.

What is needed to get accepted?

Most rejected proposals are rejected because they have a weak abstract.

We have tried many means to encourage people to ask questions and seek feedback about their proposals, and we have hosted calls providing the details of good proposals. Still, every year we get proposals that have a poorly structured, incomplete abstract, etc.

For us, a good abstract contains the following:

• Context of your talk or problem
• Definition of the problem
• Why is it important to find a solution to that problem?
• What will be discussed and what will attendees learn?
• Previous requirements or additional comments on your talk

You can also imagine a proposal like an elevator pitch. You need to describe it in a way that’s striking and motivates people to attend.

Don’t Forget About the Outline!

This year, we introduced an “outline” field for you to paste the outline of your talk, including the time you will spend on each item. This is essential to get an idea of how much you will be talking about each topic. (Hint: add up the expected times.)

The outline might sound like an obvious topic to you, but many people failed to provide a detailed one. Some even copied the abstract here, so you might understand the importance of this field as well.

Why Does It Feel Like the Same People Are Speakers Every Year?

The main reason for this is that those people followed the proper abstract structure and provided a descriptive outline. Having experience being rejected certainly helps. So we hope that after giving you detailed selection process standards, you know how to crack the selection process.

What about AI?

We discussed a few proposals that “felt AI written” and even used external tools to assess them. In the end, we didn’t have a strict ruling against people using Artificial Intelligence tools to improve their proposals.

When a proposal felt like it was AI-generated, we went deeper into the proposal and the speaker&aposs background. For example, by analysing the bio from the speaker and checking if the person was giving talks somewhere else. Most importantly, if the “speaker” was a real person.

Independently of how the Programme team feels towards AI tools, we cannot completely ignore how these tools are helping some people with structure and grammar, as well as overall assisting them in their writing process. This might change in the future, but currently, we have not written regulations against the usage of AI tools.

The 2025 Process and Final Words

As described before, the team and process can change a bit next year, but we expect the same critical aspects of a good abstract and outline to be essential to the process.

We encourage you to ask for feedback, participate in sessions teaching how to write good proposals, participate on our Speaker&aposs Mentorship programme. These can truly help you to get accepted into the conference.

Having said all this, each conference has a different selection process. Maybe the reason your proposal was not selected is due to a better proposal on the same topic, or too many similar proposals in the same track, or your proposal just did not fit this year&aposs Zeitgeist (i.e. Community Voting).

Please don’t be discouraged! We highly recommend you keep working on your proposal, tweak it, write a new one, and most importantly, try again.

Submitting Several Proposals Doesn’t Help!

We value quality over quantity and will compare your proposals against each other. This is extra work and might even give you less of a chance because of a split vote between your proposals. So submitting more than 10 proposals to get accepted is the wrong approach.

The Call for Proposals will likely be open earlier next year. We hope you can follow the recommendations in this post and get your proposal to accepted for EuroPython 2025.

And remember: Don’t be afraid to ask for feedback!

Thanks for reading! This community post is written by Cristián on behalf of EuroPython 2024 Programme team

In this episode, we worked on some core pages to round out the JourneyInbox user interface. This led us to work updating UI layout, writing copy, and doing other fundamentals for making templated pages.

pre { white-space: pre; overflow-x: auto; font-size: 80%; }

Let’s speed up some software! Our motivation: we have an image, a photo of some text from a book. We want to turn it into a 1-bit image, with just black and white, extracting the text so we can easily read it.

We’ll use an example image from scikit-image, an excellent image processing library:

from skimage.data import page import numpy as np IMAGE = page() assert IMAGE.dtype == np.uint8

Here’s what it looks like (it’s licensed under this license):

Median-based local thresholding

The task we’re trying to do—turning darker areas into black, and lighter areas into white—is called thresholding. Since the image is different in different regions, with some darker and some lighter, we’ll get the best results if we use local thresholding, where the threshold is calculated from the pixel’s neighborhood.

Simplifying somewhat, for each pixel in the image we will:

1. Calculate the median of the surrounding neighborhood.
2. Subtract a magic constant from the calculated median to calculate our local threshold.
3. If the pixel’s value is bigger than the threshold, the result is white, otherwise it’s black.

scikit-image includes an implementation of this algorithm. Here’s how we use it:

from skimage.filters import threshold_local def skimage_median_local_threshold(img, neighborhood_size, offset): threshold = threshold_local( img, block_size=neighborhood_size, method="median", offset=offset ) result = (img > threshold).astype(np.uint8) result *= 255 return result # The neighborhood size and offset value were determined "empirically", i.e. # they're manually tuning the algorithm to work well with our specific # example image. SKIMAGE_RESULT = skimage_median_local_threshold(IMAGE, 11, 10)

And here’s what the results look like:

Let’s see if we can make this faster!

Step 1. Reimplement our own version

We’re going to be using the Numba compiler, which lets us compile Python code to machine code at runtime. Here’s an initial implementation of the algorithm; it’s not quite identical to the original, for example the way edge pixels are handled, but it’s close enough for our purposes:

from numba import jit @jit def median_local_threshold1(img, neighborhood_size, offset): # Neighborhood size must be an odd number: assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) # For every pixel: for i in range(img.shape[0]): # Calculate the Y borders of the neighborhood: min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) for j in range(img.shape[1]): # Calculate the X borders of the neighborhood: min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # Calculate the median: median = np.median(img[min_y:max_y, min_x:max_x]) # Set the image to black or white, depending how it relates to # the threshold: if img[i, j] > median - offset: # White: result[i, j] = 255 else: # Black: result[i, j] = 0 return result NUMBA_RESULT1 = median_local_threshold1(IMAGE, 11, 10)

Here’s the resulting image; it looks similar enough that for our purposes:

Now we can compare the performance of the two implementations:

Code Elapsed milliseconds skimage_median_local_threshold(IMAGE, 11, 10) 76 median_local_threshold1(IMAGE, 11, 10) 87

It’s slower. But that’s OK, we’re just getting started.

Step 2: A faster implementation of the median algorithm

Calculating a median is pretty expensive, and we’re doing it for every single pixel, so let’s see if we can speed it up.

The generic median implementation Numba provides is likely to be fairly generic, since it needs to work in a wide variety of circumstances. We can hypothesize that it’s not optimized for our particular case. And even if it is, having our own implementation will allow for a second round of optimization, as we’ll see in the next step.

We’re going to implement a histogram-based median, based on the fact we’re using 8-bit images that only have a limited range of potential values. The median is the value where 50% of the pixels’ values are smaller, and 50% are bigger.

Here’s the basic algorithm for a histogram-based median:

• Each pixel’s value will go into a different bucket in the histogram; since we know our image is 8-bit, we only need 256 buckets.
• Then, we add up the size of each bucket in the histogram, from smallest to largest, until we hit 50% of the pixels we inspected.

@jit def median_local_threshold2(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) # 😎 A histogram with a bucket for each of the 8-bit values possible in # the image. We allocate this once and reuse it. histogram = np.empty((256,), dtype=np.uint32) for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # Reset the histogram to zero: histogram[:] = 0 # Populate the histogram, counting how many of each value are in # the neighborhood we're inspecting: neighborhood = img[min_y:max_y, min_x:max_x].ravel() for k in range(len(neighborhood)): histogram[neighborhood[k]] += 1 # Use the histogram to find the median; keep adding buckets until # we've hit 50% of the pixels. The corresponding bucket is the # median. half_neighborhood_size = len(neighborhood) // 2 for l in range(256): half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: break median = l if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT2 = median_local_threshold2(IMAGE, 11, 10)

Here’s the resulting image:

And here’s the performance of our new implementation:

Code Elapsed milliseconds median_local_threshold1(IMAGE, 11, 10) 86 median_local_threshold2(IMAGE, 11, 10) 18

That’s better!

Step 3: Stop recalculating the histogram from scratch

Our algorithm uses a rolling neighborhood or window over the image, calculating the median for a window around each pixel. And the neighborhood for one pixel has a significant overlap for the neighborhood of the next pixel. For example, let’s say we’re looking at a neighborhood size of 3. We might calculate the median of this area:

...... .\\\.. .\\\.. .\\\.. ...... ......

And then when process the next pixel we’ll calculate the median of this area:

...... ..///. ..///. ..///. ...... ......

If we superimpose them, we can see there’s an overlap, the X:

...... .\XX/. .\XX/. .\XX/. ...... ......

Given the histogram for the first pixel, if we remove the values marked with \ and add the ones marked with /, we’ve calculated the exact histogram for the second pixel. So for a 3×3 neighborhood, instead of processing 3 columns we process 2, a minor improvement. For a 11×11 neighborhood, we will go from processing 11 columns to 2 columns, a much more significant improvement.

Here’s what the code looks like:

@jit def median_local_threshold3(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) histogram = np.empty((256,), dtype=np.uint32) for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) # Populate histogram as if we started one pixel to the left: histogram[:] = 0 initial_neighborhood = img[min_y:max_y, 0:radius].ravel() for k in range(len(initial_neighborhood)): histogram[initial_neighborhood[k]] += 1 for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) # 😎 Instead of recalculating histogram from scratch, re-use the # previous pixel's histogram. # Substract left-most column we don't want anymore: if min_x > 0: for y in range(min_y, max_y): histogram[img[y, min_x - 1]] -= 1 # Add new right-most column: if max_x < img.shape[1]: for y in range(min_y, max_y): histogram[img[y, max_x - 1]] += 1 # Find the the median from the updated histogram: half_neighborhood_size = ((max_y - min_y) * (max_x - min_x)) // 2 for l in range(256): half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: break median = l if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT3 = median_local_threshold3(IMAGE, 11, 10)

Here’s the resulting image:

And here’s the performance of our latest code:

Code Elapsed microseconds median_local_threshold2(IMAGE, 11, 10) 17,066 median_local_threshold3(IMAGE, 11, 10) 6,386 Step #4: Adapative heuristics

Notice that a median’s definition is symmetrical:

1. The first value that is smaller than the highest 50% values.
2. Or, the first value that is larger than the lowest 50% values. We used this definition in our code above, adding up buckets from the smallest to the largest.

Depending on the distribution of values, one approach to adding up buckets to find the median may be faster than the other. For example, given a 0-255 range, if the median is going to be 10 we want to start from the smallest bucket to minimize additions. But if the median is going to be 200, we want to start from the largest bucket.

So which side we should start from? One reasonable heuristic is to look at the previous median we calculated, which most of the time will be quite similar to the new median. If the previous median was small, start from the smallest buckets; if it was large, start from the largest buckets.

@jit def median_local_threshold4(img, neighborhood_size, offset): assert neighborhood_size % 2 == 1 radius = (neighborhood_size - 1) // 2 result = np.empty(img.shape, dtype=np.uint8) histogram = np.empty((256,), dtype=np.uint32) median = 0 for i in range(img.shape[0]): min_y = max(i - radius, 0) max_y = min(i + radius + 1, img.shape[0]) histogram[:] = 0 initial_neighborhood = img[min_y:max_y, 0:radius].ravel() for k in range(len(initial_neighborhood)): histogram[initial_neighborhood[k]] += 1 for j in range(img.shape[1]): min_x = max(j - radius, 0) max_x = min(j + radius + 1, img.shape[1]) if min_x > 0: for y in range(min_y, max_y): histogram[img[y, min_x - 1]] -= 1 if max_x < img.shape[1]: for y in range(min_y, max_y): histogram[img[y, max_x - 1]] += 1 half_neighborhood_size = ((max_y - min_y) * (max_x - min_x)) // 2 # 😎 Find the the median from the updated histogram, choosing # the starting side based on the previous median; we can go from # the leftmost bucket to the rightmost bucket, or in reverse: the_range = range(256) if median < 127 else range(255, -1, -1) for l in the_range: half_neighborhood_size -= histogram[l] if half_neighborhood_size < 0: median = l break if img[i, j] > median - offset: result[i, j] = 255 else: result[i, j] = 0 return result NUMBA_RESULT4 = median_local_threshold4(IMAGE, 11, 10)

The end result is 25% faster. Since the heuristic is tied to the image contents, the performance impact will depend on the image.

Code Elapsed microseconds median_local_threshold3(IMAGE, 11, 10) 6,381 median_local_threshold4(IMAGE, 11, 10) 4,920 The big picture

Here’s a performance comparison of all the versions of the code:

Code Elapsed microseconds skimage_median_local_threshold(IMAGE, 11, 10) 76,213 median_local_threshold1(IMAGE, 11, 10) 86,494 median_local_threshold2(IMAGE, 11, 10) 17,145 median_local_threshold3(IMAGE, 11, 10) 6,398 median_local_threshold4(IMAGE, 11, 10) 4,925

Let’s go over the steps we went through:

1. Switch to a compiled language: this gives us more control.
2. Reimplement the algorithm taking advantage of constrained requirements: our median only needed to handle uint8, so a histogram was a reasonable solution.
3. Reuse previous calculations to prevent repetition: our histogram for the neighborhood of a pixel is quite similar to that of the previous pixel. This means we can reuse some of the calculations.
4. Adaptively tweak the algorithm at runtime: as we run on an actual image, we use what we’ve learned up to this point to hopefully run faster later on. The decision from which side of the histogram to start is arbirary in general. But in this specific algorithm, the overlapping pixel neighborhoods mean we can make a reasonable guess.

This process demonstrates part of why generic libraries may be slower than custom code you write for your particular use case and your particular data.

Next steps

What else can you do to speed up this algorithm? Here are some ideas:

• There may be a faster alternative to histogram-based medians.
• We’re not fully taking advantage of histogram overlap; there’s also overlap between rows.
• The cumulative sum in the histogram doesn’t benefit from instruction-level parallelism or SIMD. It’s possible that using one of those would result in faster results even if it uses more instructions.
• So far the code has only used a single CPU. Given each row is calculated independently, parallelism would probably work well if done in horizontal stripes, probably taller than one pixel so as to maximize utilization of memory caches.

Want to learn more about optimizing compiled code for Python data processing? This article is an extract from a book I’m working on; test readers are currently going through initial drafts. Aimed at Python developers, data scientists, and scientists, the book covers topics like instruction-level parallelism, memory caches, and other performance optimization techniques. Learn more and sign up to get updates here.

Read more...

I am getting increasingly frustrated by Fira Mono's lack of italic support so I am looking at alternative fonts again.

Commit Mono

This time I seem to be settling on either Commit Mono or Space Mono. For now I'm using Commit Mono because it's a little more compressed than Fira and does have a italic version. I don't like how Space Mono's parenthesis (()) is "squarish", it feels visually ambiguous with the square brackets ([]), a big no-no for my primary use case (code).

So here I am using a new font, again. It required changing a bunch of configuration files in my home directory (which is in a private repository, sorry) and Emacs configuration (thankfully that's public!).

One gotcha is I realized I didn't actually have a global font configuration in Emacs, as some Faces define their own font family, which overrides the frame defaults.

This is what it looks like, before:

Fira Mono

After:

Commit Mono

(Notice how those screenshots are not sharp? I'm surprised too. The originals look sharp on my display, I suspect this is something to do with the Wayland transition. I've tried with both grim and flameshot, for what its worth.)

They are pretty similar! Commit Mono feels a bit more vertically compressed maybe too much so, actually -- the line height feels too low. But it's heavily customizable so that's something that's relatively easy to fix, if it's really a problem. Its weight is also a little heavier and wider than Fira which I find a little distracting right now, but maybe I'll get used to it.

All characters seem properly distinguishable, although, if I'd really want to nitpick I'd say the © and ® are too different, with the latter (REGISTERED SIGN) being way too small, basically unreadable here. Since I see this sign approximately never, it probably doesn't matter at all.

I like how the ampersand (&) is more traditional, although I'll miss the exotic one Fira produced... I like how the back quotes (`, GRAVE ACCENT) drop down low, nicely aligned with the apostrophe. As I mentioned before, I like how the bar on the "f" aligns with the other top of letters, something in Fira mono that really annoys me now that I've noticed it (it's not aligned!).

A UTF-8 test file

Here's the test sheet I've made up to test various characters. I could have sworn I had a good one like this lying around somewhere but couldn't find it so here it is, I guess.

US keyboard coverage: abcdefghijklmnopqrstuvwxyz`1234567890-=[]\;',./ ABCDEFGHIJKLMNOPQRSTUVWXYZ~!@#$%^&*()_+{}|:"<>? latin1 coverage: ¡¢£¤¥¦§¨©ª«¬­®¯°±²³´µ¶·¸¹º»¼½¾¿ EURO SIGN, TRADE MARK SIGN: €™ ambiguity test: e¢coC0ODQ iI71lL!|¦ b6G&0B83 [](){}/\.…·• zs$S52Z% ´`'"‘’“”«» all characters in a sentence, uppercase: the quick fox jumps over the lazy dog THE QUICK FOX JUMPS OVER THE LAZY DOG same, in french: voix ambiguë d'un cœur qui, au zéphyr, préfère les jattes de kiwis. VOIX AMBIGUË D'UN CŒUR QUI, AU ZÉPHYR, PRÉFÈRE LES JATTES DE KIWIS. Ligatures test: -<< -< -<- <-- <--- <<- <- -> ->> --> ---> ->- >- >>- =<< =< =<= <== <=== <<= <= => =>> ==> ===> =>= >= >>= <-> <--> <---> <----> <=> <==> <===> <====> :: ::: __ <~~ </ </> /> ~~> == != /= ~= <> === !== !=== =/= =!= <: := *= *+ <* <*> *> <| <|> |> <. <.> .> +* =* =: :> (* *) /* */ [| |] {| |} ++ +++ \/ /\ |- -| <!-- <!--- Box drawing alignment tests: █ ╔══╦══╗ ┌──┬──┐ ╭──┬──╮ ╭──┬──╮ ┏━━┳━━┓ ┎┒┏┑ ╷ ╻ ┏┯┓ ┌┰┐ ▉ ╱╲╱╲╳╳╳ ║┌─╨─┐║ │╔═╧═╗│ │╒═╪═╕│ │╓─╁─╖│ ┃┌─╂─┐┃ ┗╃╄┙ ╶┼╴╺╋╸┠┼┨ ┝╋┥ ▊ ╲╱╲╱╳╳╳ ║│╲ ╱│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╿ │┃ ┍╅╆┓ ╵ ╹ ┗┷┛ └┸┘ ▋ ╱╲╱╲╳╳╳ ╠╡ ╳ ╞╣ ├╢ ╟┤ ├┼─┼─┼┤ ├╫─╂─╫┤ ┣┿╾┼╼┿┫ ┕┛┖┚ ┌┄┄┐ ╎ ┏┅┅┓ ┋ ▌ ╲╱╲╱╳╳╳ ║│╱ ╲│║ │║ ║│ ││ │ ││ │║ ┃ ║│ ┃│ ╽ │┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▍ ║└─╥─┘║ │╚═╤═╝│ │╘═╪═╛│ │╙─╀─╜│ ┃└─╂─┘┃ ░░▒▒▓▓██ ┊ ┆ ╎ ╏ ┇ ┋ ▎ ╚══╩══╝ └──┴──┘ ╰──┴──╯ ╰──┴──╯ ┗━━┻━━┛ └╌╌┘ ╎ ┗╍╍┛ ┋ ▏▁▂▃▄▅▆▇█ Dashes alignment test: HYPHEN-MINUS, MINUS SIGN, EN, EM DASH, HORIZONTAL BAR, LOW LINE -------------------------------------------------- −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−− –––––––––––––––––––––––––––––––––––––––––––––––––– —————————————————————————————————————————————————— ―――――――――――――――――――――――――――――――――――――――――――――――――― __________________________________________________

So there you have it, got completely nerd swiped by typography again. Now I can go back to writing a too-long proposal again.

Sources and inspiration for the above:

• the unicode(1) command, to lookup individual characters to disambiguate, for example, - (U+002D HYPHEN-MINUS, the minus sign next to zero on US keyboards) and − (U+2212 MINUS SIGN, a math symbol)

• searchable list of characters and their names - roughly equivalent to the unicode(1) command, but in one page, amazingly the /usr/share/unicode database doesn't have any one file like this

• bits/UTF-8-Unicode-Test-Documents - full list of UTF-8 characters

• UTF-8 encoded plain text file - nice examples of edge cases, curly quotes example and box drawing alignment test which, incidentally, showed me I needed specific faces customisation in Emacs to get the Markdown code areas to display properly, also the idea of comparing various dashes

• sample sentences in many languages - unused, "Sentences that contain all letters commonly used in a language"

• UTF-8 sampler - unused, similar

Other fonts

In my previous blog post about fonts, I had a list of alternative fonts, but it seems people are not digging through this, so I figured I would redo the list here to preempt "but have you tried Jetbrains mono" kind of comments.

My requirements are:

• no ligatures: yes, in the previous post, I wanted ligatures but I have changed my mind. after testing this, I find them distracting, confusing, and they often break the monospace nature of the display
• monospace: this is to display code
• italics: often used when writing Markdown, where I do make use of italics... Emacs falls back to underlining text when lacking italics which is hard to read
• free-ish, ultimately should be packaged in Debian

Here is the list of alternatives I have considered in the past and why I'm not using them:

• agave: recommended by tarzeau, not sure I like the lowercase a, a bit too exotic, packaged as fonts-agave

• Cascadia code: optional ligatures, multilingual, not liking the alignment, ambiguous parenthesis (look too much like square brackets), new default for Windows Terminal and Visual Studio, packaged as fonts-cascadia-code

• Fira Code: ligatures, was using Fira Mono from which it is derived, lacking italics except for forks, interestingly, Fira Code succeeds the alignment test but Fira Mono fails to show the X signs properly! packaged as fonts-firacode

• Hack: no ligatures, very similar to Fira, italics, good alternative, fails the X test in box alignment, packaged as fonts-hack

• Hermit: no ligatures, smaller, alignment issues in box drawing and dashes, packaged as fonts-hermit somehow part of cool-retro-term

• IBM Plex: irritating website, replaces Helvetica as the IBM corporate font, no ligatures by default, italics, proportional alternatives, serifs and sans, multiple languages, partial failure in box alignment test (X signs), fancy curly braces contrast perhaps too much with the rest of the font, packaged in Debian as fonts-ibm-plex

• Intel One Mono: nice legibility, no ligatures, alignment issues in box drawing, not packaged in Debian

• Iosevka: optional ligatures, italics, multilingual, good legibility, has a proportional option, serifs and sans, line height issue in box drawing, fails dash test, not in Debian

• Jetbrains Mono: (mandatory?) ligatures, good coverage, originally rumored to be not DFSG-free (Debian Free Software Guidelines) but ultimately packaged in Debian as fonts-jetbrains-mono

• Monoid: optional ligatures, feels much "thinner" than Jetbrains, not liking alignment or spacing on that one, ambiguous 2Z, problems rendering box drawing, packaged as fonts-monoid

• Mononoki: no ligatures, looks good, good alternative, suggested by the Debian fonts team as part of fonts-recommended, problems rendering box drawing, em dash bigger than en dash, packaged as fonts-mononoki

• Source Code Pro: italics, looks good, but dash metrics look whacky, not in Debian

• spleen: bitmap font, old school, spacing issue in box drawing test, packaged as fonts-spleen

• sudo: personal project, no ligatures, zero originally not dotted, relied on metrics for legibility, spacing issue in box drawing, not in Debian

So, if I get tired of Commit Mono, I might probably try, in order:

1. Hack
2. Jetbrains Mono
3. IBM Plex Mono

Iosevka, Monoki and Intel One Mono are also good options, but have alignment problems. Iosevka is particularly disappointing as the EM DASH metrics are just completely wrong (much too wide).

This was tested using the Programming fonts site which has all the above fonts, which cannot be said of Font Squirrel or Google Fonts, amazingly. Other such tools:

• Coding Font (broken in Firefox as of 2024-05-30)
• dev fonts comparator
• Font Squirrel
• Google Fonts
• Programming fonts

This episode focuses on the Harlequin application, a Python SQL IDE for your terminal written using the amazing Textual package.

I was honored to have Ted Conbeer, the creator of Harlequin, on the show to discuss his creation and the other things he does with Python.

Specifically, we focused on the following topics:

• Favorite Python packages
• Origins of Harlequin
• Why program for the terminal versus a GUI
• Lessons learned in creating the tool
• Asyncio
• and more!

Links

• Harlequin
• Textual
• BBC article on asyncio

The post Episode 42 – Harlequin – The SQL IDE for Your Terminal appeared first on Mouse Vs Python.

As of today, DEP-14 has been approved 🛫

The DEP was written and stewarded by Jake Howard. A very enthusiastic community has been active with feedback and encouragement, while the Django Steering Council gave the final inputs before its formal acceptance. The implementation of DEP-14 is expected to be a major leap forward for the “batteries included” philosophy of Django.

Whilst Django is a web framework, there's more to web applications than just the request-response lifecycle. Sending emails, communicating with external services or running complex actions should all be done outside the request-response cycle.

Django doesn't have a first-party solution for long-running tasks, however the ecosystem is filled with incredibly popular frameworks, all of which interact with Django in slightly different ways. Other frameworks such as Laravel have background workers built-in, allowing them to push tasks into the background to be processed at a later date, without requiring the end user to wait for them to occur.

Library maintainers must implement support for any possible task backend separately, should they wish to offload functionality to the background. This includes smaller libraries, but also larger meta-frameworks with their own package ecosystem such as Wagtail.

This proposal sets out to provide an interface and base implementation for long-running background tasks in Django.

• Read the full DEP-14 text
• Pull Request review

Future work

The DEP will now move on to the Implementation phase before being merged into Django itself.

If you would like to help or try it out, go have a look at django-tasks, a separate reference implementation by Jake Howard, the author of the DEP.

Jake will also be speaking about the DEP in his talk at DjangoCon Europe at DjangoCon Europe 2024 in Vigo next week.

PyCharm 2024.1.2 is here with features designed to enhance your productivity and streamline your development workflow. This update includes support for DRF viewsets and routers in the Endpoints tool window, code assistance for TypedDict and Unpack, and improved debugger performance when handling large collections.

You can download the latest version from our download page or update your current version through our free Toolbox App. 

For more details, please visit our What’s New page.

Download PyCharm 2024.1.2

Key features Support for DRF viewsets and routers in the Endpoints tool window

When working with the Django REST Framework in PyCharm, not only can you specify function-based or class-based views in the path, but you can now also specify viewsets and see the results in the Endpoints tool window. Additionally, you can map HTTP methods to viewset methods, and PyCharm will display the HTTP methods next to the relevant route, including for custom methods. Routes without @actions decorators are now displayed with the related viewset methods.

Learn more Code assistance for TypedDict and Unpack

PEP 692 made it possible to add type information for keyword arguments of different types by using TypedDict and Unpack. PyCharm allows you to use this feature confidently by providing parameter info, type checking, and code completion.

Improved debugger performance for large collections

PyCharm’s debugger now offers a smoother experience, even when very large collections are involved. You can now work on your data science projects without having to put up with high CPU loads and UI freezes.

Download PyCharm 2024.1.2

Be sure to check out our release notes to learn all of the details and ensure you don’t miss out on any new features.

We appreciate your support as we work to improve your PyCharm experience. Please report any bugs via our issue tracker so we can resolve them promptly. Connect with us on X (formerly Twitter) to share your feedback on PyCharm 2024.1.2!

This episode focuses on the Harlequin application, a Python SQL IDE for your terminal written using the amazing Textual package.

I was honored to have Ted Conbeer, the creator of Harlequin, on the show to discuss his creation and the other things he does with Python.

Specifically, we focused on the following topics:

• Favorite Python packages

• Origins of Harlequin

• Why program for the terminal versus a GUI

• Lessons learned in creating the tool

• Asyncio

• and more!

Links

• Harlequin

• Textual

• BBC article on asyncio

CRUD operations are at the heart of nearly every application you interact with. As a developer, you usually want to create data, read or retrieve data, update data, and delete data. Whether you access a database or interact with a REST API, only when all four operations are present are you able to make a complete data roundtrip in your app.

Creating, reading, updating, and deleting are so vital in software development that these methods are widely referred to as CRUD. Understanding CRUD will give you an actionable blueprint when you build applications and help you understand how the applications you use work behind the scenes. So, what exactly does CRUD mean?

Get Your Code: Click here to download the free sample code that you’ll use to learn about CRUD operations in Python.

Take the Quiz: Test your knowledge with our interactive “What Are CRUD Operations?” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

What Are CRUD Operations?

In this quiz, you'll revisit the key concepts and techniques related to CRUD operations. These operations are fundamental to any system that interacts with a database, and understanding them is crucial for effective data management.

In Short: CRUD Stands for Create, Read, Update, and Delete

CRUD operations are the cornerstone of application functionality, touching every aspect of how apps store, retrieve, and manage data. Here’s a brief overview of the four CRUD operations:

• Create: This is about adding new entries to your database. But it’s also applicable to other types of persistent storage, such as files or networked services. When you perform a create operation, you’re initiating a journey for a new piece of data within your system.
• Read: Through reading, you retrieve or view existing database entries. This operation is as basic as checking your email or reloading a website. Every piece of information you get has been received from a database, thanks to the read operation.
• Update: Updating allows you to modify the details of data already in the database. For example, when you update a profile picture or edit a chat message. Each time, there’s an update operation at work, ensuring your new data is stored in the database.
• Delete: Deleting removes existing entries from the database. Whether you’re closing an account or removing a post, delete operations ensure that unwanted or unnecessary data can be properly discarded.

CRUD operations describe the steps that data takes from creation to deletion, regardless of what programming language you use. Every time you interact with an application, you’re likely engaging in one of the four CRUD operations.

Why Are CRUD Operations Essential?

Whether you’re working on a basic task list app or a complex e-commerce platform, CRUD operations offer a universal language for designing and manipulating data models. Knowing about CRUD as a user helps you understand what’s happening behind the curtains. As a developer, understanding CRUD provides you with a structured framework for storing data in your application with persistence:

In computer science, persistence refers to the characteristic of state of a system that outlives (persists more than) the process that created it. This is achieved in practice by storing the state as data in computer data storage. (Source)

So even when a program crashes or a user disconnects, the data is safe and can be retrieved later. This also means that the order of the operations is important. You can only read, update, or delete items that were previously created.

It’s good practice to implement each CRUD operation separately in your applications. For example, when you retrieve items, then you shouldn’t update them at the same time.

Note: An exception to this rule may be when you update a “last time retrieved” value after a read operation. Although the user performs a read CRUD operation to retrieve data, you may want to trigger an update operation in the back end to keep track of a user’s retrievals. This can be handy if you want to show the last visited posts to the user.

While CRUD describes a concept that’s independent of specific programming languages, one could argue that CRUD operations are strongly connected to SQL commands and HTTP methods.

What Are CRUD Operations in SQL?

The idea of CRUD is strongly connected with databases. That’s why it’s no surprise that CRUD operations correspond almost one-to-one with SQL commands:

CRUD Operation SQL Command Create INSERT Read SELECT Update UPDATE Delete DELETE

When you create data, you’re using the INSERT command to add new records to a table. After creation, you may read data using SELECT. With a SELECT query, you’re asking the database to retrieve the specific pieces of information you need, whether it’s a single value, a set of records, or complex relationships between data points.

The update operation corresponds to the UPDATE command in SQL, which allows you to modify data. It lets you edit or change an existing item.

Lastly, the delete operation relates to the DELETE command. This is the digital equivalent of shredding a confidential document. With DELETE, you permanently remove an item from the database.

Writing CRUD Operations in Raw SQL

CRUD operations describe actions. That’s why it’s a good idea to pull up your sleeves and write some code to explore how CRUD operations translate into raw SQL commands.

In the examples below, you’ll use Python’s built-in sqlite3 package. SQLite is a convenient SQL library to try things out, as you’ll work with a single SQLite database file.

You’ll name the database birds.db. As the name suggests, you’ll use the database to store the names of birds you like. To keep the example small, you’ll only keep track of the bird names and give them an ID as a unique identifier.

Read the full article at https://realpython.com/crud-operations/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Do you want to look inside your Django request? How about all of your requests in development and see where they overlap? If that sounds useful, you should check out Kolo. It's a pretty incredible extension for your editor (VS Code at the moment, more editors to come most likely). We have Wilhelm Klopp on to tell us all about it.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Wil on Twitter</b>: <a href="https://twitter.com/wilhelmklopp" target="_blank" rel="noopener">@wilhelmklopp</a><br/> <b>Kolo</b>: <a href="https://kolo.app" target="_blank" rel="noopener">kolo.app</a><br/> <b>Kolo's info repo</b>: <a href="https://github.com/kolofordjango/kolo" target="_blank" rel="noopener">github.com</a><br/> <b>Kolo Playground</b>: <a href="https://play.kolo.app/" target="_blank" rel="noopener">play.kolo.app</a><br/> <b>Generating tests with Kolo</b>: <a href="https://blog.kolo.app/tests-no-joy.html" target="_blank" rel="noopener">kolo.app</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=NV6IfmrDY44" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/464/seeing-code-flows-and-generating-tests-with-kolo" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

Equality checks whether two objects represent the same value. Identity checks whether two variables point to the same object.

Table of contents

1. The equality operator in Python
2. The is operator in Python
3. How equality and identity work differently?
4. Inequality and non-identity operators
5. Where are identity checks used?
6. Equality vs. Identity

The equality operator in Python

Let's say we have two variables that point to two lists:

>>> a = [2, 1, 3] >>> b = [2, 1, 3, 4]

When we use the == operator to check whether these lists are equal, we'll see that they are not equal:

>>> a == b False

These lists don't have the same values right now, so they're not equal.

Let's update the first list so that these two lists do have equivalent values:

>>> a.append(4) >>> a [2, 1, 3, 4]

If we use == again, we'll see that these lists are equal now:

>>> a == b True

Python's == operator checks for equality. Two objects are equal if they represent the same data.

The is operator in Python

Python also has an is …

Read the full article: https://www.pythonmorsels.com/equality-vs-identity/

I traveled back home from PyCon US 2024 last week. This is my reflection on my time at PyCon.

Attempting to eat vegan

Since 2020, I’ve been gradually eating more plant-based and a few months ago I decided to take PyCon as an opportunity to attempt exclusively vegan eating outside my own home. As I noted on Mastodon, it was a challenge and I failed every day at least once but I found the experience worthwhile. Our food system is very dairy-oriented.

Staying hydrated and fed

One of the first things I did before heading to the convention center was walk to Target and buy snacks and drinks. When at PyCon, I prefer to spend 30 minutes and $20 to have a backup plan for last minute hydration and calories (even if not the greatest calories). I never quite know when I might sleep through breakfast, find lunch lacking, or wish I’d eaten more dinner.

A tutorial, an orientation, a lightning talk, and open spaces

My responsibilities at PyCon this year included teaching a tutorial and helping run the Newcomer’s Orientation with Kojo and Sumana.

Yngve and Marie offered to act as teaching assistants during my tutorial and I was very grateful for their help! Rodrigo and Krishna also offered to TA just before my tutorial started and I was extra grateful to have even more help than I’d expected. The attendees were mostly better prepared than I expected they would be, which was also great. It’s always great to spend less time on setup and more time exploring Python together.

The newcomer’s orientation the next day went well. We kept it fairly brief and were able to address about 10 minutes of audience questions before the opening reception started.

Once my PyCon responsibilities completed, I invented a few more (light) responsibilities for myself. 😅 I signed up to give a lightning talk on how to give a lightning talk. They slotted it as the first talk of the first lightning talk session on Friday night. I kept this talk pretty much the same as the one I presented DjangoCon 2016. I could have made the transitions fancier, but I decided to embrace the idea of simplicity with the hope that audience members might think “look if that first speaker can give such a simple and succinct presentation, maybe I can too.”

On Saturday I ran an open space on Python Learning. Some of you showed up because you’re on my mailing list or you’re paying Python Morsels subscribers. Many folks showed up because the topic was interesting, either as a learner or as a teacher. I really enjoyed the round-table-style conversation we had.

I also ran a Cabo Card game open space during lunch on Sunday on the 4th floor rooftop. Cabo is my usual conference ice breaker game and I played it at least a few nights in The Westin lobby as well.

Seeing conference friends, old and new

For me, PyCon is largely about having conversations. The talks and tutorials are great for starting me thinking about an idea. The hallway track, open spaces, and meals are great for continuing conversations about those ideas (or other ideas).

My first morning in Pittsburgh, I chatted with Naomi Ceder and Reuven Lerner. I’m glad I ran into them before the conference kicked off because (as often happens at PyCon) I only very briefly saw either of them during the rest of PyCon!

After my tutorial that afternoon, I did dinner with Marie, Yngve, and Rodrigo at Rosewater Mediterranean (good vegan options, assuming you enjoy falafel and various sauces). As sometimes happens at PyCon, another PyCon attendee, Sachin, joined our table because we noticed him eating on his own at a table near us and invited him to join us.

On Saturday, Melanie, David, Jay, and I had a sort of mini San Diego Python study group reunion dinner before inviting folks to join us for Cabo and Knucklebones one night. The 4 of us originally met each other (along with Carol and other wonderful Python folks) at the San Diego Python study group about 10 years ago.

I had some wonderful conversations about ways to improve the Python documentation over dinner (at Nicky’s Thai) on Sunday night with so many docs-concerned folks who I highly respect. I’m really excited that Python has the documentation editorial board and I’m hopeful that that board, with the help of many others community members, will usher in big improvements to the documentation in the coming years.

I also met a number of Internet acquaintances IRL for the first time at PyCon. I met Tereza and Jessica, who I know from our work in the PSF Code of Conduct workgroup. I met Steve Lott, who I originally knew as a prolific question-answerer. I also met Hugo, a CPython core dev, the Python 3.14 & 3.15 release manager, and a social media user (which is how I’ve primarily interacted with him because the Internet is occasionally lovely). I was also very excited to meet many Python Morsels members as well as folks who know me through my weekly Python tips newsletter.

I was grateful to chat with Hynek and Al about creating talks, YouTube videos, and other online content. I also enjoyed chatting with Glyph a bit about our experiences consulting and training and (in hindsight) wished I’d planned an open space for either consultants or trainers, both of which have been held at PyCon before but it just takes someone to stick it on the open space board.

Many folks I only saw very briefly (I said a quick hi and bye to Andrew over lunch during the sprints) and some I didn’t see at all (Frank was at PyCon but we never ran into each other). Some I essentially saw through playing a few rounds of Cabo (Thomas and Ethan among many others). We also ran into at least 4 other PyCon attendees in the airport on Tuesday afternoon, including Bob and Julian, who it’s always a pleasure to see.

A Mastodon-oriented PyCon

On Thursday night I had the feeling that the number of Mastodon posts I saw on the #PyConUS hashtag was greater than the number of Twitter posts. I (very unscientifically) counted up the number of posts I was seeing on each and found that my perception was correct: Mastodon seemed to slightly overtake Twitter at PyCon this year.

Over dinner on Wednesday, I tried to convince Marie, Yngve, and Rodrigo to get Mastodon accounts just to follow the hashtag during PyCon. I succeeded: Marie and Yngve and Rodrigo!

Mastodon will never be the social media platform. Its decentralized nature is too much of a barrier for many folks. However, it does seem to be used by enough somewhat nerdy Python folks to now be one the most used social media platform for PyCon posting.

The talks

I ended up spending little time in the talks during PyCon. This wasn’t on purpose. I just happened to attend many open spaces, take personal breaks, and end up in hallway conversations often. I did see many of the lightning talks live, as well as Jay, Simon, and Sumana’s keynotes (all of them were exceptional) and the opening and closing remarks. I also watched a few talks from my hotel room while taking breaks.

While I’m often a bit light on my talk load at PyCon, I do recommend folks attend a good handful of live talks during PyCon, as Jon and others recommend. I wish I had seen more talks live. I also wish I had attended a few open spaces that I missed.

At any one time, I know that I’m always missing about 90% of what’s scheduled during PyCon (if you include the talks and the open spaces). That’s assuming I don’t ditch the conference entirely for a few hours and walk across a bridge or ride a funicular (neither of which I did, as I stuck around the venue the whole time this year). I am glad I saw, did, and talked about everything I did, but there’s always something I wish I’d seen/done!

The sprints

Thanks to the documentation dinner, I had a couple documentation-related ideas in mind on the first day of sprints. But I’m also really excited about the new Python REPL coming in Python 3.13 (in case you can’t tell from how much I talk about it), so I sprinted on that instead. Łukasz assigned me the task of researching keyboard shortcuts that the new REPL is missing (compared to the current one on Linux and Mac) so I spent some time researching that. I got to see the REPL running on Anthony’s laptop on Windows and I am so excited that Windows support will be included before 3.13.0 lands! 🎉

Partly inspired by Carol Willing’s PyCon preview message, I also thanked Pablo, Łukasz, and Lysandros in-person for all their work on the new Python REPL. 🤗

Until next year

I’ll be keynoting at PyOhio this year.

Besides PyOhio, I’m not sure whether I’ll make it to another conference until PyCon US next year. I’d love to attend all of them, but I do have work and personal goals that need accomplishing too!

I hope to see you at PyCon US 2025! In the meantime, if you’re wishing we’d exchanged contact details or met in-person, please feel free to stay in touch through Mastodon, LinkedIn, my weekly emails, YouTube, or Twitter.