Planet Python

Subscribe to Planet Python feed
Planet Python - http://planetpython.org/
Updated: 8 hours 2 min ago

PyBites: Leveraging typing.Protocol: Faster Error Detection And Beyond Inheritance

Fri, 2024-02-09 10:52
Introduction

Two weeks ago I wrote an article about ABCs and interface enforcement. Shortly after I learned that you can do this as well with protocols.

Python 3.8 introduced quite a groundbreaking feature that further advanced the language’s capabilities in type checking: the typing.Protocol which allows Python developers to define and enforce interface contracts in their code.

Unlike traditional dynamic typing, which Python is well-known for, typing.Protocol brings a layer of static type checking that allows for more explicit, readable, and robust code by defining expected behaviors and structures.

This feature not only enhances code quality and developer productivity but also aligns Python more closely with the benefits seen in statically-typed languages, all while maintaining Python’s expressive and dynamic nature.

In this article, we delve into the potential of typing.Protocol through the lens of refactoring the Pybites Search feature, showcasing the immediate benefits of early error detection.

Before we dive in, in case you missed last article, let’s first clarify what “enforcing an interface” actually means …

What is interface enforcement?

Interface enforcement in object-oriented programming (OOP) is a design principle where you define a contract or a blueprint for what methods and properties should be present in a class.

This contract ensures that all classes implementing the interface adhere to a specific structure, promoting consistency and predictability across your codebase.

For example, if you define an interface for a Search functionality, every class implementing this interface must provide the specific methods defined, such as match_content.

This approach allows programmers to design components that can interact seamlessly, knowing the expected behaviors and data types, thereby reducing errors and improving code maintainability.

The ABC / Abstract Method Way

Remember how we implemented PyBites Search originally using Abstract Base Classes (ABCs) to enforce a consistent interface for search functionality:

from abc import ABC, abstractmethod class PybitesSearch(ABC): @abstractmethod def match_content(self, search: str) -> list: # >= 3.9 you can use list over typing.List ...

While ABCs served us well here, the interface enforcement only happens at runtime.

Static type checking vs runtime checks

Static type checking offers several advantages over runtime checks, primarily by shifting error detection earlier in the development process.

By identifying type mismatches and potential bugs during coding or at compile time, developers can address issues before they manifest in a running application, leading to more stable and reliable software.

Static type checking also enhances code readability and documentation, as type annotations provide clear expectations for variables, arguments, and return types.

This explicitness improves developer collaboration and facilitates easier maintenance and debugging of the codebase.

Moreover, static type checking can lead to performance optimizations since the interpreter or compiler can make certain assumptions about the types, potentially streamlining execution.

If you are new to Python type hints, check out our article.

Detecting Errors Earlier

Although ABCs and abstract methods enforce an interface, one limitation is that errors can slip through into production.

What if we can detect interface contract breaches earlier?

Enter typing.Protocol! This new feature allows for static type checking, ensuring that not only our class implements the enforced interface, it also does it with the right method signatures.

Here’s how we could refactor Pybites Search to use this (code):

from typing import Protocol class PybitesSearchProtocol(Protocol): def match_content(self, search: str) -> list[str]: """Implement in subclass to search Pybites content""" ... class CompleteSearch: def match_content(self, search: str) -> list[str]: # Implementation of search method return ["result1", "result2"] class IncompleteSearch: # Notice that we don't implement match_content here pass def perform_search(search_tool: PybitesSearchProtocol, query: str) -> None: results = search_tool.match_content(query) print(results) # Static type checking will pass for CompleteSearch perform_search(CompleteSearch(), "Python") # Static type checking will fail for IncompleteSearch perform_search(IncompleteSearch(), "Python")

In this refactored version, PybitesSearchProtocol defines an interface using typing.Protocol.

Unlike ABCs, classes like CompleteSearch and IncompleteSearch don’t need to inherit from PybitesSearchProtocol to be considered compliant.

Instead, compliance is determined by whether a class implements the required methods, in this case, match_content.

When the perform_search function is called with an instance of CompleteSearch, static type checkers like mypy will confirm that CompleteSearch satisfies the PybitesSearchProtocol because it implements match_content.

However, passing an instance of IncompleteSearch, which lacks the match_content method, will result in a type-checking error.

This approach, using typing.Protocol, offers a more flexible way of enforcing interfaces, particularly useful in scenarios where rigid class hierarchies are undesirable or unnecessary.

It aligns well with Python’s dynamic nature while still leveraging the benefits of static type checking to ensure code correctness.

Static Type Checking

Now let’s see how this protocol is enforced as I try to implement.

Step 1: Missing Method Implementation

Initially, when a class like IncompleteSearch does not implement the required match_content method.

class IncompleteSearch: pass

Failing to implement a required method could lead to AttributeError exceptions at runtime, disrupting user experience and potentially halting application functionality.

Running mypy, it catches this error:

$ mypy script.py script.py:24: error: Argument 1 to "perform_search" has incompatible type "IncompleteSearch"; expected "PybitesSearchProtocol" [arg-type] Found 1 error in 1 file (checked 1 source file) Step 2: Incorrect Method Signature

Next we implement the method but with an incorrect signature:

class IncompleteSearch: def match_content(self): pass

Implementing a method with the wrong signature may result in a TypeError that is only caught when the specific code path is executed, risking data inconsistencies or application crashes in production environments.

Mypy catches this too:

$ mypy script.py script.py:25: error: Argument 1 to "perform_search" has incompatible type "IncompleteSearch"; expected "PybitesSearchProtocol" [arg-type] script.py:25: note: Following member(s) of "IncompleteSearch" have conflicts: script.py:25: note: Expected: script.py:25: note: def match_content(self, search: str) -> list[str] script.py:25: note: Got: script.py:25: note: def match_content(self) -> Any Found 1 error in 1 file (checked 1 source file) Step 3: Incompatible Return Type

Finally, we correct the method signature but we return an incorrect type.

class IncompleteSearch: def match_content(self, search: str) -> list[str]: return (1, 2)

Returning incorrect types from methods can lead to subtle bugs, such as incorrect data processing or application logic failures, which may not be immediately apparent and could lead to significant issues over time.

Once again, mypy flags it:

$ mypy script.py script.py:15: error: Incompatible return value type (got "tuple[int, int]", expected "list[str]") [return-value] Found 1 error in 1 file (checked 1 source file)

Incorporating typing.Protocol into our Python codebase not only facilitates earlier error detection and circumvents the inheritance requirement but also subtly enforces the Liskov Substitution Principle (LSP).

By ensuring that objects are replaceable with instances of their subtypes without altering the correctness of the program, typing.Protocol aids in creating more reliable and maintainable code.

This alignment with SOLID principles highlights the broader impact of adopting modern type checking in Python, enhancing both code quality and developer productivity.

Enhancing Protocols with Runtime Checkability

To bridge the gap between static type checking and runtime flexibility, Python’s typing module includes the @typing.runtime_checkable decorator.

This feature allows protocols to be used in runtime type checks, similar to abstract base classes.

Applying @typing.runtime_checkable to a protocol makes it possible to use isinstance() and issubclass() checks, offering a layer of dynamic validation that complements static type checking.

However, it’s important to note the limitation: while runtime checkable protocols can verify the presence of required methods at runtime, they do not validate method signatures or return types as static type checking does.

This offers a practical yet limited approach to enforcing interface contracts dynamically, providing developers with additional tools to ensure their code’s correctness and robustness.

One practical use case for @typing.runtime_checkable is in developing plugins or extensions. When you’re designing an API where third-party developers can provide plugin implementations, runtime checks can be used to verify that an object passed to your API at runtime correctly implements the expected interface.

This can be especially useful in dynamically loaded modules or plugins where static type checks might not catch mismatches. For example, before invoking plugin-specific methods, you might use isinstance(plugin, PluginProtocol) to ensure that the plugin adheres to your defined protocol, enhancing reliability and reducing the risk of runtime errors.

Structural Subtyping and Embracing Python’s Duck Typing

The concept of structural subtyping, formalized through typing.Protocol, is a testament to Python’s commitment to flexibility and the “duck typing” philosophy.

Structural subtyping allows a class to be considered a subtype of another if it meets certain criteria, specifically if it has all the required methods and properties, irrespective of the inheritance relationship.

This approach enables developers to design more generic, reusable components that adhere to specified interfaces without being bound by a strict class hierarchy.

It essentially allows objects to be used based on their capabilities rather than their specific types, echoing the Pythonic saying, “If it looks like a duck and quacks like a duck, it’s a duck.”

By leveraging typing.Protocol, Python developers can enjoy the benefits of static type checking while maintaining the language’s dynamic, expressive nature, ensuring code is both flexible and type-safe.

Further reading
  • Python typing Module Documentation: Dive into the official Python documentation for an in-depth look at type hints. This guide covers everything from basic annotations to advanced features like typing.Protocol, equipping you with the knowledge to write clearer and more maintainable Python code.
  • PEP 544 – Protocols: Structural subtyping (static duck typing): Explore the proposal that introduced typing.Protocol to Python. This document provides valuable context on the motivation behind protocols, detailed examples, and insights into how they enhance Python’s type system, making it an essential read for developers interested in type checking and Python’s design philosophy.
  • Building Implicit Interfaces in Python with Protocol Classes: This article offers a practical approach to using typing.Protocol for defining and implementing interfaces in Python. It’s perfect for readers looking for actionable advice and examples on how to leverage protocols in their projects.
  • Robust Python: Patrick Viafore’s book is a treasure trove of information on Python’s typing system, including a dedicated chapter on typing.Protocol. It’s a great resource for those seeking to deepen their understanding of Python’s type hints and how to use them effectively to write robust, error-resistant code.
Conclusion

Adopting typing.Protocol in the PyBites Search feature showcased not just an alternative to ABCs and abstract methods, but also illuminated a path toward more proactive error management and a deeper alignment with Python’s dynamic and flexible design ethos.

By embracing typing.Protocol, we not only enhance our code’s reliability through early error detection but also open the door to more robust design patterns that leverage Python’s strengths in readability and expressiveness.

The exploration of runtime checkability with @typing.runtime_checkable and the discussion around structural subtyping and duck typing further underscore the versatile and powerful nature of Python’s type system.

These features collectively foster a development environment where code is not just correct, but also elegantly aligned with the principles of modern software design.

As Python continues to evolve, tools like typing.Protocol are invaluable for developers aiming to write high-quality, maintainable code that adheres to both the letter and spirit of Python’s design philosophy.

Categories: FLOSS Project Planets

TechBeamers Python: 20 Practical Pandas Tips and Tricks for Python

Fri, 2024-02-09 10:51

Welcome to this Python tutorial including Pandas tips and tricks! In this guide, we’ll share 20 practical techniques to make your data tasks easier and improve your Python data analysis. Whether you’re new or experienced, these tips will help you become more efficient in using Pandas for data manipulation. Let’s dive in and explore the […]

The post 20 Practical Pandas Tips and Tricks for Python appeared first on TechBeamers.

Categories: FLOSS Project Planets

Real Python: The Real Python Podcast – Episode #191: Focusing on Data Science & Less on Engineering and Dependencies

Fri, 2024-02-09 07:00

How do you manage the dependencies of a large-scale data science project? How do you migrate that project from a laptop to cloud infrastructure or utilize GPUs and multiple instances in parallel? This week on the show, Savin Goyal returns to discuss the updates to the open-source framework Metaflow.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Python People: Nikita Karamov - Django project maintainer, from Russia to Germany

Thu, 2024-02-08 14:08

Nikita Karamov is a Python developer and maintainer on various open source Python projects.

Some topics covered:

  • Notes on university education in programming and engineering vs theory
  • Jazzband for maintaining Django projects
  • Contributing to open source makes you a better programmer
  • Moving from Russia to Germany during college
  • Cultural differences between Russia, Germany, and Oregon
  • The nice lack of drama in the Python community
  • A lack of universities teaching Python for web development


Links from the show


The Complete pytest Course

★ Support this podcast on Patreon ★ <p>Nikita Karamov is a Python developer and maintainer on various open source Python projects.</p><p>Some topics covered:</p><ul><li>Notes on university education in programming and engineering vs theory</li><li>Jazzband for maintaining Django projects</li><li>Contributing to open source makes you a better programmer</li><li>Moving from Russia to Germany during college</li><li>Cultural differences between Russia, Germany, and Oregon</li><li>The nice lack of drama in the Python community</li><li>A lack of universities teaching Python for web development</li></ul><p><br>Links from the show</p><ul><li><a href="https://jazzband.co">Jazzband</a></li><li><a href="https://jazzband.co/projects/django-simple-menu">django-simple-menu</a></li></ul> <br><p><strong>The Complete pytest Course</strong></p><ul><li>Level up your testing skills and save time during coding and maintenance.</li><li>Check out <a href="https://courses.pythontest.com/p/complete-pytest-course">courses.pythontest.com</a></li></ul> <strong> <a href="https://www.patreon.com/PythonPeople" rel="payment" title="★ Support this podcast on Patreon ★">★ Support this podcast on Patreon ★</a> </strong>
Categories: FLOSS Project Planets

PyCharm: PyCharm 2024.1 EAP 4: Sticky Lines, and More

Thu, 2024-02-08 12:44

The Early Access Program for PyCharm 2024.1 continues with our latest build where you can preview new features, including convenient sticky lines in the editor, and more.

You can download the new version from our website, update directly from the IDE or via the free Toolbox App, or use snaps for Ubuntu.

Download PyCharm 2024.1 EAP

Learn about the key highlights below.

User experience Sticky lines in the editor

To simplify working with large files and exploring new codebases, we’ve introduced sticky lines in the editor. This feature keeps key structural elements, like the beginnings of classes or methods, pinned to the top of the editor as you scroll. This way, scopes always remain in view, and you can promptly navigate through the code by clicking on a pinned line.

As of PyCharm 2024.1, this feature will be enabled by default. You can disable it via a checkbox in Settings/Preferences | Editor | General | Appearance, where you can also set the maximum number of pinned lines.

Version control systems Option to display review branch changes in a Git Log tab

IntelliJ IDEA 2024.1 EAP 4 streamlines the code review workflow by offering a focused view of branch-related changes. For GitHub, GitLab, and Space, it is now possible to see changes in a certain branch in a separate Log tab within the Git tool window. To do so, click on the branch name in the Pull Requests tool window and pick Show in Git Log from the menu.

These are the most notable improvements in the latest PyCharm 2024.1 EAP build. Explore the full list of changes in the release notes, and stay tuned for more updates next week.

We encourage you to share your opinion about the newest additions in the comments below or by contacting us on X (formerly Twitter). If you spot a bug, please report it via our issue tracker.

Categories: FLOSS Project Planets

TechBeamers Python: Python’s Map() and List Comprehension: Tips and Best Practices

Thu, 2024-02-08 11:20

Welcome to this tutorial where we will explore Python map() and List Comprehension best practices and share some cool coding tips. These techniques, when mastered, can make your code cleaner, more concise, and efficient. In this guide, we’ll explore these concepts from the ground up, using simple and practical examples. Let’s begin mastering the map […]

The post Python’s Map() and List Comprehension: Tips and Best Practices appeared first on TechBeamers.

Categories: FLOSS Project Planets

TechBeamers Python: 20 Practical Python Data Analysis Tips and Tricks

Thu, 2024-02-08 09:00

Hey, welcome! Today, we’re talking about practical Python data analysis tips and tricks. With Pandas and NumPy, we’ll tidy up data, spot trends, and do some smart analysis. It’s all about making your data work easier and getting cool insights. Let’s dive in to optimize your data analysis tasks. Practical Python Tips for Solid Data […]

The post 20 Practical Python Data Analysis Tips and Tricks appeared first on TechBeamers.

Categories: FLOSS Project Planets

Python Software Foundation: Software Bill-of-Materials documents are now available for CPython

Thu, 2024-02-08 05:53

Our Security Developer-in-Residence, Seth Larson, has been working to improve the management of vulnerabilities for Python users. Seth has championed progress on this goal in a variety of areas:

With the release of CPython 3.12.2, the next step of the Python Software Foundation’s vulnerability management strategy is now available in the form of Software Bill-of-Materials (SBOM) documents for CPython source releases. The documents are available for download in their own column labeled “SBOM” in the “Files” table on the release page. User documentation and a getting started guide for CPython SBOMs is available on python.org.

These documents are relatively new but have been tested with multiple tools that accept SPDX SBOM documents. Please report any feedback on the SBOM to the CPython issue tracker.

What is a Software Bill-of-Materials (SBOM)?

Software Bill-of-Materials are machine-readable documents using an ecosystem-independent format like SPDX or CycloneDX to describe what a piece of software is made of and how each component within the software relates to other components. There are multiple use-cases for SBOMs, but for CPython we primarily focused on software supply chain and vulnerability management.

Many vulnerability scanning tools support passing an SBOM document as input to provide a comprehensive scan for software vulnerabilities without needing to rely on fallible software discovery. This means there’s less chances for vulnerabilities to be missed by scanners.

There are existing tools for automatically creating SBOMs for software, but SBOMs which aren’t accurate are sometimes more dangerous than having no SBOM due to causing a false sense of security. This is especially true for complex pieces of software or projects which exist outside of package ecosystems, both of which apply to CPython and make generating an SBOM difficult. For this reason the content of CPython SBOMs is curated by hand on first pass to ensure accuracy and completeness and then automated to track updates as the software changes.

SBOM documents are becoming a requirement for compliance in multiple areas and industries. In order to meet those requirements we are providing a comprehensive and accurate SBOM for CPython that will provide assurance for Python users.

What is included in CPython SBOMs?

CPython SBOMs use the SPDX SBOM standard. SBOM documents include a description of the contained software, including all of its dependencies. Information in CPython SBOMs includes:

  • Names and versions of all software components
  • Software identifiers (like CPE and Package URLs)
  • Download URLs for source code with checksums
  • File names and content checksums
  • Dependency relationships between each component

CPython SBOMs satisfy the requirements listed in the NTIA Minimum Elements for a Software Bill of Materials. Software identifiers can be used for correlating software in use to vulnerability databases like the CVE database and Open Source Vulnerability database, typically done automatically using vulnerability scanning tools.

What isn’t included in CPython SBOMs?

Keep in mind that software libraries that you supply yourself to compile CPython, such as OpenSSL and zlib, are not included in the SBOMs for source artifacts.

This is due to these libraries not being included in source artifacts, so CPython users have a choice of which version and sources to use for these third-party libraries. Folks who are compiling CPython from source are responsible for tracking their own dependencies either in a separate SBOM document or by appending new entries to your local CPython SBOM.

CPython’s SBOMs don’t include licensing information for dependencies. See the CPython licensing page for licensing information.

What is coming next for CPython SBOMs?

This is only the beginning for CPython SBOMs, as mentioned above there are only SBOM documents published for source releases today. The CPython release managers also publish binary installers for Windows and macOS on a variety of distribution channels. These artifacts will need their own SBOM documents as they are compiled with software that’s typically not available on those platforms (e.g. OpenSSL).

There’s also more infrastructure needed to reduce noise and churn for Python users and Python Security Response Team members alike. Vulnerability EXchange (VEX) statements are a set of standards which allows software producers to signal to user tooling whether a piece of software in use is affected by a vulnerability, even for vulnerabilities affecting dependencies. This is an area of active development and is being explored alongside the OpenSSF Security Tooling Working Group.

The Security Developer-in-Residence role and this work is funded by a substantial investment from the OpenSSF Alpha-Omega Project. Thanks to Alpha-Omega for their support in improving the security posture of the entire Python ecosystem.The OpenSSF is a non-profit cross-industry collaboration that unifies security initiatives and brings together leaders to improve the security of open source software by building a broader community, targeted initiatives, and best practices.

Categories: FLOSS Project Planets

Talk Python to Me: #448: Full-Time Open Source Devs Panel

Thu, 2024-02-08 03:00
So you've created a Python-based open source project and it's started to take off. You're getting contributors, lots of buzz in the podcast space, and more. But you have that day job working on Java. How do you make the transition from popular hobby project to full time job? After all, you are giving away your open source project for free, right? Well, on this episode, I have put together an amazing panel of guests who all have done exactly this: Turned their project into full time work and even companies in some cases. We have Samuel Colvin, Gina Häußge, Sebastián Ramírez, Charlie Marsh, Will McGugan and Eric Holscher on to share their stories.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/basedash'>Basedash</a><br> <a href='https://talkpython.fm/sentry-monorepo'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Will McGugan</b>: <a href="https://twitter.com/willmcgugan" target="_blank" rel="noopener">@willmcgugan</a><br/> <b>Charlie Marsh</b>: <a href="https://hachyderm.io/@charliermarsh" target="_blank" rel="noopener">@charliermarsh@hachyderm</a><br/> <b>Sebastián Ramírez</b>: <a href="https://twitter.com/tiangolo" target="_blank" rel="noopener">@tiangolo</a><br/> <b>Samuel Colvin</b>: <a href="https://twitter.com/samuel_colvin" target="_blank" rel="noopener">@samuel_colvin</a><br/> <b>Gina on Mastodon</b>: <a href="https://chaos.social/@foosel" target="_blank" rel="noopener">chaos.social/@foosel</a><br/> <b>Eric Holscher</b>: <a href="https://twitter.com/ericholscher" target="_blank" rel="noopener">@ericholscher</a><br/> <br/> <b>Pydantic</b>: <a href="https://pydantic.dev" target="_blank" rel="noopener">pydantic.dev</a><br/> <b>Astral (makes of Ruff)</b>: <a href="https://astral.sh" target="_blank" rel="noopener">astral.sh</a><br/> <b>Octoprint</b>: <a href="https://octoprint.org" target="_blank" rel="noopener">octoprint.org</a><br/> <b>Read the Docs</b>: <a href="https://about.readthedocs.com/" target="_blank" rel="noopener">readthedocs.com</a><br/> <b>FastAPI</b>: <a href="https://fastapi.tiangolo.com" target="_blank" rel="noopener">fastapi.tiangolo.com</a><br/> <b>Textual (makes of Rich)</b>: <a href="https://www.textualize.io" target="_blank" rel="noopener">textualize.io</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=HV1LKitAr44" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/448/full-time-open-source-devs-panel" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>
Categories: FLOSS Project Planets

Pages