FLOSS Project Planets
Glyph Lefkowitz: Annotated At Runtime
PEP 0593 added the ability to add arbitrary user-defined metadata to type annotations in Python.
At type-check time, such annotations are… inert. They don’t do anything. Annotated[int, X] just means int to the type-checker, regardless of the value of X. So the entire purpose of Annotated is to provide a run-time API to consume metadata, which integrates with the type checker syntactically, but does not otherwise disturb it.
Yet, the documentation for this central purpose seems, while not exactly absent, oddly incomplete.
The PEP itself simply says:
A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance()).
But it’s not clear where “the annotations” are, given that the PEP’s entire “consuming annotations” section does not even mention the __metadata__ attribute where the annotation’s arguments go, which was only even added to CPython’s documentation. Its list of examples just show the repr() of the relevant type.
There’s also a bit of an open question of what, exactly, we are supposed to isinstance()-ing here. If we want to find arguments to Annotated, presumably we need to be able to detect if an annotation is an Annotated. But isinstance(Annotated[int, "hello"], Annotated) is both False at runtime, and also a type-checking error, that looks like this:
1Argument 2 to "isinstance" has incompatible type "<typing special form>"; expected "_ClassInfo"The actual type of these objects, typing._AnnotatedAlias, does not seem to have a publicly available or documented alias, so that seems like the wrong route too.
Now, it certainly works to escape-hatch your way out of all of this with an Any, build some version-specific special-case hacks to dig around in the relevant namespaces, access __metadata__ and call it a day. But this solution is … unsatisfying.
What are you looking for?Upon encountering these quirks, it is understandable to want to simply ask the question “is this annotation that I’m looking at an Annotated?” and to be frustrated that it seems so obscure to straightforwardly get an answer to that question without disabling all type-checking in your meta-programming code.
However, I think that this is a slightly misframing of the problem. Code that is inspecting parameters for an annotation is going to do something with that annotation, which means that it must necessarily be looking for a specific set of annotations. Therefore the thing we want to pass to isinstance is not some obscure part of the annotations’ internals, but the actual interesting annotation type from your framework or application.
When consuming an Annotated parameter, there are 3 things you probably want to know:
- What was the parameter itself? (type: The type you passed in.)
- What was the name of the annotated object (i.e.: the parameter name, the attribute name) being passed the parameter? (type: str)
- What was the actual type being annotated? (type: type)
And the things that we have are the type of the Annotated we’re querying for, and the object with annotations we are interrogating. So that gives us this function signature:
1 2 3 4 5def annotated_by( annotated: object, kind: type[T], ) -> Iterable[tuple[str, T, type]]: ...To extract this information, all we need are get_args and get_type_hints; no need for __metadata__ or get_origin or any other metaprogramming. Here’s a recipe:
1 2 3 4 5 6 7 8 9 10 11 12def annotated_by( annotated: object, kind: type[T], ) -> Iterable[tuple[str, T, type]]: for k, v in get_type_hints(annotated, include_extras=True).items(): all_args = get_args(v) if not all_args: continue actual, *rest = all_args for arg in rest: if isinstance(arg, kind): yield k, arg, actualIt might seem a little odd to be blindly assuming that get_args(...)[0] will always be the relevant type, when that is not true of unions or generics. Note, however, that we are only yielding results when we have found the instance type in the argument list; our arbitrary user-defined instance isn’t valid as a type annotation argument in any other context. It can’t be part of a Union or a Generic, so we can rely on it to be an Annotated argument, and from there, we can make that assumption about the format of get_args(...).
This can give us back the annotations that we’re looking for in a handy format that’s easy to consume. Here’s a quick example of how you might use it:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15@dataclass class AnAnnotation: name: str def a_function( a: str, b: Annotated[int, AnAnnotation("b")], c: Annotated[float, AnAnnotation("c")], ) -> None: ... print(list(annotated_by(a_function, AnAnnotation))) # [('b', AnAnnotation(name='b'), <class 'int'>), # ('c', AnAnnotation(name='c'), <class 'float'>)] AcknowledgmentsThank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support me on Patreon as well! I am also available for consulting work if you think your organization could benefit from expertise on topics like “how do I do Python metaprogramming, but, like, not super janky”.
James Bennett: Use "pip install" safely
This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post for an introduction.
Managing dependencies should be boringLast year I wrote a post about “boring” dependency management in Python, where I advocated a setup based entirely around standard Python packaging tools, in that …
Reproducible Builds (diffoscope): diffoscope 253 released
The diffoscope maintainers are pleased to announce the release of diffoscope version 253. This version includes the following changes:
* Improve DOS/MBR extraction by adding support for 7z. (Closes: reproducible-builds/diffoscope#333) * Process objdump symbol comment filter inputs as the Python "bytes" type (and not str). (Closes: reproducible-builds/diffoscope#358) * Add a missing RequiredToolNotFound import. * Update copyright years.You find out more by visiting the project homepage.
Python Engineering at Microsoft: Python in Visual Studio Code – December 2023 Release
We’re excited to announce the December 2023 release of the Python and Jupyter extensions for Visual Studio Code!
This release includes the following announcements:
- Configurable debugging options added to Run button menu
- Show Type Hierarchy with Pylance
- Deactivate command support for automatically activated virtual environments in the terminal
- Setting to turn REPL Smart Send on/off and a message when it is unsupported
If you’re interested, you can check the full list of improvements in our changelogs for the Python, Jupyter and Pylance extensions.
Configurable debugging options added to Run button menuThe Python Debugger extension now has configurable debug options under the Run button menu. When you select Python Debugger: Debug using launch.json and there is an existing launch.json in your workspace, it shows all available debug configurations you can pick to start the debugger. In the case you do not have an existing launch.json, you will be prompted to select a debug configuration template to create a launch.json file for your Python application, and then can run your application using this configuration.
Show Type Hierarchy with PylanceYou can now more conveniently explore and navigate through your Python projects’ types relationships when using Pylance. This can be helpful when working with large codebases with complex type relationships.
When you right-click on a symbol, you can select Show Type Hierarchy to open the type hierarchy view. From there you can navigate through the symbol’s subtypes as well as super-types.
Deactivate command support for automatically activated virtual environments in the terminalThe Python extension has a new activation mechanism that activates the selected environment in your default terminal without running any explicit activation commands. This is currently behind an experimental flag and can be enabled through the following User setting: "python.experiments.optInto": ["pythonTerminalEnvVarActivation"] as mentioned in our August 2023 release notes.
However, one problem with this activation mechanism is that it didn’t support the deactivate command because there is no inherent activation script. We received feedback that this is an important part of some users’ workflow, so we have added support for deactivate when the selected default terminal is PowerShell or Command Prompt. We plan to add support for additional terminals in the future.
Setting to turn REPL Smart Send on/off and a message when it is unsupportedWhen attempting to use Smart Send via kbstyle(Shift+Enter) on a Python file that contains unsupported Python code (e.g., Python 2 source code), there is now a warning message and a setting to deactivate REPL Smart Send. Users are also able to change their user and workspace specific behavior for REPL Smart Send via the python.REPL.enableREPLSmartSend setting.
Other Changes and EnhancementsWe have also added small enhancements and fixed issues requested by users that should improve your experience working with Python and Jupyter Notebooks in Visual Studio Code. Some notable changes include:
- The Pylance extension has adjusted its release cadence to monthly stable releases and nightly pre-release builds, similar to the Python extension release cadence. These changes will allow for more extensive testing on stable builds and a more reliable user experience.
- String inputs for numerical values are now supported in attach debug configurations with the Python Debugger extension (@vscode-python-debugger#115).
- The Python test adapter rewrite experiment has been rolled out to 100% of users. For the time being, you can opt-out by adding "python.experiments.optOutFrom" : "pythonTestAdapter" in your settings.json, but we will soon drop this experimental flag and adopt this new architecture.
We would also like to extend special thanks to this month’s contributors:
- @zyxue Make version extraction more robust in vscode-black-formatter#306
- @loskutov Fixed mis-escaped string literal in vscode-black-formatter#312
- @bhagya-98 Improve setting description for default interpreter setting in vscode-black-formatter#324
- @Kelly-LC Remove Python 3.7 from test actions in vscode-black-formatter#322
- @aarushinair Update dependency packages in vscode-black-formatter#326
- @mvasilkov Fixed typo in README.md in vscode-black-formatter#358
- @kirankumarmanku Added workaround and early returning if file name matched excluded arg in vscode-autopep8#142
- @norasoliman Remove Python 3.7 from test actions in vscode-autopep8#165
- @wuyumin Fix config file setting in README.md in vscode-autopep8#156
- @vinitaparasrampuria Capitalize Python in README.md and settings description in vscode-autopep8#167
- @JamzumSum Improved shell identifier on case-insensitive system in vscode-python#22391
- @trysten Add consoleTitle to launch.json properties schema in vscode-python#22406
- @shanesaravia Resolve test suite discovery import errors due to path ordering in vscode-python#22454
- @johnhany97 Pass along Python interpreter to jedi-language-server in vscode-python#22466
Try out these new improvements by downloading the Python extension and the Jupyter extension from the Marketplace, or install them directly from the extensions view in Visual Studio Code (Ctrl + Shift + X or ⌘ + ⇧ + X). You can learn more about Python support in Visual Studio Code in the documentation. If you run into any problems or have suggestions, please file an issue on the Python VS Code GitHub page.
The post Python in Visual Studio Code – December 2023 Release appeared first on Python.
Dima Kogan: roslanch and =LD_PRELOAD=
This is part 2 of our series entitled "ROS people don't know how to use computers". This is about ROS1. ROS2 is presumably broken in some completely different way, but I don't know.
Unlike normal people, the ROS people don't "run" applications. They "launch" "nodes" from "packages" (these are "ROS" packages; obviously). You run
roslaunch PACKAGE THING.launchThen it tries to find this PACKAGE (using some rules that nobody understands), and tries to find the file THING.launch within this package. The .launch file contains inscrutable xml, which includes other inscrutable xml. And if you dig, you eventually find stuff like
<node pkg="PACKAGE" name="NAME" type="TYPE" args="...." ...>This defines the thing that runs. Unexpectedly, the executable that ends up running is called TYPE.
I know that my particular program is broken, and needs an LD_PRELOAD (exciting details described in another rant in the near future). But the above definition doesn't have a clear way to add that. Adding it to the type fails (with a very mysterious error message). Reading the docs tells you about launch-prefix, which sounds exactly like what I want. But when I add LD_PRELOAD=/tmp/whatever.so I get
RLException: Roslaunch got a 'No such file or directory' error while attempting to run: LD_PRELOAD=/tmp/whatever.so ..../TYPE .....But this is how you're supposed to be attaching gdb and such! Presumably it looks at the first token, and makes sure it's a file, instead of simply prepending it to the string it passes to the shell. So your options are:
- Do only approved ROS things in the docs (which are limited, since the docs were written by people who don't know how to use computers)
- Be expert-enough to work around it
I'm expert-enough. You do this:
launch-prefix="/lib64/ld-linux-x86-64.so.2 --preload /tmp/whatever.so"CodersLegacy: Pytest Tutorial: Mastering Unit Testing in Python
Welcome to a ALL-IN-ONE Tutorial designed to meet all your testing requirements. Whether you’re just starting with the fundamentals to build a solid conceptual foundation or aiming to craft professional-grade test cases for entire projects, this guide has got you covered. The focus of this tutorial will be around the popular “Pytest” library.
Table Of Contents- Understanding Unit Testing
- Why Pytest?
- Getting Started with pytest
- Pytest Tutorial: Writing your First Test
- Pytest Tutorial: Parameterized Testing
- Pytest Tutorial: Command-Line Options
- How to use Pytest effectively in Larger Projects
- Conclusion
In the world of programming, a unit is the smallest part of your code, like a single function or method. Unit testing involves examining these individual parts to ensure they’re doing what they’re supposed to. It’s like putting each piece of the puzzle under a microscope to make sure it fits perfectly and does its job without causing trouble for the whole picture.
Imagine you’re building a complex building. The traditional approach might be to wait for the entire building to be completed, then performing tests on it to test its integrity. Unit testing on the other hand, would have you test the integrity of each floor as you build it.
Here is another scenario:
You have a function that’s supposed to add two numbers together. Unit testing for this function would involve giving it different pairs of numbers and checking if it consistently produces the correct sum. It’s like asking, “Hey, can you add 2 and 3? What about 0 and 0? Or even -1 and 1?” Each time, the unit test checks if the function gives the right answer. These different pairs of numbers must be defined carefully to ensure that the function works under a variety of different circumstances. For example, the first pair might use negative numbers, second pair might use positive numbers, and third pair might target an “error” case where a string and a number are used as inputs (with the expectation of the test failing).
Why bother with this meticulous process? Because, it helps catch bugs early on, before they turn into big, tangled problems.
Unit testing ensures that each building block of your code functions as expected, creating a solid foundation for your software structure. It’s a practice that developers swear by because it not only saves time but also makes your code more reliable.
Why Pytest?Let’s explore some of the key features that make pytest a popular choice for testing in Python.
One of pytest‘s strengths is its ability to automatically discover and run tests in your project. By default, pytest identifies files with names starting with “test_” or ending with “_test.py” and considers them as test modules. It then discovers test functions or methods within these modules.
pytest uses a simplified and expressive syntax for writing tests. Test functions don’t need to be part of a class, and assertions can be made using the assert statement directly. This leads to more readable and concise test code.
Pytest supports parameterized testing, enabling developers to run the same test with multiple sets of inputs. This feature is incredibly beneficial for testing a variety of scenarios without duplicating test code.
And many more such benefits (that we can’t explain without getting too technical).
Getting Started with pytestTo get started with pytest, you need to install it. You can do this using pip, the Python package installer, with the following command:
pip install pytestOnce installed, you can run tests using the pytest command.
Pytest Tutorial: Writing your First TestLet’s start by creating a basic test using pytest. Create a file named test_example.py with the following content:
# test_example.py def add(x, y): return x + y def test_add(): assert add(2, 3) == 5 assert add(0, 0) == 0 assert add(-1, 1) == 0In this example, we define a simple add function and a corresponding test function using pytest‘s assert statement. The test checks whether the add function produces the expected results for different input values.
To execute the tests, run the following command in your terminal:
pytest test_example.pypytest will discover and run all test functions in the specified file, by looking for functions with the word “test” in their names. If the tests pass, you’ll see an output indicating success. Otherwise, pytest will provide detailed information about the failures.
Pytest Tutorial: Parameterized Testingpytest supports parameterized testing, enabling you to run the same test with multiple sets of inputs. This is achieved using the @pytest.mark.parametrize decorator.
import pytest def add(x, y): return x + y @pytest.mark.parametrize("input_a, input_b, expected", [ (2, 3, 5), (0, 0, 0), (-1, 1, 0), ]) def test_add(input_a, input_b, expected): result = add(input_a, input_b) assert result == expectedIn this example, the test_add function is executed three times with different input values, reducing code duplication and making it easier to cover various scenarios.
Pytest Tutorial: Command-Line Optionspytest provides a plethora of command-line options to customize test runs. For example, you can specify the directory or files to test, run specific tests or test classes, and control the verbosity of the output.
Here are key command-line options:
Use the -k option to specify a substring match for test names. For instance:
pytest -k test_moduleThis command runs all tests containing “test_module” in their names.
Specify a specific directory or file to test:
pytest tests/test_module.pyExecute tests from a specific file or directory, allowing targeted testing.
Adjust the verbosity level with the -v option to get more detailed output:
pytest -vDisplay test names and results. Useful for understanding test execution flow.
Increase verbosity for even more detailed information:
pytest -vvProvide additional information about skipped tests and setup/teardown stages.
Utilize custom markers to categorize and selectively run tests. For example:
pytest -m slowThis command runs tests marked with @pytest.mark.slow, allowing you to separate and focus on tests specifically categorized as slow-running.
Select tests based on their outcome, such as only running failed tests:
pytest --lfRun only the tests that failed in the last test run.
Speed up test runs by leveraging parallel execution:
pytest -n autoThis command runs tests in parallel, utilizing all available CPU cores.
Generate detailed reports in various formats, such as HTML or XML:
pytest --html=report.htmlThis command produces an HTML report for a more visual representation of test results, aiding in result analysis and sharing with stakeholders.
These command-line options empower developers to fine-tune their testing processes, making Pytest a flexible and customizable tool for projects of any scale. Whether you need to run specific tests, control output verbosity, or generate comprehensive reports, Pytest’s command-line options provide the versatility needed for efficient and effective testing.
How to use Pytest effectively in Larger ProjectsAs the size of your project grows, with the number of files and lines of code increasing significantly, the need to organize your code becomes even more important. While it may seem tempting to write all your “test” functions in the same file as your regular code, this is not an ideal solution.
Instead, it is recommended to create a separate file where all of your tests are written. Depending on the size of the project, you can even have multiple test files (e.g. one test file for each class).
Opting for this approach introduces potential complications. When test cases are written in a separate file, a common concern arises: How do we invoke the functions intended for testing?
This requires careful structuring of your project and code to ensure that individual functions and classes of your project can be imported by the pytest files.
Here is a good project structure to follow, where each of the files in src folder represent an independent module (e.g. a single class), and each of the files in the tests folder corresponds to a file in the src folder.
project_root/ |-- src/ | |-- __init__.py | |-- users.py | |-- services.py | |-- tests/ | |-- __init__.py | |-- test_users.py | |-- test_services.py | | -- main.pyThe __init__.py file is an important addition to the src folder, where all of our project files are stored (excluding the main driver code). When this file is created in a folder, that folder will be recognized by Python as a Python Package, and enables other files to import files from within this folder. You do not have to put anything in this file (leave it empty, though it can be customized with special statements).
It is also necessary to put the __init__.py file in the tests folder, in order for imports between it, and the src folder to succeed.
Example scenario: Importing the users.py file from test_users.py.
from src.users import * ConclusionThis marks the end of the Pytest tutorial.
By incorporating unit testing into your development workflow, you can catch and fix bugs early, improve code maintainability, and ensure that your software functions as intended. With pytest, the journey of mastering unit testing in Python becomes not only effective but also enjoyable. So, go ahead, write those tests, and build robust, reliable Python applications with confidence!
The post Pytest Tutorial: Mastering Unit Testing in Python appeared first on CodersLegacy.
Python Insider: Python 3.12.1 is now available
Python 3.12.1 is now available.
https://www.python.org/downloads/release/python-3121/
Python 3.12 is the newest major release of the Python programming language, and it contains many new features and optimizations. 3.12.1 is the latest maintenance release, containing more than 400 bugfixes, build improvements and documentation changes since 3.12.0.
- More flexible f-string parsing , allowing many things previously disallowed (PEP 701).
- Support for the buffer protocol in Python code (PEP 688).
- A new debugging/profiling API (PEP 669).
- Support for isolated subinterpreters with separate Global Interpreter Locks (PEP 684).
- Even more improved error messages. More exceptions potentially caused by typos now make suggestions to the user.
- Support for the Linux perf profiler to report Python function names in traces.
- Many large and small performance improvements (like PEP 709 and support for the BOLT binary optimizer), delivering an estimated 5% overall performance improvement.
- New type annotation syntax for generic classes (PEP 695).
- New override decorator for methods (PEP 698).
- The deprecated wstr and wstr_length members of the C implementation of unicode objects were removed, per PEP 623.
- In the unittest module, a number of long deprecated methods and classes were removed. (They had been deprecated since Python 3.1 or 3.2).
- The deprecated smtpd and distutils modules have been removed (see PEP 594 and PEP 632. The setuptools package continues to provide the distutils module.
- A number of other old, broken and deprecated functions, classes and methods have been removed.
- Invalid backslash escape sequences in strings now warn with SyntaxWarning instead of DeprecationWarning, making them more visible. (They will become syntax errors in the future.)
- The internal representation of integers has changed in preparation for performance enhancements. (This should not affect most users as it is an internal detail, but it may cause problems for Cython-generated code.)
For more details on the changes to Python 3.12, see What’s new in Python 3.12.
More resources- Online Documentation.
- PEP 693, the Python 3.12 Release Schedule.
- Report bugs via GitHub Issues.
- Help fund Python directly or via GitHub Sponsors, and support the Python community.
Enjoy the new releases
Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.
Your release team,
Thomas Wouters
Ned Deily
Steve Dower
Łukasz Langa