FLOSS Project Planets

Glyph Lefkowitz: Annotated At Runtime

Planet Python - Thu, 2023-12-07 20:56

PEP 0593 added the ability to add arbitrary user-defined metadata to type annotations in Python.

At type-check time, such annotations are… inert. They don’t do anything. Annotated[int, X] just means int to the type-checker, regardless of the value of X. So the entire purpose of Annotated is to provide a run-time API to consume metadata, which integrates with the type checker syntactically, but does not otherwise disturb it.

Yet, the documentation for this central purpose seems, while not exactly absent, oddly incomplete.

The PEP itself simply says:

A tool or library encountering an Annotated type can scan through the annotations to determine if they are of interest (e.g., using isinstance()).

But it’s not clear where “the annotations” are, given that the PEP’s entire “consuming annotations” section does not even mention the __metadata__ attribute where the annotation’s arguments go, which was only even added to CPython’s documentation. Its list of examples just show the repr() of the relevant type.

There’s also a bit of an open question of what, exactly, we are supposed to isinstance()-ing here. If we want to find arguments to Annotated, presumably we need to be able to detect if an annotation is an Annotated. But isinstance(Annotated[int, "hello"], Annotated) is both False at runtime, and also a type-checking error, that looks like this:

1Argument 2 to "isinstance" has incompatible type "<typing special form>"; expected "_ClassInfo"

The actual type of these objects, typing._AnnotatedAlias, does not seem to have a publicly available or documented alias, so that seems like the wrong route too.

Now, it certainly works to escape-hatch your way out of all of this with an Any, build some version-specific special-case hacks to dig around in the relevant namespaces, access __metadata__ and call it a day. But this solution is … unsatisfying.

What are you looking for?

Upon encountering these quirks, it is understandable to want to simply ask the question “is this annotation that I’m looking at an Annotated?” and to be frustrated that it seems so obscure to straightforwardly get an answer to that question without disabling all type-checking in your meta-programming code.

However, I think that this is a slightly misframing of the problem. Code that is inspecting parameters for an annotation is going to do something with that annotation, which means that it must necessarily be looking for a specific set of annotations. Therefore the thing we want to pass to isinstance is not some obscure part of the annotations’ internals, but the actual interesting annotation type from your framework or application.

When consuming an Annotated parameter, there are 3 things you probably want to know:

  1. What was the parameter itself? (type: The type you passed in.)
  2. What was the name of the annotated object (i.e.: the parameter name, the attribute name) being passed the parameter? (type: str)
  3. What was the actual type being annotated? (type: type)

And the things that we have are the type of the Annotated we’re querying for, and the object with annotations we are interrogating. So that gives us this function signature:

1 2 3 4 5def annotated_by( annotated: object, kind: type[T], ) -> Iterable[tuple[str, T, type]]: ...

To extract this information, all we need are get_args and get_type_hints; no need for __metadata__ or get_origin or any other metaprogramming. Here’s a recipe:

1 2 3 4 5 6 7 8 9 10 11 12def annotated_by( annotated: object, kind: type[T], ) -> Iterable[tuple[str, T, type]]: for k, v in get_type_hints(annotated, include_extras=True).items(): all_args = get_args(v) if not all_args: continue actual, *rest = all_args for arg in rest: if isinstance(arg, kind): yield k, arg, actual

It might seem a little odd to be blindly assuming that get_args(...)[0] will always be the relevant type, when that is not true of unions or generics. Note, however, that we are only yielding results when we have found the instance type in the argument list; our arbitrary user-defined instance isn’t valid as a type annotation argument in any other context. It can’t be part of a Union or a Generic, so we can rely on it to be an Annotated argument, and from there, we can make that assumption about the format of get_args(...).

This can give us back the annotations that we’re looking for in a handy format that’s easy to consume. Here’s a quick example of how you might use it:

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15@dataclass class AnAnnotation: name: str def a_function( a: str, b: Annotated[int, AnAnnotation("b")], c: Annotated[float, AnAnnotation("c")], ) -> None: ... print(list(annotated_by(a_function, AnAnnotation))) # [('b', AnAnnotation(name='b'), <class 'int'>), # ('c', AnAnnotation(name='c'), <class 'float'>)] Acknowledgments

Thank you to my patrons who are supporting my writing on this blog. If you like what you’ve read here and you’d like to read more of it, or you’d like to support my various open-source endeavors, you can support me on Patreon as well! I am also available for consulting work if you think your organization could benefit from expertise on topics like “how do I do Python metaprogramming, but, like, not super janky”.

Categories: FLOSS Project Planets

James Bennett: Use "pip install" safely

Planet Python - Thu, 2023-12-07 20:13

This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post for an introduction.

Managing dependencies should be boring

Last year I wrote a post about “boring” dependency management in Python, where I advocated a setup based entirely around standard Python packaging tools, in that …

Read full entry

Categories: FLOSS Project Planets

Reproducible Builds (diffoscope): diffoscope 253 released

Planet Debian - Thu, 2023-12-07 19:00

The diffoscope maintainers are pleased to announce the release of diffoscope version 253. This version includes the following changes:

* Improve DOS/MBR extraction by adding support for 7z. (Closes: reproducible-builds/diffoscope#333) * Process objdump symbol comment filter inputs as the Python "bytes" type (and not str). (Closes: reproducible-builds/diffoscope#358) * Add a missing RequiredToolNotFound import. * Update copyright years.

You find out more by visiting the project homepage.

Categories: FLOSS Project Planets

Python Engineering at Microsoft: Python in Visual Studio Code – December 2023 Release

Planet Python - Thu, 2023-12-07 16:02

We’re excited to announce the December 2023 release of the Python and Jupyter extensions for Visual Studio Code!

This release includes the following announcements:

  • Configurable debugging options added to Run button menu
  • Show Type Hierarchy with Pylance
  • Deactivate command support for automatically activated virtual environments in the terminal
  • Setting to turn REPL Smart Send on/off and a message when it is unsupported

If you’re interested, you can check the full list of improvements in our changelogs for the Python, Jupyter and Pylance extensions.

Configurable debugging options added to Run button menu

The Python Debugger extension now has configurable debug options under the Run button menu. When you select Python Debugger: Debug using launch.json and there is an existing launch.json in your workspace, it shows all available debug configurations you can pick to start the debugger. In the case you do not have an existing launch.json, you will be prompted to select a debug configuration template to create a launch.json file for your Python application, and then can run your application using this configuration.

Show Type Hierarchy with Pylance

You can now more conveniently explore and navigate through your Python projects’ types relationships when using Pylance. This can be helpful when working with large codebases with complex type relationships.

When you right-click on a symbol, you can select Show Type Hierarchy to open the type hierarchy view. From there you can navigate through the symbol’s subtypes as well as super-types.

Deactivate command support for automatically activated virtual environments in the terminal

The Python extension has a new activation mechanism that activates the selected environment in your default terminal without running any explicit activation commands. This is currently behind an experimental flag and can be enabled through the following User setting: "python.experiments.optInto": ["pythonTerminalEnvVarActivation"] as mentioned in our August 2023 release notes.

However, one problem with this activation mechanism is that it didn’t support the deactivate command because there is no inherent activation script. We received feedback that this is an important part of some users’ workflow, so we have added support for deactivate when the selected default terminal is PowerShell or Command Prompt. We plan to add support for additional terminals in the future.

Setting to turn REPL Smart Send on/off and a message when it is unsupported

When attempting to use Smart Send via kbstyle(Shift+Enter) on a Python file that contains unsupported Python code (e.g., Python 2 source code), there is now a warning message and a setting to deactivate REPL Smart Send. Users are also able to change their user and workspace specific behavior for REPL Smart Send via the python.REPL.enableREPLSmartSend setting.

Other Changes and Enhancements

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python and Jupyter Notebooks in Visual Studio Code. Some notable changes include:

  • The Pylance extension has adjusted its release cadence to monthly stable releases and nightly pre-release builds, similar to the Python extension release cadence. These changes will allow for more extensive testing on stable builds and a more reliable user experience.
  • String inputs for numerical values are now supported in attach debug configurations with the Python Debugger extension (@vscode-python-debugger#115).
  • The Python test adapter rewrite experiment has been rolled out to 100% of users. For the time being, you can opt-out by adding "python.experiments.optOutFrom" : "pythonTestAdapter" in your settings.json, but we will soon drop this experimental flag and adopt this new architecture.

We would also like to extend special thanks to this month’s contributors:

Try out these new improvements by downloading the Python extension and the Jupyter extension from the Marketplace, or install them directly from the extensions view in Visual Studio Code (Ctrl + Shift + X or ⌘ + ⇧ + X). You can learn more about Python support in Visual Studio Code in the documentation. If you run into any problems or have suggestions, please file an issue on the Python VS Code GitHub page.

The post Python in Visual Studio Code – December 2023 Release appeared first on Python.

Categories: FLOSS Project Planets

Dima Kogan: roslanch and =LD_PRELOAD=

Planet Debian - Thu, 2023-12-07 15:56

This is part 2 of our series entitled "ROS people don't know how to use computers". This is about ROS1. ROS2 is presumably broken in some completely different way, but I don't know.

Unlike normal people, the ROS people don't "run" applications. They "launch" "nodes" from "packages" (these are "ROS" packages; obviously). You run

roslaunch PACKAGE THING.launch

Then it tries to find this PACKAGE (using some rules that nobody understands), and tries to find the file THING.launch within this package. The .launch file contains inscrutable xml, which includes other inscrutable xml. And if you dig, you eventually find stuff like

<node pkg="PACKAGE" name="NAME" type="TYPE" args="...." ...>

This defines the thing that runs. Unexpectedly, the executable that ends up running is called TYPE.

I know that my particular program is broken, and needs an LD_PRELOAD (exciting details described in another rant in the near future). But the above definition doesn't have a clear way to add that. Adding it to the type fails (with a very mysterious error message). Reading the docs tells you about launch-prefix, which sounds exactly like what I want. But when I add LD_PRELOAD=/tmp/whatever.so I get

RLException: Roslaunch got a 'No such file or directory' error while attempting to run: LD_PRELOAD=/tmp/whatever.so ..../TYPE .....

But this is how you're supposed to be attaching gdb and such! Presumably it looks at the first token, and makes sure it's a file, instead of simply prepending it to the string it passes to the shell. So your options are:

  • Do only approved ROS things in the docs (which are limited, since the docs were written by people who don't know how to use computers)
  • Be expert-enough to work around it

I'm expert-enough. You do this:

launch-prefix="/lib64/ld-linux-x86-64.so.2 --preload /tmp/whatever.so"
Categories: FLOSS Project Planets

CodersLegacy: Pytest Tutorial: Mastering Unit Testing in Python

Planet Python - Thu, 2023-12-07 14:54

Welcome to a ALL-IN-ONE Tutorial designed to meet all your testing requirements. Whether you’re just starting with the fundamentals to build a solid conceptual foundation or aiming to craft professional-grade test cases for entire projects, this guide has got you covered. The focus of this tutorial will be around the popular “Pytest” library.

Table Of Contents
  1. Understanding Unit Testing
  2. Why Pytest?
  3. Getting Started with pytest
  4. Pytest Tutorial: Writing your First Test
  5. Pytest Tutorial: Parameterized Testing
  6. Pytest Tutorial: Command-Line Options
  7. How to use Pytest effectively in Larger Projects
  8. Conclusion
Understanding Unit Testing

In the world of programming, a unit is the smallest part of your code, like a single function or method. Unit testing involves examining these individual parts to ensure they’re doing what they’re supposed to. It’s like putting each piece of the puzzle under a microscope to make sure it fits perfectly and does its job without causing trouble for the whole picture.

Imagine you’re building a complex building. The traditional approach might be to wait for the entire building to be completed, then performing tests on it to test its integrity. Unit testing on the other hand, would have you test the integrity of each floor as you build it.

Here is another scenario:

You have a function that’s supposed to add two numbers together. Unit testing for this function would involve giving it different pairs of numbers and checking if it consistently produces the correct sum. It’s like asking, “Hey, can you add 2 and 3? What about 0 and 0? Or even -1 and 1?” Each time, the unit test checks if the function gives the right answer. These different pairs of numbers must be defined carefully to ensure that the function works under a variety of different circumstances. For example, the first pair might use negative numbers, second pair might use positive numbers, and third pair might target an “error” case where a string and a number are used as inputs (with the expectation of the test failing).

Why bother with this meticulous process? Because, it helps catch bugs early on, before they turn into big, tangled problems.

Unit testing ensures that each building block of your code functions as expected, creating a solid foundation for your software structure. It’s a practice that developers swear by because it not only saves time but also makes your code more reliable.

Why Pytest?

Let’s explore some of the key features that make pytest a popular choice for testing in Python.


1. Automatic Test Discovery

One of pytest‘s strengths is its ability to automatically discover and run tests in your project. By default, pytest identifies files with names starting with “test_” or ending with “_test.py” and considers them as test modules. It then discovers test functions or methods within these modules.


2. Concise Syntax

pytest uses a simplified and expressive syntax for writing tests. Test functions don’t need to be part of a class, and assertions can be made using the assert statement directly. This leads to more readable and concise test code.


3. Parameterized Testing

Pytest supports parameterized testing, enabling developers to run the same test with multiple sets of inputs. This feature is incredibly beneficial for testing a variety of scenarios without duplicating test code.


And many more such benefits (that we can’t explain without getting too technical).

Getting Started with pytest

To get started with pytest, you need to install it. You can do this using pip, the Python package installer, with the following command:

pip install pytest

Once installed, you can run tests using the pytest command.

Pytest Tutorial: Writing your First Test

Let’s start by creating a basic test using pytest. Create a file named test_example.py with the following content:

# test_example.py def add(x, y): return x + y def test_add(): assert add(2, 3) == 5 assert add(0, 0) == 0 assert add(-1, 1) == 0

In this example, we define a simple add function and a corresponding test function using pytest‘s assert statement. The test checks whether the add function produces the expected results for different input values.

To execute the tests, run the following command in your terminal:

pytest test_example.py

pytest will discover and run all test functions in the specified file, by looking for functions with the word “test” in their names. If the tests pass, you’ll see an output indicating success. Otherwise, pytest will provide detailed information about the failures.

Pytest Tutorial: Parameterized Testing

pytest supports parameterized testing, enabling you to run the same test with multiple sets of inputs. This is achieved using the @pytest.mark.parametrize decorator.

import pytest def add(x, y): return x + y @pytest.mark.parametrize("input_a, input_b, expected", [ (2, 3, 5), (0, 0, 0), (-1, 1, 0), ]) def test_add(input_a, input_b, expected): result = add(input_a, input_b) assert result == expected

In this example, the test_add function is executed three times with different input values, reducing code duplication and making it easier to cover various scenarios.

Pytest Tutorial: Command-Line Options

pytest provides a plethora of command-line options to customize test runs. For example, you can specify the directory or files to test, run specific tests or test classes, and control the verbosity of the output.

Here are key command-line options:


Specifying Test Directories or Files:

Use the -k option to specify a substring match for test names. For instance:

pytest -k test_module

This command runs all tests containing “test_module” in their names.

Specify a specific directory or file to test:

pytest tests/test_module.py

Execute tests from a specific file or directory, allowing targeted testing.


Controlling Verbosity:

Adjust the verbosity level with the -v option to get more detailed output:

pytest -v

Display test names and results. Useful for understanding test execution flow.

Increase verbosity for even more detailed information:

pytest -vv

Provide additional information about skipped tests and setup/teardown stages.


Marking and Selecting Tests:

Utilize custom markers to categorize and selectively run tests. For example:

pytest -m slow

This command runs tests marked with @pytest.mark.slow, allowing you to separate and focus on tests specifically categorized as slow-running.

Select tests based on their outcome, such as only running failed tests:

pytest --lf

Run only the tests that failed in the last test run.


Parallel Test Execution:

Speed up test runs by leveraging parallel execution:

pytest -n auto

This command runs tests in parallel, utilizing all available CPU cores.


Generating Detailed Reports:

Generate detailed reports in various formats, such as HTML or XML:

pytest --html=report.html

This command produces an HTML report for a more visual representation of test results, aiding in result analysis and sharing with stakeholders.


These command-line options empower developers to fine-tune their testing processes, making Pytest a flexible and customizable tool for projects of any scale. Whether you need to run specific tests, control output verbosity, or generate comprehensive reports, Pytest’s command-line options provide the versatility needed for efficient and effective testing.

How to use Pytest effectively in Larger Projects

As the size of your project grows, with the number of files and lines of code increasing significantly, the need to organize your code becomes even more important. While it may seem tempting to write all your “test” functions in the same file as your regular code, this is not an ideal solution.

Instead, it is recommended to create a separate file where all of your tests are written. Depending on the size of the project, you can even have multiple test files (e.g. one test file for each class).

Opting for this approach introduces potential complications. When test cases are written in a separate file, a common concern arises: How do we invoke the functions intended for testing?

This requires careful structuring of your project and code to ensure that individual functions and classes of your project can be imported by the pytest files.

Here is a good project structure to follow, where each of the files in src folder represent an independent module (e.g. a single class), and each of the files in the tests folder corresponds to a file in the src folder.

project_root/ |-- src/ | |-- __init__.py | |-- users.py | |-- services.py | |-- tests/ | |-- __init__.py | |-- test_users.py | |-- test_services.py | | -- main.py

The __init__.py file is an important addition to the src folder, where all of our project files are stored (excluding the main driver code). When this file is created in a folder, that folder will be recognized by Python as a Python Package, and enables other files to import files from within this folder. You do not have to put anything in this file (leave it empty, though it can be customized with special statements).

It is also necessary to put the __init__.py file in the tests folder, in order for imports between it, and the src folder to succeed.

Example scenario: Importing the users.py file from test_users.py.

from src.users import * Conclusion

This marks the end of the Pytest tutorial.

By incorporating unit testing into your development workflow, you can catch and fix bugs early, improve code maintainability, and ensure that your software functions as intended. With pytest, the journey of mastering unit testing in Python becomes not only effective but also enjoyable. So, go ahead, write those tests, and build robust, reliable Python applications with confidence!

The post Pytest Tutorial: Mastering Unit Testing in Python appeared first on CodersLegacy.

Categories: FLOSS Project Planets

Python Insider: Python 3.12.1 is now available

Planet Python - Thu, 2023-12-07 14:50

 

Python 3.12.1 is now available.

https://www.python.org/downloads/release/python-3121/

 

This is the first maintenance release of Python 3.12

Python 3.12 is the newest major release of the Python programming language, and it contains many new features and optimizations. 3.12.1 is the latest maintenance release, containing more than 400 bugfixes, build improvements and documentation changes since 3.12.0.

 Major new features of the 3.12 series, compared to 3.11  New features  Type annotations  Deprecations
  • The deprecated wstr and wstr_length members of the C implementation of unicode objects were removed, per PEP 623.
  • In the unittest module, a number of long deprecated methods and classes were removed. (They had been deprecated since Python 3.1 or 3.2).
  • The deprecated smtpd and distutils modules have been removed (see PEP 594 and PEP 632. The setuptools package continues to provide the distutils module.
  • A number of other old, broken and deprecated functions, classes and methods have been removed.
  • Invalid backslash escape sequences in strings now warn with SyntaxWarning instead of DeprecationWarning, making them more visible. (They will become syntax errors in the future.)
  • The internal representation of integers has changed in preparation for performance enhancements. (This should not affect most users as it is an internal detail, but it may cause problems for Cython-generated code.)

For more details on the changes to Python 3.12, see What’s new in Python 3.12.

 More resources  
Enjoy the new releases

Thanks to all of the many volunteers who help make Python Development and these releases possible! Please consider supporting our efforts by volunteering yourself or through organization contributions to the Python Software Foundation.

Your release team,
Thomas Wouters
Ned Deily
Steve Dower
Łukasz Langa

Categories: FLOSS Project Planets

Promet Source: Exploring AI for Drupal Development

Planet Drupal - Thu, 2023-12-07 14:36
While 2023 appears to be the year that Artificial Intelligence emerged from the shadows and into the mainstream, the potential of AI has barely scratched the surface. AI is here and its impact on life and work is developing at an exponential pace.  As this disruptive technology is generating quick answers, streamlining processes, and creating vast new efficiencies, hundreds of possibilities for AI – ranging from healthcare diagnoses, to cybersecurity threat detection, content creation, software development, and many, many more – are taking shape.
Categories: FLOSS Project Planets

Promet Source: Exploring AI for Drupal Development and More

Planet Drupal - Thu, 2023-12-07 11:01
While 2023 appears to be the year that Artificial Intelligence emerged from the shadows and into the mainstream, the potential of AI has barely scratched the surface. AI is here and its impact on life and work is developing at an exponential pace.  As this disruptive technology is generating quick answers, streamlining processes, and creating vast new efficiencies, hundreds of possibilities for AI – ranging from healthcare diagnoses, to cybersecurity threat detection, content creation, software development, and many, many more – are taking shape.
Categories: FLOSS Project Planets

Pages