Planet Python

Subscribe to Planet Python feed
Planet Python - http://planetpython.org/
Updated: 10 hours 21 min ago

Will McGugan: A prettier Python REPL with Rich

Sun, 2020-08-09 11:08

Rich 5.1.0 adds the ability to integrate highlighted pretty printing with the Python REPL. Here's how you would use it in a session:

>>> from rich import pretty >>> pretty.install()

Assuming you have installed Rich of course.

Now any data will be formatted and highlighted. Here's an example:

© 2020 Will McGugan

Rich will expand builtin containers until the output fits in to the width of the screen.

In addition to Python data structures, any object supporting the Rich Console Protocol will be displayed. Here's an example:

© 2020 Will McGugan

The Panel class is one of a number of renderables in Rich.

The formatting and highlighting really comes in to its own when you're working with JSON data. The following video shows an example of pretty printing data queried from a web service. In this instance, its cat facts.

Just discovered Rich?

Follow @willmcgugan for updates regarding Rich.

Categories: FLOSS Project Planets

PSF GSoC students blogs: [Week 10] Check-in

Sun, 2020-08-09 09:18
1. What did you do this week?
  • Complete np.reshape and np.stack.

  • Split raw_functions & nograd_functions

  • Fix higher-order derivatives.

2. Difficulty

No difficulties this week.

3. What is coming up next?
  • Publish a simple version of udiff on pypi.
  • Add more testcases.
Categories: FLOSS Project Planets

PSF GSoC students blogs: Weekly Check-In: Week 11

Sun, 2020-08-09 03:56


Make sure to check out Project FURY : https://github.com/fury-gl/fury

Hey Everyone! 
This week , more shaders

What did I do this week

The task assigned for this week was to explore more shader techniques which could be implemented using FURY and which would demonstrate the capability of FURY shader system. So i decided to work on implementing shading examples such as Gooch shading and reflection shader using textures.

Below are the outputs of the techniques i worked on :

The shader demos are available at :
https://github.com/lenixlobo/fury/tree/shader-demos

What's coming up next

The next week will involve working on more such demos which can demonstrate the capabilities of FURY

Did I get stuck anywhere

No issues were faced this week

Categories: FLOSS Project Planets

Montreal Python User Group: Montréal-Python 79 – Quadratic Judo

Sun, 2020-08-09 00:00

Summer nears its end and Montréal-Python is coming back from vacation. Before going back to school or to work, come and tell us what new Pythonic things during the summer. As always, we are looking for passionate people who want to share with the community. Be it about a project that you have built or a library that you learned, send us your talk proposal to mtlpyteam@googlegroups.com . Talks duration can be anywhere between 5 and 20 minutes. We accept all reasonable proposals until the program is complete. We can't wait to read your proposal!

Montréal-Python 79, Quadratic Judo, will be live streamed on our Youtube Channel on August 24 at 5:30 PM EST.

As usual, we want to bring your attention to the Montréal-Python code of conduct. We remind you that we are an open community and actively invite people of all minorities to come and share their experiences and their knowledge. To align with our values, we also remember those wise words by the Python Software Foundation:

In silence we are complicit. To eradicate racism, xenophobia, and all other forms of inequality, we must take action. It is our responsibility to use our platforms to amplify the voices of the oppressed and marginalized.

We ask today – and always – that the Python community comes together in support of the Black community, and makes Black voices heard.

Black lives matter.

And finally, PyBay 2020 moved to a fully online conference taking place next weekend, August 15 and 16. They offer free tickets to women and to members of many minorities. Check out their website for all the details:

https://pybay.com/

Categories: FLOSS Project Planets

Full Stack Python: How to Transcribe Speech Recordings into Text with Python

Sun, 2020-08-09 00:00

When you have a recording where one or more people are talking, it's useful to have a highly accurate and automated way to extract the spoken words into text. Once you have the text, you can use it for further analysis or as an accessibility feature.

In this tutorial, we'll use a high accuracy speech-to-text web application programming interface called AssemblyAI to extract text from an MP3 recording (many other formats are supported as well).

With the code from this tutorial, you will be able to take an audio file that contains speech such as this example one I recorded and output a highly accurate text transcription like this:

An object relational mapper is a code library that automates the transfer of data stored in relational, databases into objects that are more commonly used in application code or EMS are useful because they provide a high level abstraction upon a relational database that allows developers to write Python code instead of sequel to create read update and delete, data and schemas in their database. Developers can use the programming language. They are comfortable with to work with a database instead of writing SQL... (the text goes on from here but I abbreviated it at this point) Tutorial requirements

Throughout this tutorial we are going to use the following dependencies, which we will install in just a moment. Make sure you also have Python 3, preferrably 3.6 or newer installed, in your environment:

We will use the following dependencies to complete this tutorial:

All code in this blog post is available open source under the MIT license on GitHub under the transcribe-speech-text-script directory of the blog-code-examples repository. Use the source code as you desire for your own projects.

Setting up the development environment

Change into the directory where you keep your Python virtual environments. I keep mine in a subdirectory named venvs within my user's home directory. Create a new virtualenv for this project using the following command.

python3 -m venv ~/venvs/pytranscribe

Activate the virtualenv with the activate shell script:

source ~/venvs/pytranscribe/bin/activate

After the above command is executed, the command prompt will change so that the name of the virtualenv is prepended to the original command prompt format, so if your prompt is simply $, it will now look like the following:

(pytranscribe) $

Remember, you have to activate your virtualenv in every new terminal window where you want to use dependencies in the virtualenv.

We can now install the requests package into the activated but otherwise empty virtualenv.

pip install requests==2.24.0

Look for output similar to the following to confirm the appropriate packages were installed correctly from PyPI.

(pytranscribe) $ pip install requests==2.24.0 Collecting requests==2.24.0 Using cached https://files.pythonhosted.org/packages/45/1e/0c169c6a5381e241ba7404532c16a21d86ab872c9bed8bdcd4c423954103/requests-2.24.0-py2.py3-none-any.whl Collecting certifi>=2017.4.17 (from requests==2.24.0) Using cached https://files.pythonhosted.org/packages/5e/c4/6c4fe722df5343c33226f0b4e0bb042e4dc13483228b4718baf286f86d87/certifi-2020.6.20-py2.py3-none-any.whl Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests==2.24.0) Using cached https://files.pythonhosted.org/packages/9f/f0/a391d1463ebb1b233795cabfc0ef38d3db4442339de68f847026199e69d7/urllib3-1.25.10-py2.py3-none-any.whl Collecting chardet<4,>=3.0.2 (from requests==2.24.0) Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl Collecting idna<3,>=2.5 (from requests==2.24.0) Using cached https://files.pythonhosted.org/packages/a2/38/928ddce2273eaa564f6f50de919327bf3a00f091b5baba8dfa9460f3a8a8/idna-2.10-py2.py3-none-any.whl Installing collected packages: certifi, urllib3, chardet, idna, requests Successfully installed certifi-2020.6.20 chardet-3.0.4 idna-2.10 requests-2.24.0 urllib3-1.25.10

We have all of our required dependencies installed so we can get started coding the application.

Uploading, initiating and transcribing audio

We have everything we need to start building our application that will transcribe audio into text. We're going to build this application in three files:

  1. upload_audio_file.py: uploads your audio file to a secure place on AssemblyAI's service so it can be access for processing. If your audio file is already accessible with a public URL, you don't need to do this step, you can just follow this quickstart
  2. initiate_transcription.py: tells the API which file to transcribe and to start immediately
  3. get_transcription.py: prints the status of the transcription if it is still processing, or displays the results of the transcription when the process is complete

Create a new directory named pytranscribe to store these files as we write them. Then change into the new project directory.

mkdir pytranscibe cd pytranscribe

We also need to export our AssemblyAI API key as an environment variable. Sign up for an AssemblyAI account and log in to the AssemblyAI dashboard, then copy "Your API token" as shown in this screenshot:

export ASSEMBLYAI_KEY=your-api-key-here

Note that you must use the export command in every command line window that you want this key to be accessible. The scripts we are writing will not be able to access the API if you do not have the token exported as ASSEMBLYAI_KEY in the environment you are running the script.

Now that we have our project directory created and the API key set as an environment variable, let's move on to writing the code for the first file that will upload audio files to the AssemblyAI service.

Uploading the audio file for transcription

Create a new file named upload_audio_file.py and place the following code in it:

import argparse import os import requests API_URL = "https://api.assemblyai.com/v2/" def upload_file_to_api(filename): """Checks for a valid file and then uploads it to AssemblyAI so it can be saved to a secure URL that only that service can access. When the upload is complete we can then initiate the transcription API call. Returns the API JSON if successful, or None if file does not exist. """ if not os.path.exists(filename): return None def read_file(filename, chunk_size=5242880): with open(filename, 'rb') as _file: while True: data = _file.read(chunk_size) if not data: break yield data headers = {'authorization': os.getenv("ASSEMBLYAI_KEY")} response = requests.post("".join([API_URL, "upload"]), headers=headers, data=read_file(filename)) return response.json()

The above code imports the argparse, os and requests packages so that we can use them in this script. The API_URL is a constant that has the base URL of the AssemblyAI service. We define the upload_file_to_api function with a single argument, filename that should be a string with the absolute path to a file and its filename.

Within the function, we check that the file exists, then use Request's chunked transfer encoding to stream large files to the AssemblyAI API.

The os module's getenv function reads the API that was set on the command line using the export command with the getenv. Make sure that you use that export command in the terminal where you are running this script otherwise that ASSEMBLYAI_KEY value will be blank. When in doubt, use echo $ASSEMBLY_AI to see if the value matches your API key.

To use the upload_file_to_api function, append the following lines of code in the upload_audio_file.py file so that we can properly execute this code as a script called with the python command:

if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("filename") args = parser.parse_args() upload_filename = args.filename response_json = upload_file_to_api(upload_filename) if not response_json: print("file does not exist") else: print("File uploaded to URL: {}".format(response_json['upload_url']))

The code above creates an ArgumentParser object that allows the application to obtain a single argument from the command line to specify the file we want to access, read and upload to the AssmeblyAI service.

If the file does not exist, the script will print a message that the file couldn't be found. In the happy path where we do find the correct file at that path, then the file is uploaded using the code in upload_file_to_api function.

Execute the completed upload_audio_file.py script by running it on the command line with the python command. Replace FULL_PATH_TO_FILE with an absolute path to the file you want to upload, such as /Users/matt/devel/audio.mp3.

python upload_audio_file.py FULL_PATH_TO_FILE

Assuming the file is found at the location that you specified, when the script finishes uploading the file, it will print a message like this one with a unique URL:

File uploaded to URL: https://cdn.assemblyai.com/upload/463ce27f-0922-4ea9-9ce4-3353d84b5638

This URL is not public, it can only be used by the AssemblyAI service, so no one else will be able to access your file and its contents except for you and their transcription API.

The part that is important is the last section of the URL, in this example it is 463ce27f-0922-4ea9-9ce4-3353d84b5638. Save that unique identifier because we need to pass it into the next script that initiates the transcription service.

Initiate transcription

Next, we'll write some code to kick off the transcription. Create a new file named initiate_transcription.py. Add the following code to the new file.

import argparse import os import requests API_URL = "https://api.assemblyai.com/v2/" CDN_URL = "https://cdn.assemblyai.com/" def initiate_transcription(file_id): """Sends a request to the API to transcribe a specific file that was previously uploaded to the API. This will not immediately return the transcription because it takes a moment for the service to analyze and perform the transcription, so there is a different function to retrieve the results. """ endpoint = "".join([API_URL, "transcript"]) json = {"audio_url": "".join([CDN_URL, "upload/{}".format(file_id)])} headers = { "authorization": os.getenv("ASSEMBLYAI_KEY"), "content-type": "application/json" } response = requests.post(endpoint, json=json, headers=headers) return response.json()

We have the same imports as the previous script and we've added a new constant, CDN_URL that matches the separate URL where AssemblyAI stores the uploaded audio files.

The initiate_transcription function essentially just sets up a single HTTP request to the AssemblyAI API to start the transcription process on the audio file at the specific URL passed in. This is why passing in the file_id is important: that completes the URL of the audio file that we are telling AssemblyAI to retrieve.

Finish the file by appending this code so that it can be easily invoked from the command line with arguments.

if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("file_id") args = parser.parse_args() file_id = args.file_id response_json = initiate_transcription(file_id) print(response_json)

Start the script by running the python command on the initiate_transcription file and pass in the unique file identifier you saved from the previous step.

# the FILE_IDENTIFIER is returned in the previous step and will # look something like this: 463ce27f-0922-4ea9-9ce4-3353d84b5638 python initiate_transcription.py FILE_IDENTIFIER

The API will send back a JSON response that this script prints to the command line.

{'audio_end_at': None, 'acoustic_model': 'assemblyai_default', 'text': None, 'audio_url': 'https://cdn.assemblyai.com/upload/463ce27f-0922-4ea9-9ce4-3353d84b5638', 'speed_boost': False, 'language_model': 'assemblyai_default', 'redact_pii': False, 'confidence': None, 'webhook_status_code': None, 'id': 'gkuu2krb1-8c7f-4fe3-bb69-6b14a2cac067', 'status': 'queued', 'boost_param': None, 'words': None, 'format_text': True, 'webhook_url': None, 'punctuate': True, 'utterances': None, 'audio_duration': None, 'auto_highlights': False, 'word_boost': [], 'dual_channel': None, 'audio_start_from': None}

Take note of the value of the id key in the JSON response. This is the transcription identifier we need to use to retrieve the transcription result. In this example, it is gkuu2krb1-8c7f-4fe3-bb69-6b14a2cac067. Copy the transcription identifier in your own response because we will need it to check when the transcription process has completed in the next step.

Retrieving the transcription result

We have uploaded and begun the transcription process, so let's get the result as soon as it is ready.

How long it takes to get the results back can depend on the size of the file, so this next script will send an HTTP request to the API and report back the status of the transcription, or print the output if it's complete.

Create a third Python file named get_transcription.py and put the following code into it.

import argparse import os import requests API_URL = "https://api.assemblyai.com/v2/" def get_transcription(transcription_id): """Requests the transcription from the API and returns the JSON response.""" endpoint = "".join([API_URL, "transcript/{}".format(transcription_id)]) headers = {"authorization": os.getenv('ASSEMBLYAI_KEY')} response = requests.get(endpoint, headers=headers) return response.json() if __name__ == "__main__": parser = argparse.ArgumentParser() parser.add_argument("transcription_id") args = parser.parse_args() transcription_id = args.transcription_id response_json = get_transcription(transcription_id) if response_json['status'] == "completed": for word in response_json['words']: print(word['text'], end=" ") else: print("current status of transcription request: {}".format( response_json['status']))

The code above has the same imports as the other scripts. In this new get_transcription function, we simply call the AssemblyAI API with our API key and the transcription identifier from the previous step (not the file identifier). We retrieve the JSON response and return it.

In the main function we handle the transcription identifier that is passed in as a command line argument and pass it into the get_transcription function. If the response JSON from the get_transcription function contains a completed status then we print the results of the transcription. Otherwise, print the current status which is either queued or processing before it is completed.

Call the script using the command line and the transcription identifier from the previous section:

python get_transcription.py TRANSCRIPTION_ID

If the service has not yet started working on the transcript then it will return queued like this:

current status of transcription request: queued

When the service is currently working on the audio file it will return processing:

current status of transcription request: processing

When the process is completed, our script will return the text of the transcription, like you see here:

An object relational mapper is a code library that automates the transfer of data stored in relational, databases into objects that are more commonly used in application code or EMS are useful because they provide a high level ...(output abbreviated)

That's it, we've got our transcription!

You may be wondering what to do if the accuracy isn't where you need it to be for your situation. That is where boosting accuracy for keywords or phrases and selecting a model that better matches your data come in. You can use either of those two methods to boost the accuracy of your recordings to an acceptable level for your situation.

What's next?

We just finished writing some scripts that call the AssemblyAI API to transcribe recordings with speech into text output.

Next, take a look at some of their more advanced documentation that goes beyond the basics in this tutorial:

Questions? Let me know via an issue ticket on the Full Stack Python repository, on Twitter @fullstackpython or @mattmakai. See something wrong with this post? Fork this page's source on GitHub and submit a pull request.

Categories: FLOSS Project Planets

Catalin George Festila: Python 3.8.5 : Pearson Product Moment Correlation with corrcoef from numpy.

Sat, 2020-08-08 23:36
The python package named numpy come with corrcoef function to return Pearson product-moment correlation coefficients. This method has a limitation in that it can compute the correlation matrix between two variables only. The full name is the Pearson Product Moment Correlation (PPMC). The PPMC is not able to tell the difference between dependent variables and independent variables. The
Categories: FLOSS Project Planets

Zato Blog: Zato and Docker installation options - general overview

Sat, 2020-08-08 21:47

Docker is a containerization platform that gained immense popularity in the IT world as a tool that can contain an application and help to deploy it to multiple environments.

History

Before Docker, there were different approaches whose goal was to obtain a reliable development and deployment process, which means testing and putting the application into production as a reproducible unit where the same result is always achieved.

The leading way to isolate and organize applications along with their their dependencies was to place each and every application in its own virtual machine. Physical hardware runs multiple such environments and the whole process is called virtualization.

Containerization

Containerization is a type of virtualization which brings virtualization to the operating system level. Containerization brings abstraction to the operating system, while virtualization brings abstraction to the hardware.

  • Containers are lightweight and typically faster than Virtual Machines
  • Containers share relevant libraries and resources
  • The boot up process is very fast since the applications run on the host kernel
Docker

Docker is a tool used to easily package, distribute and deploy applications with all its dependencies into a standardized unit called a Docker image. It provides the ability to create predictable environments which can be used for development, testing and deploying the application in multiple environments, including production.

Some Docker concepts:

  • Image: an executable package that includes everything needed to run the application
  • Container: a runtime instance of an image

A Docker container can be connected with other containers that provide functionality to the application like a database or a load balancer.

In Zato we create Docker images of two kinds:

The idea behind the Zato quickstart image is to very quickly provision a fully working all-in-one Zato cluster that can be used to test a new service locally or explore Zato but, because this is a real Zato cluster, it can also be used for production purposes.

The image used for cloud environments provides more flexibility and can be used to deploy an instance of a Zato component like server, scheduler or web admin into the Zato cluster, each under its own container. That is, whereas Zato quickstart contains all the Zato components in a single image, the cloud one splits the components into separate Docker containers.

Wrapping up

This is it for now - we will be talking about these two types of Zato images and how to use them in upcoming articles.

Categories: FLOSS Project Planets

PSF GSoC students blogs: Week 6 Check-in

Sat, 2020-08-08 11:02

What I have done this week

Works towards analyzing multistage dockerfile. I combined the draft PR and the review from my mentors, the new commit is the first step of my plan. We split the multistage dockerfile into seperate dockefiles for build. Here are the changes in the new commit.

1. Modified function check_multistage_dockerfile() to return.

2. Remove function split_multistage_dockerfile() since we are working on the building stage. split_multistage_dockerfile() can be improved on analyze stage.

To Do

1. Improve readability for function check_multistage_dockerfile().

2. Try build images and analyze on them.

Did I get stuck somewhere?

Not yet.

 

 

Categories: FLOSS Project Planets

Codementor: Send WhatsApp media/message using Python.

Sat, 2020-08-08 07:17
Send WhatsApp message/media using python.
Categories: FLOSS Project Planets

Sandipan Dey: Simulating a Turing Machine with Python and executing programs on it

Fri, 2020-08-07 20:01
In this article, we shall implement a basic version of a Turing Machine in python and write a few simple programs to execute them on the Turing machine.  This article is inspired by the edX / MITx course Paradox and Infinity and few of the programs to be executed on the (simulated) Turing machine are … Continue reading Simulating a Turing Machine with Python and executing programs on it
Categories: FLOSS Project Planets

Test and Code: 125: pytest 6 - Anthony Sottile

Fri, 2020-08-07 15:00

pytest 6 is out. Specifically, 6.0.1, as of July 31.

And there's lots to be excited about.
Anthony Sottile joins the show to discuss features, improvements, documentation updates and more.

Full release notes / changelog

Some of what we talk about:

  • How to update (at least, how I do it)
    • Run your test suites with 5.4.3 or whatever the last version you were using
    • Update to 6
    • Run again. Same output? Probably good.
    • If there are any warnings, maybe fix those.
    • You can also run with pytest -W error to turn warnings into errors.
    • Then find out all the cool stuff you can do now
  • New Features
    • pytest now supports pyproject.toml files for configuration.
      • but remember, toml syntax is different than ini files. mostly quotes are needed
    • pytest now includes inline type annotations and exposes them to user programs.
      • Most of the user-facing API is covered, as well as internal code.
    • New command-line flags --no-header and --no-summary
    • A warning is now shown when an unknown key is read from a config INI file.
      • The --strict-config flag has been added to treat these warnings as errors.
    • New required_plugins configuration option allows the user to specify a list of plugins, including version information, that are required for pytest to run. An error is raised if any required plugins are not found when running pytest.
  • Improvements
    • You can now pass output to things like less and head that close the pipe passed to them.
      • thank you!!!
    • Improved precision of test durations measurement.
      • use --durations=10 -vv to capture and show durations
    • Rich comparison for dataclasses and attrs-classes is now recursive.
    • pytest --version now displays just the pytest version,
      • while pytest --version --version displays more verbose information including plugins.
    • --junitxml now includes the exception cause in the message XML attribute for failures during setup and teardown.
  • Improved Documentation
    • Add a note about --strict and --strict-markers and the preference for the latter one.
    • Explain indirect parametrization and markers for fixtures.
  • Bug Fixes
  • Deprecations
  • Trivial/Internal Changes
  • Breaking Changes you might need to care about before upgrading
    • PytestDeprecationWarning are now errors by default.
    • -k and -m internals were rewritten to stop using eval(), this results in a few slight changes but overall makes them much more consistent
    • testdir.run().parseoutcomes() now always returns the parsed nouns in plural form.
      • I'd say that's an improvement

Special Guest: Anthony Sottile.

Sponsored By:

Support Test & Code : Python Testing for Software Engineering

Links:

<p>pytest 6 is out. Specifically, 6.0.1, as of July 31.<br><br> And there&#39;s lots to be excited about.<br> Anthony Sottile joins the show to discuss features, improvements, documentation updates and more.</p> <p><a href="https://docs.pytest.org/en/stable/changelog.html" rel="nofollow">Full release notes / changelog</a></p> <p>Some of what we talk about:</p> <ul> <li>How to update (at least, how I do it) <ul> <li>Run your test suites with 5.4.3 or whatever the last version you were using</li> <li>Update to 6</li> <li>Run again. Same output? Probably good.</li> <li>If there are any warnings, maybe fix those.</li> <li>You can also run with <code>pytest -W error</code> to turn warnings into errors.</li> <li>Then find out all the cool stuff you can do now</li> </ul></li> <li>New Features <ul> <li>pytest now supports pyproject.toml files for configuration. <ul> <li>but remember, toml syntax is different than ini files. mostly quotes are needed </li> </ul></li> <li>pytest now includes inline type annotations and exposes them to user programs. <ul> <li>Most of the user-facing API is covered, as well as internal code.</li> </ul></li> <li>New command-line flags <code>--no-header</code> and <code>--no-summary</code></li> <li>A warning is now shown when an unknown key is read from a config INI file. <ul> <li>The <code>--strict-config</code> flag has been added to treat these warnings as errors.</li> </ul></li> <li>New required_plugins configuration option allows the user to specify a list of plugins, including version information, that are required for pytest to run. An error is raised if any required plugins are not found when running pytest.</li> </ul></li> <li>Improvements <ul> <li>You can now pass output to things like less and head that close the pipe passed to them. <ul> <li>thank you!!!</li> </ul></li> <li>Improved precision of test durations measurement. <ul> <li>use <code>--durations=10 -vv</code> to capture and show durations</li> </ul></li> <li>Rich comparison for dataclasses and attrs-classes is now recursive.</li> <li>pytest <code>--version</code> now displays just the pytest version, <ul> <li>while <code>pytest --version --version</code> displays more verbose information including plugins. </li> </ul></li> <li><code>--junitxml</code> now includes the exception cause in the message XML attribute for failures during setup and teardown.</li> </ul></li> <li>Improved Documentation <ul> <li>Add a note about <code>--strict</code> and <code>--strict-markers</code> and the preference for the latter one.</li> <li>Explain indirect parametrization and markers for fixtures.</li> </ul></li> <li>Bug Fixes</li> <li>Deprecations</li> <li>Trivial/Internal Changes</li> <li>Breaking Changes you might need to care about before upgrading <ul> <li>PytestDeprecationWarning are now errors by default. <ul> <li>Check the <a href="https://docs.pytest.org/en/latest/deprecations.html" rel="nofollow">deprecations and removals</a> page if you are curious.</li> </ul></li> <li><code>-k</code> and <code>-m</code> internals were rewritten to stop using eval(), this results in a few <em>slight</em> changes but overall makes them much more consistent</li> <li><code>testdir.run().parseoutcomes()</code> now always returns the parsed nouns in plural form. <ul> <li>I&#39;d say that&#39;s an improvement</li> </ul></li> </ul></li> </ul><p>Special Guest: Anthony Sottile.</p><p>Sponsored By:</p><ul><li><a href="https://testandcode.com/datadog" rel="nofollow">Datadog</a>: <a href="https://testandcode.com/datadog" rel="nofollow">Modern monitoring & security. See inside any stack, any app, at any scale, anywhere. Visit testandcode.com/datadog to get started.</a></li></ul><p><a href="https://www.patreon.com/testpodcast" rel="payment">Support Test & Code : Python Testing for Software Engineering</a></p><p>Links:</p><ul><li><a href="https://docs.pytest.org/en/stable/changelog.html" title="pytest Changelog / Release Notes" rel="nofollow">pytest Changelog / Release Notes</a></li><li><a href="https://docs.pytest.org/en/latest/deprecations.html" title="Deprecations and Removals — pytest documentation" rel="nofollow">Deprecations and Removals — pytest documentation</a></li></ul>
Categories: FLOSS Project Planets

Python Engineering at Microsoft: Need an Intro to VS Code? Let Tech with Tim Help!

Fri, 2020-08-07 11:58

We have been spoiled to have Tim Ruscica, founder and star of the very informative and popular “Tech with Tim” YouTube channel, work on our team as a Software Engineering intern this summer! During his time here he implemented several features for our extension, including the much desired export notebook to HTML and PDF, as well as influence the discussion and design of other features we have been actively working on and shipping. We’re honored to have him spotlight a few of his favorite VS Code Python Extension features (not just those he worked on) in a recent video! Please check it out and give Tim some much deserved kudos!

 

If you’re interested in learning about something specific, follow the timestamps below!

00:22 – VSCode Python Setup

00:50 – Jupyter Notebooks

01:38 – Interactive Window

03:28 – Variable Explorer

04:41 – Run By Line

05:50 – Dataframe Viewer

06:10 – Notebook Exports

06:56 – Gather

08:28 – Pylance

 

Did Tim’s video give you an idea for a feature? Let us know here!

The post Need an Intro to VS Code? Let Tech with Tim Help! appeared first on Python.

Categories: FLOSS Project Planets

Stack Abuse: Integrating H2 with Python and Flask

Fri, 2020-08-07 08:30
Introduction

H2 is a lightweight database server written in Java. It can be embedded in Java applications, or run as a standalone server.

In this tutorial, we'll review why H2 can be a good option for your projects. We'll also learn how to integrate H2 with Python by building a simple Flask API.

The Features of H2

H2 was built with performance in mind.

"H2 is a combination of: fast, stable, easy to use, and features".

Although H2 is prominent mainly because it can be embedded in Java applications, it has some interesting features that also apply to its server version. Let's see some of them next.

Size and Performance

The .jar file used for the server version is around 2MB. We can download it from the H2 site, bundled with extra scripts and documentation. If we search in Maven Central though, we can download the .jar file on its own.

H2 performance shines in its embedded version. Even so, the official benchmark shows that its client-server version is also impressive.

In-Memory Databases and Encryption

In-Memory databases are not persistent. All data is stored in memory, so speed is greatly increased.

The H2 site explains that In-Memory databases are particularly useful when prototyping, or when using read-only databases.

Encryption is another useful feature to protect data at rest. Databases can be encrypted with the AES-128 algorithm.

Other Useful Features

H2 also provides a cluster mode, the ability to run multiple servers and connect them together. Writes are done in all servers at the same time, while reads are done from the first server in the cluster.

H2 surprises for its simplicity. It provides several useful features and it's easy to set up.

Let's start an H2 server in preparation for the following sections:

$ java -cp ./h2-1.4.200.jar org.h2.tools.Server -tcp -tcpAllowOthers -tcpPort 5234 -baseDir ./ -ifNotExists

The arguments that start with tcp enable communication to the server. The ifNotExists argument allows the database to be created when accessing it for the first time.

Description of the API and General Diagram

Let's suppose we are writing an API to register all exoplanets found to date. Exoplanets are planets found outside our Solar System, orbiting other stars.

This is our simple API definition, a CRUD for one resource:

This definition along with the rest of the code we'll see next is available in this GitHub repo.

This is how our application will look like at the end of this tutorial:

Left of the diagram we see the API Client. That client can be the "Try it out" function of the Swagger Editor, or any other client, like Postman or cURL.

On the other end we find the H2 database server, running on TCP port 5234 as explained above.

Finally, our application in the middle is composed of three Python files. The first one will have the Flask app that will answer all REST API requests. All endpoints we described in the definition above will be added to this file.

The second file will have the persistence, functions that access the database to execute the CRUD operations, using the JayDeBeApi package.

Lastly, a third file will contain a schema representing the resource the API manages, the Exoplanet. We'll use the Marshmallow package to represent that schema. The first two python files will use this schema to represent resources and pass them to each other.

Let's start off with the persistence file.

Database Schema

To store the Exoplanet resource to an H2 database we should write the basic CRUD functions first. Let's start by writing the creation of the database. We use the JayDeBeApi package to access databases through JDBC:

import jaydebeapi def initialize(): _execute( ("CREATE TABLE IF NOT EXISTS exoplanets (" " id INT PRIMARY KEY AUTO_INCREMENT," " name VARCHAR NOT NULL," " year_discovered SIGNED," " light_years FLOAT," " mass FLOAT," " link VARCHAR)")) def _execute(query, returnResult=None): connection = jaydebeapi.connect( "org.h2.Driver", "jdbc:h2:tcp://localhost:5234/exoplanets", ["SA", ""], "../h2-1.4.200.jar") cursor = connection.cursor() cursor.execute(query) if returnResult: returnResult = _convert_to_schema(cursor) cursor.close() connection.close() return returnResult

The initialize() function is simple enough because of the helper functions after. It creates the exoplanets table if it doesn't exist already. This function should be executed before our API starts receiving requests. We'll see later where to do that with Flask.

The _execute() function contains the connection string and credentials to access the database server. It is simpler for this example, but there is room for improvement regarding security. We could save our credentials elsewhere, like environment variables for example.

Also, we added the path to the H2 jar file to the connect() method, as it has the driver we need to connect to H2 - org.h2.Driver.

The JDBC connection string ends in /exoplanets. This means that when connecting for the first time a database called exoplanets will be created.

You may have noticed that _execute() can return the result of the SQL query using the _convert_to_schema() function. Let's now see how that function works.

Marshmallow Schemas and CRUD Database Functions

Some SQL queries return tabular results, particularly the SELECT statement. JayDeBeApi will format those results as a list of tuples. For example, for the schema defined in the last section we could get a result similar to this:

>>> connection = jaydebeapi.connect(... >>> cursor = connection.cursor() >>> cursor.execute("SELECT * FROM exoplanets") >>> cursor.fetchall() [(1, 'Sample1', 2019, 4.5, 1.2, 'http://sample1.com')]

Nothing's stopping us from managing results in this format and eventually return it to the API client. But looking ahead we know we will use Flask, so it would be good to already return results in a format Flask recommends.

In particular, we'll be using Flask-RESTful to facilitate the use of API routes. That package recommends to use Marshmallow to parse requests. This step allows for normalization of the objects. This way we can discard unknown properties and highlight validation errors, for example.

Let's see how the Exoplanet class would look so we can discuss further:

from marshmallow import Schema, fields, EXCLUDE class ExoplanetSchema(Schema): id = fields.Integer(allow_none=True) name = fields.Str(required=True, error_messages={"required": "An exoplanet needs at least a name"}) year_discovered = fields.Integer(allow_none=True) light_years = fields.Float(allow_none=True) mass = fields.Float(allow_none=True) link = fields.Url(allow_none=True) class Meta: unknown = EXCLUDE

The definition of the properties looks familiar. It's the same as the database schema, including the definition of required fields. All fields have a type that defines some default validation. For example, the link field is defined as an URL, so a string that doesn't look like an URL won't be valid.

Specific error messages can also be included here, like the validation for a name.

For this example project, we want to discard, or exclude, all unknown fields and API client may send erroneously. This is achieved in the Meta nested class.

Now we can use the load() and loads() Marshmallow methods to convert and validate our resources.

Now that we're acquainted with Marshmallow, we can explain what the _convert_to_schema() does:

def _convert_to_schema(cursor): column_names = [record[0].lower() for record in cursor.description] column_and_values = [dict(zip(column_names, record)) for record in cursor.fetchall()] return ExoplanetSchema().load(column_and_values, many=True)

In JayDeBeApi, column names are saved in the description field of the cursor, while the data can be obtained with the fetchall() method. We used list comprehensions in the first two lines to get the column names and values, and zip() to merge them.

The last line takes the merged result and converts them to ExoplanetSchema objects that Flask can further process.

Now that we explained the _execute() function and the ExoplanetSchema class, let's see all the CRUD database functions:

def get_all(): return _execute("SELECT * FROM exoplanets", returnResult=True) def get(Id): return _execute("SELECT * FROM exoplanets WHERE id = {}".format(Id), returnResult=True) def create(exoplanet): count = _execute("SELECT count(*) AS count FROM exoplanets WHERE name LIKE '{}'".format(exoplanet.get("name")), returnResult=True) if count[0]["count"] > 0: return columns = ", ".join(exoplanet.keys()) values = ", ".join("'{}'".format(value) for value in exoplanet.values()) _execute("INSERT INTO exoplanets ({}) VALUES({})".format(columns, values)) return {} def update(exoplanet, Id): count = _execute("SELECT count(*) AS count FROM exoplanets WHERE id = {}".format(Id), returnResult=True) if count[0]["count"] == 0: return values = ["'{}'".format(value) for value in exoplanet.values()] update_values = ", ".join("{} = {}".format(key, value) for key, value in zip(exoplanet.keys(), values)) _execute("UPDATE exoplanets SET {} WHERE id = {}".format(update_values, Id)) return {} def delete(Id): count = _execute("SELECT count(*) AS count FROM exoplanets WHERE id = {}".format(Id), returnResult=True) if count[0]["count"] == 0: return _execute("DELETE FROM exoplanets WHERE id = {}".format(Id)) return {}

All functions are mainly SQL queries, but create() and update() deserve some more explanation.

The INSERT SQL statement can receive column and values separated, in the form INSERT INTO table (column1Name) VALUES ('column1Value'). We can use the join() function to merge all columns and separate them with commas, and do something similar to join all values we want to insert.

The UPDATE SQL statement is a bit more complex. Its form is UPDATE table SET column1Name = 'column1Value'. So we need to alternate keys and values, and we did it using the zip() function.

All these functions return None when there is a problem. Later when we call them we will have to check for that value.

Let's save all database functions on its own file, persistence.py, so we can add some context when we call the functions, like this:

import persistence persistence.get_all() REST API With Flask

Now that we wrote a layer to abstract the access to the database, we are ready to write the REST API. We'll use the Flask and Flask-RESTful packages to make our definition as easy as possible. As we learned before, we'll also use Marshmallow to validate resources.

Flask-RESTful requires to define one class per API resource, in our case the Exoplanet resource only. Then we can associate that resource with a route like this:

from flask import Flask from flask_restful import Resource, Api app = Flask(__name__) api = Api(app) class Exoplanet(Resource): # ... api.add_resource(Exoplanet, "/exoplanets", "/exoplanets/<int:Id>")

This way all our routes, /exoplanets and /exoplanets/<int:Id> will be directed to the class we defined.

For example, the GET /exoplanets endpoint will be answered by a method called get() inside the Exoplanet class. Because we also have the GET /exoplanet/<Id> endpoint, that get() method must have an optional parameter called Id.

Let's see the whole class to understand this better:

from flask import request from flask_restful import Resource, abort from marshmallow import ValidationError import persistence class Exoplanet(Resource): def get(self, Id=None): if Id is None: return persistence.get_all() exoplanet = persistence.get(Id) if not exoplanet: abort(404, errors={"errors": {"message": "Exoplanet with Id {} does not exist".format(Id)}}) return exoplanet def post(self): try: exoplanet = ExoplanetSchema(exclude=["id"]).loads(request.json) if not persistence.create(exoplanet): abort(404, errors={"errors": {"message": "Exoplanet with name {} already exists".format(request.json["name"])}}) except ValidationError as e: abort(405, errors=e.messages) def put(self, Id): try: exoplanet = ExoplanetSchema(exclude=["id"]).loads(request.json) if not persistence.update(exoplanet, Id): abort(404, errors={"errors": {"message": "Exoplanet with Id {} does not exist".format(Id)}}) except ValidationError as e: abort(405, errors=e.messages) def delete(self, Id): if not persistence.delete(Id): abort(404, errors={"errors": {"message": "Exoplanet with Id {} does not exist".format(Id)}})

The remaining HTTP verbs are processed in the same way as GET, by the methods with name post(), put() and delete() .

As we said before, logic errors when accessing the database will cause the functions to return None. Those errors are captured here when needed.

Also, exceptions that represent validation errors are triggered by Marshmallow, so those errors are also captured and returned to the user along with an appropriate return error.

Conclusion

H2 is a useful database server, performant and easy to use. Although it is a Java package, it can also run as a standalone server, so we can use it in Python with the JayDeBeApi package.

In this tutorial we defined a simple CRUD application to illustrate how to access the database, and which functions are available. After that, we defined a REST API with Flask and Flask-RESTful.

Although several concepts were omitted for the sake of brevity, like authentication and paging, this tutorial is a good reference to start using H2 in our Flask projects.

Categories: FLOSS Project Planets

Real Python: The Real Python Podcast – Episode #21: Exploring K-means Clustering and Building a Gradebook With Pandas

Fri, 2020-08-07 08:00

Do you want to learn the how and when of implementing K-means clustering in Python? Would you like to practice your pandas skills with a real-world project? This week on the show, David Amos is back with another batch of PyCoder’s Weekly articles and projects.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

PSF GSoC students blogs: Week 10 Blog

Thu, 2020-08-06 17:05

Hello everyone

I know I am late for the weekly blog. My apologies for that.
The previous week was my first week post my exams. I had mentioned in the last blog that I will be working on the documentation for panda3d.
So, just as I mentioned, I was able to complete the documentation part in one week. The documentation includes documentation for both python and c++, the languages supported by panda3d for game development. The appropriate code snippets have been included along with the screenshots of the corresponding output.
The challenges were to understand the reStructured text format by Sphinx ans write the code for documentation with proper identation and tags so as to build separate docs for python and c++.
You may find the Pull Request here: #71

This week I am working on the sample code. Some part of work has been done and hopefully it will be completed by the weekend. I will share more about the same in my next blog.

Thank you!

Stay Safe!

Categories: FLOSS Project Planets

Codementor: How to use DLLs/COM objects from Python or - How to send a fax with Python

Thu, 2020-08-06 13:34
Leverage DLLs from Python. Send faxes with Python without writing any code to send faxes.
Categories: FLOSS Project Planets

PyCharm: Webinar: “Django Database Performance Tips” with Andrew Brookins

Thu, 2020-08-06 08:23

Django is one of the most popular Python Web Frameworks. However, optimizing django performance can often be difficult. Most performance problems boil down to one thing: the database. According to our guest speaker, Andrew Brookins, the only way to master Django performance is to master database performance.

In this Webinar, Andrew is going to take us on a journey through an app that he created to demonstrate all the bottlenecks that arise when trying to optimize your Django app.

  • Monday, August 10
  • 7:00PM CEST – 1:00PM Eastern Daylight Time
  • Register here

Outline
  • Querying
    • Tackling the N+1 Problem: Reducing the number of queries to make your pages faster.
    • Annotations: Getting your database to do the heavy lifting for you
    • Iterator(): Get Django to chuck responses from your database
  • Indexing
    • Covering Indexes: Turning queries that take several hundred milliseconds to tens of milliseconds.
    • Materialized Views: One of the most powerful ways to have large queries created for you
  • Caching with Redis
    • Using Redis as a Cache
    • Custom Redis Authentication
About the Speaker

Andrew Brookins has over a decade of experience with Django and Relational Databases. That experience spans DevOps, application development, and platform engineering. He has worked on ecommerce sites, mobile APIs, comic book readers, particle simulation systems, learning and volunteering apps, and now works on a Databases as a Service at Redis Labs.

Andrew has also authored "Temple of Django Database Performance" which dives into the even greater detail than this webinar does. Furthermore, he has recently released a course on "Redis for Python Developers" at Redis University.

Andrew lives in Portland, Oregon with his wife and two wonderful children.

Categories: FLOSS Project Planets

Talk Python to Me: #276 Geekout: Life in the solar system and beyond

Thu, 2020-08-06 04:00
We're back with another GeekOut episode. Richard Campbell, a developer and podcaster who also dives deep into science and tech topics, is back for our second GeekOut episode. Last time we geeked out about the real science and progress around a moon base. This time it's why is there life on Earth, where could it be or have been in the solar system, and beyond. <br/> <br/> In case you didn't catch the first GeekOut, episode 253, this one is more of a general science and tech episode. I love digging into the deep internals of all the tools of the Python space, but given all that is going on in the world, I thought it'd be fun to take a step back and just enjoy some fun geekery and give you all something to just sit back and let your mind dream.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Richard on Twitter</b>: <a href="https://twitter.com/richcampbell" target="_blank" rel="noopener">@richcampbell</a><br/> <b>All Richard's GeekOut Episodes</b>: <a href="http://geekout.show" target="_blank" rel="noopener">geekout.show</a><br/> <b>Moonbase Geekout Episode</b>: <a href="https://talkpython.fm/253" target="_blank" rel="noopener">talkpython.fm/253</a><br/> <b>High Altitude Venus Operational Concept (HAVOC)</b>: <a href="https://sacd.larc.nasa.gov/smab/havoc/" target="_blank" rel="noopener">sacd.larc.nasa.gov</a><br/> <b>New Horizons</b>: <a href="https://solarsystem.nasa.gov/missions/new-horizons/in-depth/" target="_blank" rel="noopener">solarsystem.nasa.gov</a><br/> <b>The Planets: Saturn - NOVA documentary (Cassini)</b>: <a href="https://www.pbs.org/wgbh/nova/video/the-planets-saturn/" target="_blank" rel="noopener">pbs.org</a><br/> <b>Mission to Jupiter - Galileo</b>: <a href="https://www.jpl.nasa.gov/missions/galileo/" target="_blank" rel="noopener">jpl.nasa.gov</a><br/></div><br/> <strong>Sponsors</strong><br/> <br/> <a href='https://talkpython.fm/brilliant'>Brilliant</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>
Categories: FLOSS Project Planets

Pages