FLOSS Project Planets

Anarcat: Mumble dreams

Planet Python - Thu, 2020-04-09 12:08

With everyone switching to remote tools for social distancing, I've been using Mumble more and more. That's partly by choice -- I don't like videoconferencing much, frankly -- and partly by necessity: sometimes my web browser fails and Mumble is generally more reliable.

Some friend on a mailing list recently asked "shouldn't we make Mumble better?" and opened the door for me to go on a long "can I get a pony?" email. Because I doubt anyone on that mailing list has the time or capacity to actually fix those issues, I figured I would copy this to a broader audience in the hope that someone else would pick it up.

Why Mumble rocks

Before I go on with the UI critique, I should show why care: Mumble is awesome.

When you do manage to configure it correctly, Mumble just works; it's highly reliable. It uses little CPU, both on the client and the server side, and can have rooms with tens if not hundreds of participants. The server can be easily installed and configured: there's a Debian package and resource requirements are minimal. It's basically network-bound. There are at least three server implementations, the official one called Murmur, the minimalist umurmur and Grumble, a Go rewrite.

It has great quality: echo canceling, when correctly configured, is solid and latency is minimal. It has "overlays" so you can use it while gaming or demo'ing in full screen while still having an idea of who's talking. It also supports positional audio for gaming that integrates with popular games like Counterstrike or Half-Life.

It's moderately secure: it doesn't support end-to-end encryption, but client/server communication is encrypted with TLS. It supports a server password and some moderation mechanisms.

UI improvements

Mumble should be smarter about a bunch of things. Having all those settings is nice for geeky control freaks, but it makes the configuration absolutely unusable for most people. Hide most settings by default, and make better defaults.

Specifically, those should be on by default:

  • RNNoise
  • echo cancellation (the proper "monitor" channels)
  • pre-configured shortcut for PTT (Push To Talk) -- right-shift is my favorite
  • "double-PTT" to hold it enabled
  • be more silent by default (I understand why it would want to do voice synthesis, but it would need to be much better at it before it's default)

The echo test should be more accessible, one or two clicks away from the main UI. I have only found out about that feature when someone told me where to find it. This basically means to take it out of the settings page and into its own dialog.

The basic UI should be much simpler. It could look something like Jitsi: just one giant mute button with a list of speakers. Basically:

  1. Take that status bar and make it use the entire space of the main window

  2. Push the chat and room list dialog to separate, optional dialog (e.g. the room list could be a popup on login, but we don't need to continuously see the damn thing)

  3. Show the name of the person talking in the main UI, along with other speakers (Big Blue Button does this well: just a label that fades away with time after a person talks)

Some features could be better explained. For example, the "overlay" feature makes no sense at all for most users. It only makes sense when you're a gamer and use Mumble alongside another full-screen program, to show you who's talking.

Improved authentication. The current authentication systems in Mumble are somewhat limited: the server can have a shared password to get access to it, and from there it's pretty much free-for-all. There are client certificates but those are hard to understand and the most common usage scenario is that someone manages to configure it once, forgets about it and then cannot login again with the same username.

It should be easier to get the audio right. Now, to be fair, this is hard to do in any setup, and Mumble is only a part of this. There are way too many moving parts in Linux for this to be easy: between your hardware, ALSA drivers, Pulseaudio mixers and Mumble, too many things can go wrong. So this is a general problem when doing multimedia in general, and the Linux ecosystem in particular, but Mumble is especially hard to configure in there.

Improved speaker stats. When you right-click on a user in Mumble, you get detailed stats about the user: packet loss, latency, bandwidth, codecs... It's pretty neat. But that is hard to parse for a user. Jitsi, in contrast, shows a neat little "bar graph" (similar to what you get on a cell phone) with a color code to show network conditions for that user. Then you can drill down to show more information. Having that info for the user would be really useful to figure out which user is causing that echo or latency. Heck, while I'm dreaming, we could do the same thing Jitsi and tell the user when we detect too much noise on their side and suggest muting!

There's probably more UI issues, but at that point you have basically rebuilt the entire user interface. This problem is hard to fix because UX people are unlikely to have the skills required to hack at an (old) Qt app, and C++ hackers are unlikely to have the best UX skills...

Missing features

Video. It has been on the roadmap since 2011, so I'm not holding my breath. It is, obviously, the key feature missing from the software when compared to other conferencing tools and it's nice to see they are considering it. Screensharing and whiteboarding would also be a nice addition. Unfortunately, all that is a huge undertaking and it's unlikely to happen in the short term. And even if it does, it's possible hard-core Mumble users would be really upset at the change...

A good web app -- a major blocker to the adoption of Mumble is the need for that complex app. If users could join just with a web browser, adoption would be much easier. There is a web app called mumble-web out there, but it seems to work only for listening as there are numerous problems with recording: quality issues, audio glitches, voice activation, voice activation.. The CCC seems to be using that app to stream talk translation, so that part supposedly works correctly.

Dial-in -- allow plain old telephones to call into conferences. There seems to be a program called mumsi that can do this, but it's unmaintained and it's unclear if any of the forks work at all.

Caveats

Now the above will probably not happen soon. Unfortunately, Mumble has had trouble with their release process recently. It took them a long time to even agree on releasing 1.3, and when they did agree, it took them a long time again to actually do the release. There has been much more activity on the Mumble client and web app recently, so hopefully I will be proven wrong. The 1.3.1 release actually came out recently which is encouraging.

All in all, mumble has some deeply ingrained UI limitations. it's built like an app from the 1990, all the way down to the menu system and "status bar" buttons. It's definitely not intuitive for a new user and while there's an audio wizard that can help you get started, it doesn't always work and can be confusing in itself.

I understand that I'm just this guy saying "please make this for me ktxbye". I'm not writing this as a critic of Mumble: I love the little guy, the underdog. Mumble has been around forever and it kicks ass. I'm writing this in a spirit of solidarity, in the hope the feedback can be useful and to provide useful guidelines on how things could be improved. I wish I had the time to do this myself and actually help the project beyond just writing, but unfortunately the reality is I'm a poor UI designer and I have little time to contribute to more software projects.

So hopefully someone could take those ideas and make Mumble even greater. And if not, we'll just have to live with it.

Thanks to all the Mumble developers who, over all those years, managed to make and maintain such an awesome product. You rock!

Categories: FLOSS Project Planets

Antoine Beaupré: Mumble dreams

Planet Debian - Thu, 2020-04-09 12:08

With everyone switching to remote tools for social distancing, I've been using Mumble more and more. That's partly by choice -- I don't like videoconferencing much, frankly -- and partly by necessity: sometimes my web browser fails and Mumble is generally more reliable.

Some friend on a mailing list recently asked "shouldn't we make Mumble better?" and opened the door for me to go on a long "can I get a pony?" email. Because I doubt anyone on that mailing list has the time or capacity to actually fix those issues, I figured I would copy this to a broader audience in the hope that someone else would pick it up.

Why Mumble rocks

Before I go on with the UI critique, I should show why care: Mumble is awesome.

When you do manage to configure it correctly, Mumble just works; it's highly reliable. It uses little CPU, both on the client and the server side, and can have rooms with tens if not hundreds of participants. The server can be easily installed and configured: there's a Debian package and resource requirements are minimal. It's basically network-bound. There are at least three server implementations, the official one called Murmur, the minimalist umurmur and Grumble, a Go rewrite.

It has great quality: echo canceling, when correctly configured, is solid and latency is minimal. It has "overlays" so you can use it while gaming or demo'ing in full screen while still having an idea of who's talking. It also supports positional audio for gaming that integrates with popular games like Counterstrike or Half-Life.

It's moderately secure: it doesn't support end-to-end encryption, but client/server communication is encrypted with TLS. It supports a server password and some moderation mechanisms.

UI improvements

Mumble should be smarter about a bunch of things. Having all those settings is nice for geeky control freaks, but it makes the configuration absolutely unusable for most people. Hide most settings by default, and make better defaults.

Specifically, those should be on by default:

  • RNNoise
  • echo cancellation (the proper "monitor" channels)
  • pre-configured shortcut for PTT (Push To Talk) -- right-shift is my favorite
  • "double-PTT" to hold it enabled
  • be more silent by default (I understand why it would want to do voice synthesis, but it would need to be much better at it before it's default)

The echo test should be more accessible, one or two clicks away from the main UI. I have only found out about that feature when someone told me where to find it. This basically means to take it out of the settings page and into its own dialog.

The basic UI should be much simpler. It could look something like Jitsi: just one giant mute button with a list of speakers. Basically:

  1. Take that status bar and make it use the entire space of the main window

  2. Push the chat and room list dialog to separate, optional dialog (e.g. the room list could be a popup on login, but we don't need to continuously see the damn thing)

  3. Show the name of the person talking in the main UI, along with other speakers (Big Blue Button does this well: just a label that fades away with time after a person talks)

Some features could be better explained. For example, the "overlay" feature makes no sense at all for most users. It only makes sense when you're a gamer and use Mumble alongside another full-screen program, to show you who's talking.

Improved authentication. The current authentication systems in Mumble are somewhat limited: the server can have a shared password to get access to it, and from there it's pretty much free-for-all. There are client certificates but those are hard to understand and the most common usage scenario is that someone manages to configure it once, forgets about it and then cannot login again with the same username.

It should be easier to get the audio right. Now, to be fair, this is hard to do in any setup, and Mumble is only a part of this. There are way too many moving parts in Linux for this to be easy: between your hardware, ALSA drivers, Pulseaudio mixers and Mumble, too many things can go wrong. So this is a general problem when doing multimedia in general, and the Linux ecosystem in particular, but Mumble is especially hard to configure in there.

Improved speaker stats. When you right-click on a user in Mumble, you get detailed stats about the user: packet loss, latency, bandwidth, codecs... It's pretty neat. But that is hard to parse for a user. Jitsi, in contrast, shows a neat little "bar graph" (similar to what you get on a cell phone) with a color code to show network conditions for that user. Then you can drill down to show more information. Having that info for the user would be really useful to figure out which user is causing that echo or latency. Heck, while I'm dreaming, we could do the same thing Jitsi and tell the user when we detect too much noise on their side and suggest muting!

There's probably more UI issues, but at that point you have basically rebuilt the entire user interface. This problem is hard to fix because UX people are unlikely to have the skills required to hack at an (old) Qt app, and C++ hackers are unlikely to have the best UX skills...

Missing features

Video. It has been on the roadmap since 2011, so I'm not holding my breath. It is, obviously, the key feature missing from the software when compared to other conferencing tools and it's nice to see they are considering it. Screensharing and whiteboarding would also be a nice addition. Unfortunately, all that is a huge undertaking and it's unlikely to happen in the short term. And even if it does, it's possible hard-core Mumble users would be really upset at the change...

A good web app -- a major blocker to the adoption of Mumble is the need for that complex app. If users could join just with a web browser, adoption would be much easier. There is a web app called mumble-web out there, but it seems to work only for listening as there are numerous problems with recording: quality issues, audio glitches, voice activation, voice activation.. The CCC seems to be using that app to stream talk translation, so that part supposedly works correctly.

Dial-in -- allow plain old telephones to call into conferences. There seems to be a program called mumsi that can do this, but it's unmaintained and it's unclear if any of the forks work at all.

Caveats

Now the above will probably not happen soon. Unfortunately, Mumble has had trouble with their release process recently. It took them a long time to even agree on releasing 1.3, and when they did agree, it took them a long time again to actually do the release. There has been much more activity on the Mumble client and web app recently, so hopefully I will be proven wrong. The 1.3.1 release actually came out recently which is encouraging.

All in all, mumble has some deeply ingrained UI limitations. it's built like an app from the 1990, all the way down to the menu system and "status bar" buttons. It's definitely not intuitive for a new user and while there's an audio wizard that can help you get started, it doesn't always work and can be confusing in itself.

I understand that I'm just this guy saying "please make this for me ktxbye". I'm not writing this as a critic of Mumble: I love the little guy, the underdog. Mumble has been around forever and it kicks ass. I'm writing this in a spirit of solidarity, in the hope the feedback can be useful and to provide useful guidelines on how things could be improved. I wish I had the time to do this myself and actually help the project beyond just writing, but unfortunately the reality is I'm a poor UI designer and I have little time to contribute to more software projects.

So hopefully someone could take those ideas and make Mumble even greater. And if not, we'll just have to live with it.

Thanks to all the Mumble developers who, over all those years, managed to make and maintain such an awesome product. You rock!

Categories: FLOSS Project Planets

Test and Code: 108: PySpark - Jonathan Rioux

Planet Python - Thu, 2020-04-09 11:00

Apache Spark is a unified analytics engine for large-scale data processing.
PySpark blends the powerful Spark big data processing engine with the Python programming language to provide a data analysis platform that can scale up for nearly any task.

Johnathan Rioux, author of "PySpark in Action", joins the show and gives us a great introduction of Spark and PySpark to help us decide how to get started and decide whether or not to decide if Spark and PySpark are right you.

Special Guest: Jonathan Rioux.

Sponsored By:

Support Test & Code : Python Testing for Software Engineers

Links:

<p>Apache Spark is a unified analytics engine for large-scale data processing.<br> PySpark blends the powerful Spark big data processing engine with the Python programming language to provide a data analysis platform that can scale up for nearly any task.</p> <p>Johnathan Rioux, author of &quot;PySpark in Action&quot;, joins the show and gives us a great introduction of Spark and PySpark to help us decide how to get started and decide whether or not to decide if Spark and PySpark are right you.</p><p>Special Guest: Jonathan Rioux.</p><p>Sponsored By:</p><ul><li><a href="https://testandcode.com/pycharm" rel="nofollow">PyCharm Professional</a>: <a href="https://testandcode.com/pycharm" rel="nofollow">Try PyCharm Pro for 4 months and learn how PyCharm will save you time.</a> Promo Code: TESTNCODE2020</li></ul><p><a href="https://www.patreon.com/testpodcast" rel="payment">Support Test & Code : Python Testing for Software Engineers</a></p><p>Links:</p><ul><li><a href="https://www.manning.com/books/pyspark-in-action" title="PySpark in Action" rel="nofollow">PySpark in Action</a></li><li><a href="https://spark.apache.org/docs/latest/api/python/index.html" title="Spark" rel="nofollow">Spark</a></li><li><a href="https://spark.apache.org/docs/latest/api/python/pyspark.html" title="PySpark documentation" rel="nofollow">PySpark documentation</a></li><li><a href="https://www.youtube.com/watch?v=o64FV-ez6Gw" title="Joel Grus, livecoding" rel="nofollow">Joel Grus, livecoding</a></li></ul>
Categories: FLOSS Project Planets

Matt Layman: Onboarding Forms - Building SaaS #51

Planet Python - Wed, 2020-04-08 20:00
In this episode, we added the first form to collect data in the onboarding flow. We used a CreateView and defined all the fields that are needed in the HTML form. I started by filling in the HTML form structure of the page. Once the dummy version was in place, we changed from a TemplateView to a CreateView and began fixing each configuration error that the new view type reported as missing like missing a model field declaration.
Categories: FLOSS Project Planets

Promet Source: Pandemic Reveals New Potential for Data and Analytics

Planet Drupal - Wed, 2020-04-08 19:57
COVID-19 has fueled, among many things, a hefty appetite for data and analytics. Having witnessed a rapid-fire evolution from a few, isolated cases in another corner of the world, to a pandemic that has the globe in its grips, data visualizations are now helping to tell the story and reveal the kinds of big data insights that are now possible. 
Categories: FLOSS Project Planets

Louis-Philippe Véronneau: Using Jitsi Meet with Puppet for self-hosted video conferencing

Planet Debian - Wed, 2020-04-08 16:45

Here's a blog post I wrote for the puppet.com blog. Many thanks to Ben Ford and all their team!.

With everything that is currently happening around the world, many of us IT folks have had to solve complex problems in a very short amount of time. Pretty quickly at work, I was tasked with finding a way to make virtual meetings easy, private and secure.

Whereas many would have turned to a SaaS offering, we decided to use Jitsi Meet, a modern and fully on-premise FOSS videoconferencing solution. Jitsi works on all platforms by running in a browser and comes with nifty Android and iOS applications.

We've been using our instance quite a bit, and so far everyone from technical to non-technical users have been pretty happy with it.

Jitsi Meet is powered by WebRTC and can be broken into multiple parts across multiple machines if needed. In addition to the webserver running the Jitsi Meet JavaScript code, the base configuration uses the Videobridge to manage users' video feeds, Jicofo as a conference focus to manage media sessions and the Prosody XMPP server to tie it all together.

Here's a network diagram I took from their documentation to show how those applications interact:

Getting started with the Jitsi Puppet module

First of all, you'll need a valid domain name and a server with decent bandwidth. Jitsi has published a performance evaluation of the Videobridge to help you spec your instance appropriately. You will also need to open TCP ports 443, 4443 and UDP port 10000 in your firewall. The puppetlabs/firewall module could come in handy here.

Once that is done, you can use the smash/jitsimeet Puppet module on a Debian 10 (Buster) server to spin up an instance. A basic configuration would look like this:

class { 'jitsimeet': fqdn => 'jitsi.example.com', repo_key => puppet:///files/apt/jitsimeet.gpg, manage_certs => true, jitsi_vhost_ssl_key => '/etc/letsencrypt/live/jitsi.example.com/privkey.pem' jitsi_vhost_ssl_cert => '/etc/letsencrypt/live/jitsi.example.com/cert.pem' auth_vhost_ssl_key => '/etc/letsencrypt/live/auth.jitsi.example.com/privkey.pem' auth_vhost_ssl_cert => '/etc/letsencrypt/live/auth.jitsi.example.com/cert.pem' jvb_secret => 'mysupersecretstring', focus_secret => 'anothersupersecretstring', focus_user_password => 'yetanothersecret', meet_custom_options => { 'enableWelcomePage' => true, 'disableThirdPartyRequests' => true, }; }

The jitsimeet module is still pretty young: it clearly isn't perfect and some external help would be very appreciated. If you have some time, here are a few things that would be nice to work on:

  • Tests using puppet-rspec
  • Support for other OSes (only Debian 10 at the moment)
  • Integration with the Apache and Ngnix modules

If you use this module to manage your Jitsi Meet instance, please send patches and bug reports our way!

Learn more
Categories: FLOSS Project Planets

Lullabot: Naming Content Types Using a Ubiquitous Language

Planet Drupal - Wed, 2020-04-08 15:56

The team of editors who populate content on Newspaper, Inc. has asked the development team to add another image field on the articles they write, one that caters to Pinterest sharing. The project manager writes up the ticket, and the development team gets to work. They add the field to the Article content type. They demo it later that week, and everyone seems happy.

Categories: FLOSS Project Planets

David Bremner: Tangling multiple files

Planet Debian - Wed, 2020-04-08 12:35

I have lately been using org-mode literate programming to generate example code and beamer slides from the same source. I hit a wall trying to re-use functions in multiple files, so I came up with the following hack. Thanks 'ngz' on #emacs and Charles Berry on the org-mode list for suggestions and discussion.

(defun db-extract-tangle-includes () (goto-char (point-min)) (let ((case-fold-search t) (retval nil)) (while (re-search-forward "^#[+]TANGLE_INCLUDE:" nil t) (let ((element (org-element-at-point))) (when (eq (org-element-type element) &aposkeyword) (push (org-element-property :value element) retval)))) retval)) (defun db-ob-tangle-hook () (let ((includes (db-extract-tangle-includes))) (mapc #&aposorg-babel-lob-ingest includes))) (add-hook &aposorg-babel-pre-tangle-hook #&aposdb-ob-tangle-hook t)

Use involves something like the following in your org-file.

#+SETUPFILE: presentation-settings.org #+SETUPFILE: tangle-settings.org #+TANGLE_INCLUDE: lecture21.org #+TITLE: GC V: Mark & Sweep with free list

For batch export with make, I do something like

%.tangle-stamp: %.org emacs --batch --quick -l org -l ${HOME}/.emacs.d/org-settings.el --eval "(org-babel-tangle-file \"$<\")" touch $@
Categories: FLOSS Project Planets

GNU Guix: A “Hello World” virtual machine running the Hurd

GNU Planet! - Wed, 2020-04-08 11:46

Hello GNU World!

There’s been a bit of speculation as to whether our April 1st post was a joke. Part of it was a joke: we’re not deprecating Linux-libre, fear not! But when we published it, it was already April 2nd in Eastern parts of the world and thus, not surprisingly, the remainder of the post was less of a joke.

Getting to a bootable system

For all you who tried our April 1st image and ran guix we sure hope you had a good laugh. We set out to cross-build that virtual machine (VM) image using Guix and while we made some good progress on Wednesday, in the end we decided to cheat to make the release deadline.

What we got stuck on for a while was to get past the ext2fs translator (the user-land process that implements the ext2 file system) seemingly freezing on boot, saying:

start ext2fs:

and then nothing... Running ext2fs cross-built with Guix on Debian GNU/Hurd would hang similarly. The kernel debugger would show an intriguing backtrace in ext2fs suggesting that ext2fs was not handling page fault messages. Long story short: we eventually realized that the server interfaces were compiled with a 64-bit MiG whereas we were targeting a 32-bit platform. From there on, we embarked on a delightful hacking journey ensuring the Hurd boot process would correctly run in our VM up to a proper login prompt.

Today we have a humble gift for you: On the wip-hurd-vm branch we have an initial hurd.scm system description that can be used to cross build a VM running the Hurd.

Running:

./pre-inst-env guix build -f gnu/system/hurd.scm

cross-compiles all the relevant packages for GNU/Hurd—specifically the i586-pc-gnu triplet—and produces a VM image:

/gnu/store/yqnabv1zmlkviwzikc23w9qvfnyfwvj7-qemu-image

You can build it and start it from your GNU/Linux machine with this command:

qemu-system-i386 -enable-kvm -m 512 -snapshot -hda \ $(./pre-inst-env guix build -f gnu/system/hurd.scm)

and voilà:

Woohoo! (Actually we already have more stuff not shown here, such as guix itself running… for a future post! :-))

Why bother?

Why bother with the Hurd anyway? Isn’t it a pipe dream or “vaporware”, depending on one’s perspective? There’s some unquestionable truth in that: we know that Hurd development started in the early 90’s, months before Linux development started, and yet it still lacks so much in terms of hardware support, even though significant progress was made in recent years in particular with the use of Rump kernels.

The more we witness how new features are retrofitted in the kernel Linux, the more we think the Hurd’s design is better suited to today’s needs. Linux namespaces, the foundation of “containers”, are such an example of an afterthought; unprivileged user namespaces, which allow unprivileged users to benefit from lightweight “container” virtualization, are still often disabled by distros due to a lack of confidence. This is in sharp contrast with the Hurd’s inherent unrestricted support for fine-grain virtualization: a PID namespace is just another proc server, and file system name space is just another root file system server, and so on. Container-like lightweight virtualization is native on the Hurd.

Last but not least, with an eye on the security and transparency of free software systems, a microkernel-based systems seems to naturally lend itself well to bootstrapping from a reduced trusted base. This is one of the topics we discussed on the last Reproducible Builds Summit.

The question is not so much whether 2020 or 2021 will be the year of the Hurd. It’s more about the kind of systems we want to build. A lot of work remains to be done, but we think, in 2020 more than ever, that this is a promising approach for the betterment of the security of our systems and the freedom of users.

We also have to admit that this is an amazing system to hack on, even more so when combined with Guix, so… happy hacking! :-)

About GNU Guix

GNU Guix is a transactional package manager and an advanced distribution of the GNU system that respects user freedom. Guix can be used on top of any system running the Hurd or the Linux kernel, or it can be used as a standalone operating system distribution for i686, x86_64, ARMv7, and AArch64 machines.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. When used as a standalone GNU/Linux distribution, Guix offers a declarative, stateless approach to operating system configuration management. Guix is highly customizable and hackable through Guile programming interfaces and extensions to the Scheme language.

About the GNU Hurd

The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More info.

The mission of the GNU Hurd project is to create a general-purpose kernel suitable for the GNU operating system, which is viable for everyday use, and gives users and programs as much control over their computing environment as possible.

Categories: FLOSS Project Planets

Learn PyQt: Packaging PyQt5 &amp; PySide2 applications for Windows, with PyInstaller

Planet Python - Wed, 2020-04-08 10:46

There is not much fun in creating your own desktop applications if you can't share them with other people — whether than means publishing it commercially, sharing it online or just giving it to someone you know. Sharing your apps allows other people to benefit from your hard work!

The good news is there are tools available to help you do just that with your Python applications which work well with apps built using Qt5. In this tutorial we'll look at the most popular tool for packaging Python applications: PyInstaller.

This tutorial is broken down into a series of steps, using PyInstaller to build first simple, and then increasingly complex PyQt5 applications into distributable EXE files on Windows. You can choose to follow it through completely, or skip ahead to the examples that are most relevant to your own project.

We finish off by using InstallForge to create a distributable Windows installer for Piecasso — a completely functional Paint clone made with Python 3 & Qt5

You always need to compile your app on your target system. So, if you want to create a Mac .app you need to do this on a Mac, for an EXE you need to use Windows.

Piecasso Screenshot, showing my artistic skills

If you're impatient, you can download the Piecasso Installer for Windows right away! Piecasso is one of our 15 Minute Apps, a collection of minute (small) apps built with Python & Qt5 and including all source code.

Requirements

PyInstaller works out of the box with both PyQt PyQt5 and Qt for Python PySide2 and, as of writing, PyInstaller is compatible up to 3.7. Whatever project you're working on, you should be able to package your apps.

You can install PyInstaller using pip.

pip3 install PyInstaller Install in virtual environment (optional)

You can also opt to install PyQt5 and PyInstaller in a virtual environment (or your applications virtual environment) to keep your environment clean.

python3 -m venv packenv

Once created, activate the virtual environment by running from the command line —

call packenv\scripts\activate.bat

Finally, install the required libraries. For PyQt5 you would use —

pip3 install PyQt5 PyInstaller

Or for Qt for Python (PySide2) —

pip3 install PySide2 PyInstaller Getting Started

It's a good idea to start packaging your application from the very beginning so you can confirm that packaging is still working as you develop it. This is particularly important if you add additional dependencies. If you only think about packaging at the end, it can be difficult to debug exactly where the problems are.

T> If you've already got an application you've created and want to know how to package that you may want to skip ahead to the advanced examples.

For this example we're going to start with a simple skeleton app, which doesn't do anything interesting. Once we've got the basic packaging process working, we'll extend the application to include icons and data files. We'll confirm the build as we go along.

To start with, create a new folder for your application and then add the following skeleton app in a file named app.py. You can also download the source code and associated files

python from PyQt5 import QtWidgets import sys class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) w = MainWindow() app.exec() python from PySide2 import QtWidgets import sys class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) w = MainWindow() app.exec_()

This is a basic bare-bones application which creates a custom QMainWindow and adds a simple widget QLabel to it. You can run this app as follows.

python app.py

This should produce the following window (on Windows 10).

Simple skeleton app PyQt5/PySide2 Build #1, a basic app

Now we have our simple application skeleton in place, we can run our first build test to make sure everything is working.

Open your terminal (command prompt) and navigate to the folder containing your project. You can now run the following command to run the PyInstaller build.

pyinstaller app.py

You'll see a number of messages output, giving debug information about what PyInstaller is doing. These are useful for debugging issues in your build, but can otherwise be ignored. The output that I get for running the command on Windows 10 is shown below.

U:\home\martin\helloworld>pyinstaller app.py INFO: PyInstaller: 3.6 INFO: Python: 3.7.6 INFO: Platform: Windows-10-10.0.18362-SP0 INFO: wrote U:\home\martin\helloworld\app.spec INFO: UPX is not available. INFO: Extending PYTHONPATH with paths ['U:\\home\\martin\\helloworld', 'U:\\home\\martin\\helloworld'] INFO: checking Analysis INFO: Building Analysis because Analysis-00.toc is non existent INFO: Initializing module dependency graph... INFO: Caching module graph hooks... INFO: Analyzing base_library.zip ... INFO: Caching module dependency graph... INFO: running Analysis Analysis-00.toc INFO: Adding Microsoft.Windows.Common-Controls to dependent assemblies of final executable required by c:\users\gebruiker\appdata\local\programs\python\python37\python.exe INFO: Analyzing U:\home\martin\helloworld\app.py INFO: Processing module hooks... INFO: Loading module hook "hook-encodings.py"... INFO: Loading module hook "hook-pydoc.py"... INFO: Loading module hook "hook-PyQt5.py"... WARNING: Hidden import "sip" not found! INFO: Loading module hook "hook-PyQt5.QtWidgets.py"... INFO: Loading module hook "hook-xml.py"... INFO: Loading module hook "hook-PyQt5.QtCore.py"... INFO: Loading module hook "hook-PyQt5.QtGui.py"... INFO: Looking for ctypes DLLs INFO: Analyzing run-time hooks ... INFO: Including run-time hook 'pyi_rth_pyqt5.py' INFO: Looking for dynamic libraries INFO: Looking for eggs INFO: Using Python library c:\users\gebruiker\appdata\local\programs\python\python37\python37.dll INFO: Found binding redirects: [] INFO: Warnings written to U:\home\martin\helloworld\build\app\warn-app.txt INFO: Graph cross-reference written to U:\home\martin\helloworld\build\app\xref-app.html INFO: checking PYZ INFO: Building PYZ because PYZ-00.toc is non existent INFO: Building PYZ (ZlibArchive) U:\home\martin\helloworld\build\app\PYZ-00.pyz INFO: Building PYZ (ZlibArchive) U:\home\martin\helloworld\build\app\PYZ-00.pyz completed successfully. INFO: checking PKG INFO: Building PKG because PKG-00.toc is non existent INFO: Building PKG (CArchive) PKG-00.pkg INFO: Building PKG (CArchive) PKG-00.pkg completed successfully. INFO: Bootloader c:\users\gebruiker\appdata\local\programs\python\python37\lib\site-packages\PyInstaller\bootloader\Windows-64bit\run.exe INFO: checking EXE INFO: Building EXE because EXE-00.toc is non existent INFO: Building EXE from EXE-00.toc INFO: Appending archive to EXE U:\home\martin\helloworld\build\app\app.exe INFO: Building EXE from EXE-00.toc completed successfully. INFO: checking COLLECT INFO: Building COLLECT because COLLECT-00.toc is non existent INFO: Building COLLECT COLLECT-00.toc INFO: Building COLLECT COLLECT-00.toc completed successfully.

If you look in your folder you'll notice you now have two new folders dist and build.

build & dist folders created by PyInstaller

Below is a truncated listing of the folder content, showing the build and dist folders.

. ├── app.py ├── app.spec ├── build │   └── app │   ├── Analysis-00.toc │   ├── COLLECT-00.toc │   ├── EXE-00.toc │   ├── PKG-00.pkg │   ├── PKG-00.toc │   ├── PYZ-00.pyz │   ├── PYZ-00.toc │   ├── app.exe │   ├── app.exe.manifest │   ├── base_library.zip │   ├── warn-app.txt │   └── xref-app.html └── dist └── app ├── MSVCP140.dll ├── PyQt5 ├── app.exe ├── app.exe.manifest ├── Qt5Core.dll ...

The build folder is used by PyInstaller to collect and prepare the files for bundling, it contains the results of analysis and some additional logs. For the most part, you can ignore the contents of this folder, unless you're trying to debug issues.

The dist (for "distribution") folder contains the files to be distributed. This includes your application, bundled as an executable file, together with any associated libraries (for example PyQt5) and binary .dll files.

Everything necessary to run your application will be in this folder, meaning you can take this folder and "distribute" it to someone else to run your app.

You can try running your app yourself now, by running the executable file, named app.exe from the dist folder. After a short delay you'll see the familiar window of your application pop up as shown below.

Simple app, running after being packaged

You may also notice a console/terminal window pop up as your application runs. We'll cover how to stop that happening shortly.

In the same folder as your Python file, alongside the build and dist folders PyInstaller will have also created a .spec file. In the next section we'll take a look at this file, what it is and what it does.

The Spec file

The .spec file contains the build configuration and instructions that PyInstaller uses to package up your application. Every PyInstaller project has a .spec file, which is generated based on the command line options you pass when running pyinstaller.

When we ran pyinstaller with our script, we didn't pass in anything other than the name of our Python application file. This means our spec file currently contains only the default configuration. If you open it, you'll see something similar to what we have below.

# -*- mode: python ; coding: utf-8 -*- block_cipher = None a = Analysis(['app.py'], pathex=['U:\\home\\martin\\helloworld'], binaries=[], datas=[], hiddenimports=[], hookspath=[], runtime_hooks=[], excludes=[], win_no_prefer_redirects=False, win_private_assemblies=False, cipher=block_cipher, noarchive=False) pyz = PYZ(a.pure, a.zipped_data, cipher=block_cipher) exe = EXE(pyz, a.scripts, [], exclude_binaries=True, name='app', debug=False, bootloader_ignore_signals=False, strip=False, upx=True, console=True ) coll = COLLECT(exe, a.binaries, a.zipfiles, a.datas, strip=False, upx=True, upx_exclude=[], name='app')

The first thing to notice is that this is a Python file, meaning you can edit it and use Python code to calculate values for the settings. This is mostly useful for complex builds, for example when you are targeting different platforms and want to conditionally define additional libraries or dependencies to bundle.

Once a .spec file has been generated, you can pass this to pyinstaller instead of your script to repeat the previous build process. Run this now to rebuild your executable.

pyinstaller app.spec

The resulting build will be identical to the build used to generate the .spec file (assuming you have made no changes). For many PyInstaller configuration changes you have the option of passing command-line arguments, or modifying your existing .spec file. Which you choose is up to you.

Tweaking the build

So far we've created a simple first build of a very basic application. Now we'll look at a few of the most useful options that PyInstaller provides to tweak our build. Then we'll go on to look at building more complex applications.

Naming your app

One of the simplest changes you can make is to provide a proper "name" for your application. By default the app takes the name of your source file (minus the extension), for example main or app. This isn't usually what you want.

You can provide a nicer name for PyInstaller to use for the executable (and dist folder) either by editing the .spec file to add a name= under the app block.

exe = EXE(pyz, a.scripts, [], exclude_binaries=True, name='app', name='Hello World', debug=False, bootloader_ignore_signals=False, strip=False, upx=True, console=False # False = do not show console. )

Alternatively, you can re-run the pyinstaller command and pass the -n or --name configuration flag along with your app.py script.

pyinstaller -n "Hello World" app.py # or pyinstaller --name "Hello World" app.py

The resulting EXE file will is given the name Hello World.exe and placed in the folder dist\Hello World\.

Application with custom name "Hello World"

The name of the .spec file is taken from the name passed in on the command line, so this will also create a new spec file for you, called Hello World.spec in your root folder.

Hiding the console window

When you run your packaged application you will notice that a console window runs in the background. If you try and close this console window your application will also close. You almost never want this window in a GUI application and PyInstaller provides a simple way to turn this off.

Application running with terminal in background

You can fix this in one of two ways. Firstly, you can edit the previously created .spec file setting console=False under the EXE block as shown below.

exe = EXE(pyz, a.scripts, [], exclude_binaries=True, name='app', debug=False, bootloader_ignore_signals=False, strip=False, upx=True, console=False # False = do not show console. )

Alternatively, you can re-run the pyinstaller command and pass the -w, --noconsole or --windowed configuration flag along with your app.py script.

pyinstaller -w app.py # or pyinstaller --windowed app.py # or pyinstaller --noconsole app.py

There is no difference between any of the options.

Re-running pyinstaller will re-generate the .spec file. If you've made any other changes to this file these will be lost.

One File Build

On Windows PyInstaller has the ability to create a one-file build, that is, a single EXE file which contains all your code, libraries and data files in one. This can be a convenient way to share simple applications, as you don't need to provide an installer or zip up a folder of files.

To specify a one-file build provide the --onefile flag at the command line.

pyinstaller --onefile app.py Result of a one-file build

Note that while the one-file build is easier to distribute, it is slower to execute than a normally built application. This is because every time the application is run it must create a temporary folder to unpack the contents of the executable. Whether this trade-off is worth the convenience for your app is up to you!

Using the --onefile option makes quite a few changes to the .spec file. You can make these changes manually, but it's much simpler to use the command line switch when first creating your .spec

Since debugging a one file app is much harder, you should make sure everything is working with a normal build before you create a one-file package.

Setting an application Icon

By default PyInstaller EXE files come with the following icon in place.

Default PyInstaller application icon, on app.exe

You will probably want to customize this to make your application more recognisable. This can be done easily using the --icon=<filename> command-line switch to PyInstaller. On Windows the icon should be provided as an .ico file.

pyinstaller --windowed --icon=hand_icon.ico app.py

The portable version of IcoFx is a good free tool to create icons on Windows.

Or, by adding the icon= parameter to your .spec file.

exe = EXE(pyz, a.scripts, [], exclude_binaries=True, name='blarh', debug=False, bootloader_ignore_signals=False, strip=False, upx=True, console=False, icon='hand_icon.ico')

If you now re-run the build (by using the command line arguments, or running with your modified .spec file) you'll see the specified icon file is now set on your application's EXE file.

Custom application icon (a hand) on app.exe

However, if you run your application, you're going to be disappointed.

The custom EXE icon is not applied to the window

The specified icon is not showing up on the window, and it will also not appear on your taskbar.

Why not? Because the icon used for the window isn't determined by the icons in the executable file, but by the application itself. To show an icon on our window we need to modify our simple application a little bit, to add a call to .setWindowIcon().

python from PyQt5 import QtWidgets, QtGui import sys class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon('hand_icon.ico')) w = MainWindow() app.exec() python from PySide2 import QtWidgets, QtGui import sys class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon('hand_icon.ico')) w = MainWindow() app.exec_()

Here we've added the .setWindowIcon call to the app instance. This defines a default icon to be used for all windows of our application. You can override this on a per-window basis if you like, by calling .setWindowIcon on the window itself.

If you run the above application you should now see the icon appears on the window.

Window showing the custom hand icon

But, unfortunately, it may still not show on the taskbar.

If it does for you, great! But it may not work when you distribute your application, so it's probably a good idea to follow the next steps anyway.

Custom icon is not shown on the toolbar

The final tweak we need to make to get the icon showing on the taskbar is to add some cryptic incantations to the top of our Python file.

When you run your application, Windows looks at the executable and tries to guess what "application group" it belongs to. By default, any Python scripts (including your application) are grouped under the same "Python" group, and so will show the Python icon. To stop this happening, we need to provide Windows with a different application identifier.

The code below does this, by calling QtWin.setCurrentProcessExplicitAppUserModelID() with a custom application id.

python from PyQt5 import QtWidgets, QtGui try: # Include in try/except block if you're also targeting Mac/Linux from PyQt5.QtWinExtras import QtWin myappid = 'mycompany.myproduct.subproduct.version' QtWin.setCurrentProcessExplicitAppUserModelID(myappid) except ImportError: pass # ..or.. # import ctypes # myappid = 'mycompany.myproduct.subproduct.version' # ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(myappid) import sys class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon('hand_icon.ico')) w = MainWindow() app.exec() python from PySide2 import QtWidgets, QtGui try: # Include in try/except block if you're also targeting Mac/Linux from PySide2.QtWinExtras import QtWin myappid = 'mycompany.myproduct.subproduct.version' QtWin.setCurrentProcessExplicitAppUserModelID(myappid) except ImportError: pass # ..or.. # import ctypes # myappid = 'mycompany.myproduct.subproduct.version' # ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID(myappid) import sys class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon('hand_icon.ico')) w = MainWindow() app.exec_()

You can, alternatively, use the Python call ctypes.windll.shell32.SetCurrentProcessExplicitAppUserModelID. There is no difference.

The listing above shows a generic mycompany.myproduct.subproduct.version string, but you should change this to reflect your actual application. It doesn't really matter what you put for this purpose, but the convention is to use reverse-domain notation, com.mycompany for the company identifier.

With this added to your script, running it should now show the icon on your window and taskbar. The final step is to ensure that this icon is correctly packaged with your application and continues to be shown when run from the dist folder.

Try it, it wont.

The issue is that our application now has a dependency on a external data file (the icon file) that's not part of our source. For our application to work, we now need to distribute this data file along with it. PyInstaller can do this for us, but we need to tell it what we want to include, and where to put it in the output.

In the next section we'll look at the options available to you for managing data files associated with your app.

Data files and Resources

So far we successfully built a simple app which had no external dependencies. However, once we needed to load an external file (in this case an icon) we hit upon a problem. The file wasn't copied into our dist folder and so could not be loaded.

In this section we'll look at the options we have to be able to bundle external resources, such as icons or Qt Designer .ui files, with our applications.

Bundling data files with PyInstaller

The simplest way to get these data files into the dist folder is to just tell PyInstaller to copy them over. PyInstaller accepts a list of individual file paths to copy over, together with a folder path relative to the dist folder where it should to copy them to.

As with other options, this can be specified by command line arguments, --add-data

pyinstaller --windowed --icon=hand_icon.ico --add-data="hand_icon.ico;." app.py

You can provide `--add-data` multiple times. Note that the path separator is platform-specific, on Windows use `;` while on Linux or Mac use `:`

Or via the datas list in the Analysis section of the spec file, shown below.

a = Analysis(['app.py'], pathex=['U:\\home\\martin\\helloworld'], binaries=[], datas=[('hand_icon.ico', '.')], hiddenimports=[], hookspath=[], runtime_hooks=[], excludes=[], win_no_prefer_redirects=False, win_private_assemblies=False, cipher=block_cipher, noarchive=False)

And then execute the .spec file with

pyinstaller app.spec

In both cases we are telling PyInstaller to copy the specified file hand_icon.ico to the location . which means the output folder dist. We could specify other locations here if we wanted. On the command line the source and destination are separated by the path separator ;, whereas in the .spec file, the values are provided as a 2-tuple of strings.

If you run the build, you should see your .ico file now in the output folder dist ready to be distributed with your application.

The icon file copied to the dist folder

If you run your app from dist you should now see the icon on the window, and on the taskbar as expected.

The hand icon showing on the toolbar

The file must be loaded in Qt using a relative path, and be in the same relative location to the EXE as it was to the .py file for this to work.

If your icon looks blurry it means you don't have large-enough icon variations in your .ico file. An .ico file can contain multiple different sized icons in the same file. Ideally you want to have 16x16, 32x32, 48x48 and 256x256 pixel sizes included, although fewer will still work.

The main advantage of using PyInstaller to bundle your files in this way is you can use Python in your .spec file to search and add the files to bundle. So for example, you could get a list of all files in a folder named icons and add them to the datas= parameter. Then, as you add more icons to that folder they would be bundled automatically.

The trade-off is that you can sometimes hit problems, particularly when bundling your applications cross-platform. In the next section we'll look at an alternative, often more reliable, method using the Qt Resources system.

Qt Resources

Bundling data files with PyInstaller usually works quite well. However, there can be issues with using relative paths, particularly when bundling cross-platform applications, as different systems have different standards for dealing with data files. If you hit these problems they can unfortunately be quite difficult to debug.

Thankfully, Qt comes to the rescue with it's resource system. Since we're using Qt for our GUI we can make use of Qt Resources to bundle, identify and load resources in our application. The resulting bundled data is included in your application as Python code, so PyInstaller will automatically pick it up and we can be sure it will end up in the right place.

In this section we'll look at how to bundle files with our application using the Qt Resources system.

The QRC file

The core of the Qt Resources system is the resource file or QRC. The .qrc file is a simple XML file, which can be opened and edited with any text editor.

You can also create QRC files and add and remove resources using Qt Designer, which we'll cover later.

Simple QRC example

A very simple resource file is shown below, containing a single resource (our application icon).

<!DOCTYPE RCC> <RCC version="1.0"> <qresource prefix="icons"> <file alias="hand_icon.ico">hand_icon.ico</file> </qresource> </RCC>

The name between the <file> </file> tags is the path to the file, relative to the resource file. The alias is the name which this resource will be known by from within your application. You can use this rename icons to something more logical or simpler in your app, while keeping the original name externally.

For example, if we wanted to use the name application_icon.ico internally, we could change this line to.

<file alias="application_icon.ico">hand_icon.ico</file>

This only changes the name used *inside* your application, the filename remains unchanged.

Outside this tag is the qresource tag which specifies a prefix. This is a namespace which can be used to group resources together. This is effectively a virtual folder, under which nested resources can all be found.

Using a QRC file

To use a .qrc file in your application you first need to compile it to Python. PyQt5 & PySide2 come with a command line tool to do this, which takes a .qrc file as input and outputs a Python file containing the compiled data. This can then be imported into your app as for any other Python file or module.

To compile our resources.qrc file to a Python file named resources.py we can use

bash pyrcc5 resources.qrc -o resources.py bash pyside2-rcc resources.qrc -o resources.py

To use the resource file in our application we need to make a few small changes. Firstly, we need to import resources at the top of our app, to load the resources into the Qt resource system, and then secondly we need to update the path to the icon file to use the resource path format as follows:

app.setWindowIcon(QtGui.QIcon(':/icons/hand_icon.ico'))

The prefix :/ indicates that this is a resource path. The first name "icons" is the prefix namespace and the filename is taken from the file alias, both as defined in our resources.qrc file.

The updated application is shown below.

python from PyQt5 import QtWidgets, QtGui try: # Include in try/except block if you're also targeting Mac/Linux from PyQt5.QtWinExtras import QtWin myappid = 'com.learnpyqt.examples.helloworld' QtWin.setCurrentProcessExplicitAppUserModelID(myappid) except ImportError: pass import sys import resources # Import the compiled resource file. class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon(':/icons/hand_icon.ico')) w = MainWindow() app.exec() python from PySide2 import QtWidgets, QtGui try: # Include in try/except block if you're also targeting Mac/Linux from PySide2.QtWinExtras import QtWin myappid = 'com.learnpyqt.examples.helloworld' QtWin.setCurrentProcessExplicitAppUserModelID(myappid) except ImportError: pass import sys import resources # Import the compiled resource file. class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self.setWindowTitle("Hello World") l = QtWidgets.QLabel("My simple app.") l.setMargin(10) self.setCentralWidget(l) self.show() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon(':/icons/hand_icon.ico')) w = MainWindow() app.exec_()

You can run the build as follows,

pyinstaller --windowed --icon=hand_icon.ico app.py

or re-run it using your existing .spec file.

pyinstaller app.spec

If you run the resulting application in dist you should see the icon is working as intended.

The hand icon showing on the toolbar

The advantage of this method is that your data files are guaranteed to be bundled as they are treated as code — PyInstaller finds them through the imports in your source. You also don't need to worry about platform-specific locations for data files. You only need to take care to rebuild the resources.py file any time you add or remove data files from your project.

Of course, this approach isn't appropriate for any files you want to be readable or editable by end-users. However, there is nothing stopping you from combining this approach with the previous one as needed.

Build #2, bundling Qt Designer UIs and Icons

We've now managed to build a simple app with a single external icon file as a dependency. Now for something a bit more realistic!

In complex Qt applications it's common to use Qt Designer to define the the UI, including icons on buttons and menus. How can we distribute UI files with our applications and ensure the linked icons continue to work as expected?

Below is the UI for a demo application we'll use to demonstrate this. The app is a simple counter, which allows you to increase, decrease or reset the counter by clicking the respective buttons. You can also download the source code and associated files.

The counter UI created in Qt Designer

The UI consists of a QMainWindow with a vertical layout containing a single QLabel and 3 QPushButton widgets. The buttons have Increment, Decrement and Reset labels, along with icons from the Fugue set by p.yusukekamiyamane. The application icon is a free icon from Freepik.

The UI was created in Qt Designer as described in this tutorial.

Resources

The icons in this project were added to the buttons from within Qt Designer. When doing this you have two options —

  1. add the icons as files, and ensure that the relative path locations of icons are maintained after installation (not always possible, or fun)
  2. add the icons using the Qt Resource system

Here we're using approach (2) because it's less prone to errors.

The method for Qt Resources in your UI differs depending on whether you're using Qt Creator or Qt Designer standalone. The steps are described below.

Adding Resources in Qt Designer (Preferred)

If you're using the standalone Qt Designer, the resource browser is available as a dockable widget, visible in the bottom right by default. If the Resource Browser is hidden you can show it through the "View" menu on the toolbar.

To add, edit and remove resource files click on the pencil icon in the Resource browser panel. This will open the resource editor.

Qt Designer resource browser

In the resource editor view you can open an existing resource file by clicking on the document folder icon (middle icon) on the bottom left.

Qt Designer resource editor

On the left hand panel you can also create and delete resource files from your UI. While on the right you can create new prefixes, add files to the prefix and delete items. Changes to the resource file are saved automatically.

Adding Resources in Qt Creator

In order to be able to add icons using the Qt Resource system from within Qt Creator you need to have an active Qt Project, and add both your UI and resource files to it.

If you don't have a Qt Creator project set up you can create one in your existing source folder. Qt Creator will prompt before overwriting any of your files. Click on "+ New", choose "Qt for Python - Empty" for project type. Select the folder above your source folder for "Create in", and provide the name of your source folder as the project name. You can delete any files created, except the .pyproject which holds the project settings.

This message will be shown when creating a new Qt Creator project in an existing folder

To add resources to your existing project, select the "Edit" view on the left hand panel. You will see a file tree browser in the left hand panel. Right-click on the folder and choose "Add existing files…" and add your existing .qrc file to the project.

Qt Creator "Edit" view, showing a list of files in the project

The UI doesn't update when you add/remove things here, this seems to be a bug in Qt Creator. If you close and re-open Qt Creator the files will be there.

Once you have added the QRC file to the file listing you should be able to expand the file as if it were a folder, and browser the resources within. You can also add and remove resources using this interface.

Using resources in Qt Creator and Qt Designer

Once the Resource file is loaded you will be able to access it from the designer properties. The screenshot below shows the Designer with our counter app open, and the increment button selected. The icon for the button can be chosen by clicking the small black down arrow and selecting "Choose Resource…"

Setting the icon for a button in Qt Designer (or Qt Creator)

The Resource chooser window that appears allows you to pick icons from the resource file(s) in the project to use in your UI.

Selecting a resource in the Qt Designer resource dialog

Selecting the icons from the resource file in this way ensures that they will always work, as long as you compile and bundle the compiled resource file with your app.

Loading UIs from the Resource file (PyQt5 only)

The last change required is in the loading of the UI file. In PyQt5 the .loadUi handler does not understand resource paths, and so we cannot load our mainwindow.ui file directly with it. Instead we need to load the file manually, and pass the result into .loadUi ourselves. The code to do that is shown below.

# Load the UI fileh = QtCore.QFile(':/ui/mainwindow.ui') fileh.open(QtCore.QFile.ReadOnly) uic.loadUi(fileh, self) fileh.close()

You may want to wrap this in a function if you're using it a lot. The PySide2 handler understands resource paths natively, so we don't need this workaround.

Alternatively, you can also compile your UI files to Python. This is covered later.

The finished app

Below is our updated app.py which loads the mainwindow.ui file and defines 3 custom slots to increment, decrement and reset the number. These are connected to signals of the widgets defined in the UI (btn_inc, btn_dec and btn_reset for the 3 buttons respectively) along with a method to update the displayed number (label for the QLabel).

python from PyQt5 import QtWidgets, QtCore, uic import sys import resources # Import our compiled resources file. try: # Include in try/except block if you're also targeting Mac/Linux from PyQt5.QtWinExtras import QtWin myappid = 'com.learnpyqt.examples.counter' QtWin.setCurrentProcessExplicitAppUserModelID(myappid) except ImportError: pass class MainWindow(QtWidgets.QMainWindow): def __init__(self, *args, **kwargs): super(MainWindow, self).__init__(*args, **kwargs) # Load the UI fileh = QtCore.QFile(':/ui/mainwindow.ui') fileh.open(QtCore.QFile.ReadOnly) uic.loadUi(fileh, self) fileh.close() # Set value of counter self.counter = 0 self.update_counter() # Bind self.btn_inc.clicked.connect(self.inc) self.btn_dec.clicked.connect(self.dec) self.btn_reset.clicked.connect(self.reset) def update_counter(self): self.label.setText(str(self.counter)) def inc(self): self.counter += 1 self.update_counter() def dec(self): self.counter -= 1 self.update_counter() def reset(self): self.counter = 0 self.update_counter() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon(':/icons/counter.ico')) main = MainWindow() main.show() sys.exit(app.exec_()) python from PySide2 import QtWidgets, QtGui, QtCore from PySide2.QtUiTools import QUiLoader import sys import resources loader = QUiLoader() try: # Include in try/except block if you're also targeting Mac/Linux from PyQt5.QtWinExtras import QtWin myappid = 'com.learnpyqt.examples.counter' QtWin.setCurrentProcessExplicitAppUserModelID(myappid) except ImportError: pass class WindowWrapper(QtCore.QObject): def __init__(self, *args, **kwargs): super(WindowWrapper, self).__init__() self.ui = loader.load(':/ui/mainwindow.ui', None) self.ui.show() # Set value of counter self.counter = 0 self.update_counter() # Bind self.ui.btn_inc.clicked.connect(self.inc) self.ui.btn_dec.clicked.connect(self.dec) self.ui.btn_reset.clicked.connect(self.reset) def update_counter(self): self.ui.label.setText(str(self.counter)) def inc(self): self.counter += 1 self.update_counter() def dec(self): self.counter -= 1 self.update_counter() def reset(self): self.counter = 0 self.update_counter() if __name__ == '__main__': app = QtWidgets.QApplication(sys.argv) app.setWindowIcon(QtGui.QIcon(':/icons/counter.ico')) main = WindowWrapper() sys.exit(app.exec_())

If you have made any changes to the resources.qrc file, or haven't compiled it yet do so now using pyrcc5 resources.qrc -o resources.py for PyQt5 or pyside2-rcc resources.qrc -o resources.py for PySide2.

If you run this application you should see the following window.

Counter app, with all icons showing

We'll build our app as before using the command line to perform an initial build and generate a .spec file for us. We can use that .spec file in future to repeat the build.

pyinstaller --windowed --icon=resources/counter.ico app.py

PyInstaller will analyse our app.py file, bundling all the necessary dependencies, including our compiled resources.py into the dist folder.

Once the build process is complete, open the dist folder and run the application. You should find it works, with all icons — from the application itself, through to the icons embedded in our UI file — working as expected.

Counter app, with all icons showing

This shows the advantage of using this approach — if your application works before bundling, you can be pretty sure it will continue to work after.

Build #3, bunding a complete app

The applications so far haven't done very much. Next we'll look at something more complete — our custom Piecasso Paint application. The source code is available to download here or in the 15 minute apps repository.

The source code is not covered in depth here, only the steps required to package the application. The source and more info is available in the 15 minute apps repository of example Qt applications. The custom application icons were created using icon art by Freepik.

Prepared for packaging the project has the following structure (truncated for clarity).

. ├── paint.py ├── Piecasso.spec ├── mainwindow.ui ├── MainWindow.py ├── README.md ├── requirements.txt ├── resources.qrc ├── resources_rc.py ├── screenshot-paint1.jpg ├── screenshot-paint2.jpg ├── icons │   ├── blue-folder-open-image.png │   ├── border-weight.png │   ├── cake.png │   ├── disk.png │   ├── document-image.png │   ├── edit-bold.png │   ... └── stamps ├── pie-apple.png ├── pie-cherry.png ├── pie-cherry2.png ├── pie-lemon.png ├── pie-moon.png ├── pie-pork.png ├── pie-pumpkin.png └── pie-walnut.png

The main source of the application is in the paint.py file.

Packaging Resources

The resources for Piecasso are bundled using the Qt Resource system, referenced from the resources.qrc file in the root folder. There are two folders of images, icons which contains the icons for the interface and stamps which contains the pictures of pie for "stamping" the image when the application is running.

The icons were added to the UI in Qt Designer, while the stamps are loaded in the application itself using Qt Resource paths, e.g. :/stamps/<image name>.

The icons folder also contains the application icon, in .ico format.

The UI in Qt Designer

The UI for Piecasso was designed using Qt Designer. Icons on the buttons and actions were set from the Qt resources file already described.

Piecasso UI, created in Qt Designer

The resulting UI file was saved in the root folder of the project as mainwindow.ui and then compiled using the UI compiler to produce an importable .py file, as follows.

pyuic5 mainwindow.ui -o MainWindow.py

T> For more on building UIs with Qt Designer see the introductory tutorial.

This build process also adds imports to MainWindow.py for the compiled version of the resources using in the UI, in our case resources.qrc. This means we do not need to import the resources separately into our app. However, we still need to build them, and use the specific name that is used for the import in MainWindow.py, here resources_rc.

pyrcc5 resources.qrc -o resources_rc.py

pyuic5 follows the pattern <resource name>_rc.py when adding imports for the resource file, so you will need to follow this when compiling resources yourself. You can check your compiled UI file (e.g. MainWindow.py) to double check the name of the import if you have problems.

Building the app

With all the above setup, we can build Piecasso as follows from the source folder.

pyinstaller --windowed --icon=icons/piecasso.ico --name Piecasso paint.py

If you download the source code, you will also be able to run the same build using the provided .spec file.

pyinstaller Piecasso.spec

This packages everything up ready to distribute in the dist/Piecasso folder. We can run the executable to ensure everything is bundled correctly, and see the following window, minus the terrible drawing.

Piecasso Screenshot, with a poorly drawn cat Creating an installer

So far we've used PyInstaller to bundle applications for distribution. The output of this bundling process is a folder, named dist which contains all the files our application needs to run.

While you could share this folder with your users as a ZIP file it's not the best user experience. Desktop applications are normally distributed with installers which handle the process of putting the executable (and any other files) in the correct place, adding Start Menu shortcuts and the like.

Now we've successfully bundled our complex application, we'll next look at how we can take our dist folder and use it to create a functioning Windows installer.

To create our installer we'll be using a tool called InstallForge. InstallForge is free and you can download the installer from this page.

The InstallForge configuration is also in the Piecasso source folder, Piecasso.ifp however bear in mind that the source paths will need to be updated for your system.

Another popular tool is NSIS which is a scriptable installer, meaning you configure it's behaviour by writing custom scripts. If you're going to be building your application frequently and want to automate the process, it's definitely worth a look.

We'll now walk through the basic steps of creating an installer with InstallForge. If you're impatient, you can download the Piecasso Installer for Windows here.

General

When you first run InstallForge you'll be presented with this General tab. Here you can enter the basic information about your application, including the name, program version, company and website.

InstallForge initial view, showing General settings

You can also select the target platforms for the installer, from various versions of Windows that are available. For desktop applications you currently probably only want to target Windows 7, 8 and 10.

Setup

Click on the left sidebar to open the "Files" page under "Setup". Here you can specify the files to be bundled in the installer.

Use "Add Files…" and select all the files in the dist/Piecasso folder produced by PyInstaller. The file browser that pops up allows multiple file selections, so you can add them all in a single go, however you need to add folders separately. Click "Add Folder…" and add any folders under dist/Piecasso such as the PyQt5 folder.

InstallForge Files view, add all files & folders to be packaged

Once you're finished scroll through the list to the bottom and ensure that the folders are listed to be included. You want all files and folders under dist/Piecasso to be present. But the folder dist/Piecasso itself should not be listed.

The default install path can be left as-is. The values between angled brackets, e.g. <company> , are variables and will be filled automatically.

Next, it's nice to allow your users to uninstall your application. Even though it's undoubtedly awesome, they may want to remove it at some time in the future. You can do this under the "Uninstall" tab, simply by ticking the box. This will also make the application appear in "Add or Remove Programs".

InstallForge add Uninstaller for your app Dialogs

The "Dialogs" section can be used to show custom messages, splash screens or license information to the user. The "Finish" tab lets you control what happens once the installer is complete, and it's helpful here to give the user the option to run your program.

To do this you need to tick the box next to "Run program" and add your own application EXE into the box. Since <installpath>\ is already specified, we can just add Piecasso.exe.

InstallForge configure optional run program on finish install System

Under "System" select "Shortcuts" to open the shortcut editor. Here you can specify shortcuts for both the Start Menu and Desktop if you like.

InstallForge configure Shortcuts, for Start Menu and Desktop

Click "Add…" to add new shortcuts for your application. Choose between Start menu and Desktop shortcuts, and fill in the name and target file. This is the path your application EXE will end up at once installed. Since <installpath>\ is already specified, you simply need to add your application's EXE name onto the end, here Piecasso.exe

InstallForge, adding a Shortcut Build

With the basic settings in place, you can now build your installer.

At this point you can save your InstallForge project so you can re-build the installer from the same settings in future.

Click on the "Build" section at the bottom to open the build panel.

InstallForge, ready to build

Click on the large icon button to start the build process. If you haven't already specified a setup file location you will be prompted for one. This is the location where you want the completed installer to be saved.

Don't save it in your dist folder.

The build process will began, collecting and compressing the files into the installer.

InstallForge, build complete

Once complete you will be prompted to run the installer. This is entirely optional, but a handy way to find out if it works.

Running the installer

The installer itself shouldn't have any surprises, working as expected. Depending on the options selected in InstallForge you may have extra panels or options.

InstallForge, running the resulting installer

Step through the installer until it is complete. You can optionally run the application from the last page of the installer, or you can find it in your start menu.

Piecasso in the Start Menu on Windows 10 Wrapping up

In this tutorial we've covered how to build your PyQt5 or PySide2 applications into a distributable EXE using PyInstaller. Following this we walked through the steps of using InstallForge to build an installer for the app. Following these steps you should be able to package up your own applications and make them available to other people.

For a complete view of all PyInstaller bundling options take a look at the PyInstaller usage documentation.

Continue reading: “Packaging PyQt5 apps with fbs”

Categories: FLOSS Project Planets

PyCharm: PyCharm 2020.1 Out Now

Planet Python - Wed, 2020-04-08 10:28

Rebase your branch with ease, debug smarter, and use a font designed for programming. Download the new version now, or upgrade from within your IDE.

New in PyCharm
  • Interactive Rebasing: Commits can get messy, which is why you need to be able to rebase your branch. Now, PyCharm allows you to do so visually and with unprecedented ease.
  • Smarter Debugging: PyCharm’s debugger now makes it easy to see what’s happening in the middle of a complex statement. When stepping in, the debugger will ask you exactly what part of the statement you want to investigate further.
  • JetBrains Mono: PyCharm helps you read your code by navigating your project effectively, and now also by presenting your code in a font designed specifically for it.
  • Django Template Imports: PyCharm Professional Edition makes life easy for web developers by offering code completion (and debugging) in Django templates. We’ve made a couple improvements on this front: auto import for custom tags and better completion.

And many more improvements, like faster indexing of interpreters, read about them all on our What’s New page.

Categories: FLOSS Project Planets

Real Python: How to Provide Test Fixtures for Django Models in Pytest

Planet Python - Wed, 2020-04-08 10:00

If you’re working in Django, pytest fixtures can help you create tests for your models that are uncomplicated to maintain. Writing good tests is a crucial step in sustaining a successful app, and fixtures are a key ingredient in making your test suite efficient and effective. Fixtures are little pieces of data that serve as the baseline for your tests.

As your test scenarios change, it can be a pain to add, modify, and maintain your fixtures. But don’t worry. This tutorial will show you how to use the pytest-django plugin to make writing new test cases and fixtures a breeze.

In this tutorial, you’ll learn:

  • How to create and load test fixtures in Django
  • How to create and load pytest fixtures for Django models
  • How to use factories to create test fixtures for Django models in pytest
  • How to create dependencies between test fixtures using the factory as fixture pattern

The concepts described in this tutorial are suited for any Python project using pytest. For convenience, the examples use the Django ORM, but the results can be reproduced in other types of ORMs and even in projects that don’t use an ORM or a database.

Free Bonus: Click here to get access to a free Django Learning Resources Guide (PDF) that shows you tips and tricks as well as common pitfalls to avoid when building Python + Django web applications.

Fixtures in Django

To get started, you’re going to set up a fresh Django project. Throughout this tutorial, you’ll write some tests using the built-in authentication module.

Setting Up a Python Virtual Environment

When you create a new project, it’s best to also create a virtual environment for it. A virtual environment allows you to isolate the project from other projects on your computer. This way, different projects can use different versions of Python, Django, or any other package without interfering with each other.

Here’s how you can create your virtual environment in a new directory:

$ mkdir django_fixtures $ cd django_fixtures django_fixtures $ python -m venv venv

For step-by-step instructions on how to create a virtual environment, check out Python Virtual Environments: A Primer.

Running this command will create a new directory called venv. This directory will store all the packages you install inside the virtual environment.

Setting Up a Django Project

Now that you have a fresh virtual environment, it’s time to set up a Django project. In your terminal, activate the virtual environment and install Django:

$ source venv/bin/activate $ pip install django

Now that you have Django installed, you can create a new Django project called django_fixtures:

$ django-admin startproject django_fixtures

After running this command, you’ll see that Django created new files and directories. For more about how to start a new Django project, check out Starting a Django Project.

To finish setting up your Django project, apply the migrations for the built-in modules:

$ cd django_fixtures $ python manage.py migrate Operations to perform: Apply all migrations: admin, auth, contenttypes, sessions Running migrations: Applying contenttypes.0001_initial... OK Applying auth.0001_initial... OK Applying admin.0001_initial... OK Applying admin.0002_logentry_remove_auto_add... OK Applying admin.0003_logentry_add_action_flag_choices... OK Applying contenttypes.0002_remove_content_type_name... OK Applying auth.0002_alter_permission_name_max_length... OK Applying auth.0003_alter_user_email_max_length... OK Applying auth.0004_alter_user_username_opts... OK Applying auth.0005_alter_user_last_login_null... OK Applying auth.0006_require_contenttypes_0002... OK Applying auth.0007_alter_validators_add_error_messages... OK Applying auth.0008_alter_user_username_max_length... OK Applying auth.0009_alter_user_last_name_max_length... OK Applying auth.0010_alter_group_name_max_length... OK Applying auth.0011_update_proxy_permissions... OK Applying sessions.0001_initial... OK

The output lists all the migrations Django applied. When starting a new project, Django applies migrations for built-in apps such as auth, sessions, and admin.

Now you’re ready to start writing tests and fixtures!

Creating Django Fixtures

Django provides its own way of creating and loading fixtures for models from files. Django fixture files can be written in either JSON or YAML. In this tutorial, you’ll work with the JSON format.

The easiest way to create a Django fixture is to use an existing object. Start a Django shell:

$ python manage.py shell Python 3.8.0 (default, Oct 23 2019, 18:51:26) [GCC 9.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole)

Inside the Django shell, create a new group called appusers:

>>>>>> from django.contrib.auth.models import Group >>> group = Group.objects.create(name="appusers") >>> group.pk 1

The Group model is part of Django’s authentication system. Groups are very useful for managing permissions in a Django project.

You created a new group called appusers. The primary key of the group you just created is 1. To create a fixture for the group appusers, you are going to use the Django management command dumpdata.

Exit the Django shell with exit() and execute the following command from your terminal:

$ python manage.py dumpdata auth.Group --pk 1 --indent 4 > group.json

In this example, you’re using the dumpdata command to generate fixture files from existing model instances. Let’s break it down:

  • auth.Group: This describes which model to dump. The format is <app_label>.<model_name>.

  • --pk 1: This describes which object to dump. The value is a comma-delimited list of primary keys, such as 1,2,3.

  • --indent 4: This is an optional formatting argument that tells Django how many spaces to add before each indention level in the generated file. Using indentions makes the fixture file more readable.

  • > group.json: This describes where to write the output of the command. In this case, the output will be written to a file called group.json.

Next, inspect the contents of the fixture file group.json:

[ { "model": "auth.group", "pk": 1, "fields": { "name": "appusers", "permissions": [] } } ]

The fixture file contains a list of objects. In this case, you have only one object in the list. Each object includes a header with the name of the model and the primary key, as well as a dictionary with the value for each field in the model. You can see that the fixture contains the name of the group appusers.

You can create and edit fixture files manually, but it’s usually more convenient to create the object beforehand and use Django’s dumpdata command to create the fixture file.

Loading Django Fixtures

Now that you have a fixture file, you want to load it into the database. But before you do that, you should open a Django shell and delete the group that you already created:

>>>>>> from django.contrib.auth.models import Group >>> Group.objects.filter(pk=1).delete() (1, {'auth.Group_permissions': 0, 'auth.User_groups': 0, 'auth.Group': 1})

Now that the group is deleted, load the fixture using the loaddata command:

$ python manage.py loaddata group.json Installed 1 object(s) from 1 fixture(s)

To make sure the new group was loaded, open a Django shell and fetch it:

>>>>>> from django.contrib.auth.models import Group >>> group = Group.objects.get(pk=1) >>> vars(group) {'_state': <django.db.models.base.ModelState at 0x7f3a012d08b0>, 'id': 1, 'name': 'appusers'}

Great! The group was loaded. You just created and loaded your first Django fixture.

Loading Django Fixtures in Tests

So far you’ve created and loaded a fixture file from the command line. Now how can you use it for testing? To see how fixtures are used in Django tests, create a new file called test.py, and add the following test:

from django.test import TestCase from django.contrib.auth.models import Group class MyTest(TestCase): def test_should_create_group(self): group = Group.objects.get(pk=1) self.assertEqual(group.name, "appusers")

The test is fetching the group with the primary key 1 and testing that its name is appusers.

Run the test from your terminal:

$ python manage.py test test Creating test database for alias 'default'... System check identified no issues (0 silenced). E ====================================================================== ERROR: test_should_create_group (test.MyTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/django_fixtures/django_fixtures/test.py", line 9, in test_should_create_group group = Group.objects.get(pk=1) File "/django_fixtures/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 82, in manager_method return getattr(self.get_queryset(), name)(*args, **kwargs) File "/django_fixtures/venv/lib/python3.8/site-packages/django/db/models/query.py", line 415, in get raise self.model.DoesNotExist( django.contrib.auth.models.Group.DoesNotExist: Group matching query does not exist. ---------------------------------------------------------------------- Ran 1 test in 0.001s FAILED (errors=1) Destroying test database for alias 'default'...

The test failed because a group with primary key 1 does not exist.

To load the fixture in the test, you can use a special attribute of the class TestCase called fixtures:

from django.test import TestCase from django.contrib.auth.models import Group class MyTest(TestCase): fixtures = ["group.json"] def test_should_create_group(self): group = Group.objects.get(pk=1) self.assertEqual(group.name, "appusers")

Adding this attribute to a TestCase tells Django to load the fixtures before executing each test. Notice that fixtures accepts an array, so you can provide multiple fixture files to load before each test.

Running the test now produces the following output:

$ python manage.py test test Creating test database for alias 'default'... System check identified no issues (0 silenced). . ---------------------------------------------------------------------- Ran 1 test in 0.005s OK Destroying test database for alias 'default'...

Amazing! The group was loaded and the test passed. You can now use the group appusers in your tests.

Referencing Related Objects in Django Fixtures

So far you’ve used just one file with a single object. Most of the time, however, you will have many models in your app, and you’ll need more than one model in a test.

To see how dependencies between objects look in Django fixtures, create a new user instance and then add it to the appusers group you created before:

>>>>>> from django.contrib.auth.models import User, Group >>> appusers = Group.objects.get(name="appusers") >>> haki = User.objects.create_user("haki") >>> haki.pk 1 >>> haki.groups.add(appusers)

The user haki is now a member of the appusers group. To see what a fixture with a foreign key looks like, generate a fixture for user 1:

$ python manage.py dumpdata auth.User --pk 1 --indent 4 [ { "model": "auth.user", "pk": 1, "fields": { "password": "!M4dygH3ZWfd0214U59OR9nlwsRJ94HUZtvQciG8y", "last_login": null, "is_superuser": false, "username": "haki", "first_name": "", "last_name": "", "email": "", "is_staff": false, "is_active": true, "date_joined": "2019-12-07T09:32:50.998Z", "groups": [ 1 ], "user_permissions": [] } } ]

The structure of the fixture is similar to the one you saw earlier.

A user can be associated with multiple groups, so the field group contains the IDs for all the groups to which the user belongs. In this case, the user belongs the group with primary key 1, which is your appusers group.

Using primary keys to reference objects in fixtures is not always a good idea. The primary key of a group is an arbitrary identifier that the database assigns to the group when it is created. In another environment, or on another computer, the appusers group can have a different ID and it wouldn’t make any difference on the object.

To avoid using arbitrary identifiers, Django defines the concept of natural keys. A natural key is a unique identifier of an object that is not necessarily the primary key. In the case of groups, two groups can’t have the same name, so a natural key for the group can be its name.

To use natural keys instead of primary keys to reference related objects in a Django fixture, add the --natural-foreign flag to the dumpdata command:

$ python manage.py dumpdata auth.User --pk 1 --indent 4 --natural-foreign [ { "model": "auth.user", "pk": 1, "fields": { "password": "!f4dygH3ZWfd0214X59OR9ndwsRJ94HUZ6vQciG8y", "last_login": null, "is_superuser": false, "username": "haki", "first_name": "", "last_name": "", "email": "benita", "is_staff": false, "is_active": true, "date_joined": "2019-12-07T09:32:50.998Z", "groups": [ [ `appusers` ] ], "user_permissions": [] } } ]

Django generated the fixture for the user, but instead of using the primary key of the appusers group, it used the group’s name.

You can also add the --natural-primary flag to exclude an object’s primary key from the fixture. When pk is null, the primary key will be set at runtime, usually by the database.

Maintaining Django Fixtures

Django fixtures are great, but they also pose some challenges:

  • Keeping fixtures updated: Django fixtures must contain all the required fields of the model. If you add a new field that is not nullable, you must update the fixtures. Otherwise, they will fail to load. Keeping Django fixtures updated can become a burden when you have lots of them.

  • Maintaining dependencies between fixtures: Django fixtures that depend on other fixtures must be loaded together and in a particular order. Keeping up with fixtures as new test cases are added and old test cases are modified can be challenging.

For these reasons, Django fixtures are not an ideal choice for models that change often. For example, it would be very difficult to maintain Django fixtures for models that are used to represent core objects in the app such as sales, orders, transactions, or reservations.

On the other hand, Django fixtures are a great option for the following use cases:

  • Constant data: This applies to models that rarely change, such as country codes and zip codes.

  • Initial data: This applies to models that store your app’s lookup data, such as product categories, user groups, and user types.

pytest Fixtures in Django

In the previous section, you used the built-in tools provided by Django to create and load fixtures. The fixtures provided by Django are great for some use cases, but not ideal for others.

In this section, you’re going to experiment with a very different type of fixture: the pytest fixture. pytest provides a very extensive fixture system that you can use to create a reliable and maintainable test suite.

Setting Up pytest for a Django Project

To get started with pytest, you first need to install pytest and the Django plugin for pytest. Execute the following commands in your terminal while the virtual environment is activated:

$ pip install pytest $ pip install pytest-django

The pytest-django plugin is maintained by the pytest development team. It provides useful tools for writing tests for Django projects using pytest.

Next, you need to let pytest know where it can locate your Django project settings. Create a new file in the project’s root directory called pytest.ini, and add the following lines to it:

[pytest] DJANGO_SETTINGS_MODULE=django_fixtures.settings

This is the minimum amount of configuration needed to make pytest work with your Django project. There are many more configuration options, but this is enough to get started.

Finally, to test your setup, replace the contents of test.py with this dummy test:

def test_foo(): assert True

To run the dummy test, use the pytest command from your terminal:

$ pytest test.py ============================== test session starts ====================== platform linux -- Python 3.7.4, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 Django settings: django_fixtures.settings (from ini file) rootdir: /django_fixtures, inifile: pytest.ini plugins: django-3.5.1 test.py . [100%] ============================= 1 passed in 0.05s =========================

You just completed setting up a new Django project with pytest! Now you’re ready to dig deeper.

For more about how to set up pytest and write tests, check out Test-Driven Development With pytest.

Accessing the Database From Tests

In this section, you’re going to write tests using the built-in authentication module django.contrib.auth. The most familiar models in this module are User and Group.

To get started with both Django and pytest, write a test to check if the function create_user() provided by Django is setting the username correctly:

from django.contrib.auth.models import User def test_should_create_user_with_username() -> None: user = User.objects.create_user("Haki") assert user.username == "Haki"

Now, try to execute the test from your command like:

$ pytest test.py ================================== test session starts =============== platform linux -- Python 3.7.4, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 Django settings: django_fixtures.settings (from ini file) rootdir: /django-django_fixtures/django_fixtures, inifile: pytest.ini plugins: django-3.5.1 collected 1 item test.py F =============================== FAILURES ============================= ____________________test_should_create_user_with_username ____________ def test_should_create_user_with_username() -> None: > user = User.objects.create_user("Haki") self = <mydbengine.base.DatabaseWrapper object at 0x7fef66ed57d0>, name = None def _cursor(self, name=None): > self.ensure_connection() E Failed: Database access not allowed, use the "django_db" mark, or the "db" or "transactional_db" fixtures to enable it.

The command failed, and the test did not execute. The error message gives you some useful information: To access the database in a test you need to inject a special fixture called db. The db fixture is part of the django-pytest plugin you installed earlier, and it’s required to access the database in tests.

Inject the db fixture into the test:

from django.contrib.auth.models import User def test_should_create_user_with_username(db) -> None: user = User.objects.create_user("Haki") assert user.username == "Haki"

Run the test again:

$ pytest test.py ================================== test session starts =============== platform linux -- Python 3.7.4, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 Django settings: django_fixtures.settings (from ini file) rootdir: /django_fixtures, inifile: pytest.ini plugins: django-3.5.1 collected 1 item test.py .

Great! The command completed successfully and your test passed. You now know how to access the database in tests. You also injected a fixture into a test case along the way.

Creating Fixtures for Django Models

Now that you’re familiar with Django and pytest, write a test to check that a password set with set_password() is validated as expected. Replace the contents of test.py with this test:

from django.contrib.auth.models import User def test_should_check_password(db) -> None: user = User.objects.create_user("A") user.set_password("secret") assert user.check_password("secret") is True def test_should_not_check_unusable_password(db) -> None: user = User.objects.create_user("A") user.set_password("secret") user.set_unusable_password() assert user.check_password("secret") is False

The first test checks that a user with a usable password is being validated by Django. The second test checks an edge case in which the user’s password is unusable and should not be validated by Django.

There’s an important distinction to be made here: The test cases above don’t test create_user(). They test set_password(). That means a change to create_user() should not affect these test cases.

Also, notice that the User instance is created twice, once for each test case. A large project can have many tests that require a User instance. If every test case will create its own user, you might have trouble in the future if the User model changes.

To reuse an object in many test cases, you can create a test fixture:

import pytest from django.contrib.auth.models import User @pytest.fixture def user_A(db) -> User: return User.objects.create_user("A") def test_should_check_password(db, user_A: User) -> None: user_A.set_password("secret") assert user_A.check_password("secret") is True def test_should_not_check_unusable_password(db, user_A: User) -> None: user_A.set_password("secret") user_A.set_unusable_password() assert user_A.check_password("secret") is False

In the above code, you created a function called user_A() that creates and returns a new User instance. To mark the function as a fixture, you decorated it with the pytest.fixture decorator. Once a function is marked as a fixture, it can be injected into test cases. In this case, you injected the fixture user_A into two test cases.

Maintaining Fixtures When Requirements Change

Let’s say you’ve added a new requirement to your application, and now every user must belong to a special "app_user" group. Users in that group can view and update their own personal details. To test your app, you need your test users to belong to the "app_user" group as well:

import pytest from django.contrib.auth.models import User, Group, Permission @pytest.fixture def user_A(db) -> Group: group = Group.objects.create(name="app_user") change_user_permissions = Permission.objects.filter( codename__in=["change_user", "view_user"], ) group.permissions.add(*change_user_permissions) user = User.objects.create_user("A") user.groups.add(group) return user def test_should_create_user(user_A: User) -> None: assert user_A.username == "A" def test_user_is_in_app_user_group(user_A: User) -> None: assert user_A.groups.filter(name="app_user").exists()

Inside the fixture you created the group "app_user" and added the relevant change_user and view_user permissions to it. You then created the test user and added them to the "app_user" group.

Previously, you needed to go over every test case that created a user and add it to the group. Using fixtures, you were able to make the change just once. Once you changed the fixture, the same change appeared in every test case you injected user_A into. Using fixtures, you can avoid repetition and make your tests more maintainable.

Injecting Fixtures Into Other Fixtures

Large applications usually have more than just one user, and it’s often necessary to test them with multiple users. In this situation, you can add another fixture to create the test user_B:

import pytest from django.contrib.auth.models import User, Group, Permission @pytest.fixture def user_A(db) -> User: group = Group.objects.create(name="app_user") change_user_permissions = Permission.objects.filter( codename__in=["change_user", "view_user"], ) group.permissions.add(*change_user_permissions) user = User.objects.create_user("A") user.groups.add(group) return user @pytest.fixture def user_B(db) -> User: group = Group.objects.create(name="app_user") change_user_permissions = Permission.objects.filter( codename__in=["change_user", "view_user"], ) group.permissions.add(*change_user_permissions) user = User.objects.create_user("B") user.groups.add(group) return user def test_should_create_two_users(user_A: User, user_B: User) -> None: assert user_A.pk != user_B.pk

In your terminal, try running the test:

$ pytest test.py ==================== test session starts ================================= platform linux -- Python 3.7.4, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 Django settings: django_fixtures.settings (from ini file) rootdir: /django_fixtures, inifile: pytest.ini plugins: django-3.5.1 collected 1 item test.py E [100%] ============================= ERRORS ====================================== _____________ ERROR at setup of test_should_create_two_users ______________ self = <django.db.backends.utils.CursorWrapper object at 0x7fc6ad1df210>, sql ='INSERT INTO "auth_group" ("name") VALUES (%s) RETURNING "auth_group"."id"' ,params = ('app_user',) def _execute(self, sql, params, *ignored_wrapper_args): self.db.validate_no_broken_transaction() with self.db.wrap_database_errors: if params is None: # params default might be backend specific. return self.cursor.execute(sql) else: > return self.cursor.execute(sql, params) E psycopg2.IntegrityError: duplicate key value violates unique constraint "auth_group_name_key" E DETAIL: Key (name)=(app_user) already exists. ======================== 1 error in 4.14s ================================

The new test throws an IntegrityError. The error message originates from the database, so it might look a bit different depending on the database you are using. According to the error message, the test violates the unique constraint on the group’s name. When you look at your fixtures, it makes sense. The "app_user" group is created twice, once in the fixture user_A and once again in the fixture user_B.

An interesting observation we’ve overlooked to this point is that the fixture user_A is using the fixture db. This means that fixtures can be injected into other fixtures. You can use this feature to address the IntegrityError above. Create the "app_user" group just once in a fixture, and inject it into both the user_A and user_B fixtures.

To do so, refactor your test and add an "app user" group fixture:

import pytest from django.contrib.auth.models import User, Group, Permission @pytest.fixture def app_user_group(db) -> Group: group = Group.objects.create(name="app_user") change_user_permissions = Permission.objects.filter( codename__in=["change_user", "view_user"], ) group.permissions.add(*change_user_permissions) return group @pytest.fixture def user_A(db, app_user_group: Group) -> User: user = User.objects.create_user("A") user.groups.add(app_user_group) return user @pytest.fixture def user_B(db, app_user_group: Group) -> User: user = User.objects.create_user("B") user.groups.add(app_user_group) return user def test_should_create_two_users(user_A: User, user_B: User) -> None: assert user_A.pk != user_B.pk

In your terminal, run your tests:

$ pytest test.py ================================== test session starts =============== platform linux -- Python 3.7.4, pytest-5.2.0, py-1.8.0, pluggy-0.13.0 Django settings: django_fixtures.settings (from ini file) rootdir: /django_fixtures, inifile: pytest.ini plugins: django-3.5.1 collected 1 item test.py .

Amazing! Your tests pass. The group fixture encapsulates the logic related to the "app user" group, such as setting permissions. You then injected the group into two separate user fixtures. By constructing your fixtures this way, you’ve made your tests less complicated to read and maintain.

Using a Factory

So far, you’ve created objects with very few arguments. However, some objects may be more complicated, featuring many arguments with many possible values. For such objects, you might want to create several test fixtures.

For example, if you provide all arguments to create_user(), this is what the fixture would look like:

import pytest from django.contrib.auth.models import User @pytest.fixture def user_A(db, app_user_group: Group) -> User user = User.objects.create_user( username="A", password="secret", first_name="haki", last_name="benita", email="me@hakibenita.com", is_staff=False, is_superuser=False, is_active=True, ) user.groups.add(app_user_group) return user

Your fixture just got a lot more complicated! A user instance can now have many different variations, such as superuser, staff user, inactive staff user, and inactive regular user.

In previous sections, you learned that it can be hard to maintain complicated setup logic in each test fixture. So, to avoid having to repeat all the values every time you create a user, add a function that uses create_user() to create a user according to your app’s specific needs:

from typing import List, Optional from django.contrib.auth.models import User, Group def create_app_user( username: str, password: Optional[str] = None, first_name: Optional[str] = "first name", last_name: Optional[str] = "last name", email: Optional[str] = "foo@bar.com", is_staff: str = False, is_superuser: str = False, is_active: str = True, groups: List[Group] = [], ) -> User: user = User.objects.create_user( username=username, password=password, first_name=first_name, last_name=last_name, email=email, is_staff=is_staff, is_superuser=is_superuser, is_active=is_active, ) user.groups.add(*groups) return user

The function creates an app user. Each argument is set with a sensible default according to your app’s specific requirements. For example, your app might require that every user has an email address, but Django’s built-in function does not enforce such a restriction. You can enforce that requirement in your function instead.

Functions and classes that create objects are often referred to as factories. Why? It’s because these functions act as factories that produce instances of a specific class. For more about factories in Python, check out The Factory Method Pattern and Its Implementation in Python.

The function above is a straightforward implementation of a factory. It holds no state and it’s not implementing any complicated logic. You can refactor your tests so that they use the factory function to create user instances in your fixtures:

@pytest.fixture def user_A(db, app_user_group: Group) -> User: return create_user(username="A", groups=[app_user_group]) @pytest.fixture def user_B(db, app_user_group: Group) -> User: return create_user(username="B", groups=[app_user_group]) def test_should_create_user(user_A: User, app_user_group: Group) -> None: assert user_A.username == "A" assert user_A.email == "foo@bar.com" assert user_A.groups.filter(pk=app_user_group.pk).exists() def test_should_create_two_users(user_A: User, user_B: User) -> None: assert user_A.pk != user_B.pk

Your fixtures got shorter, and your tests are now more resilient to change. For example, if you used a custom user model and you just added a new field to the model, you would only need to change create_user() for your tests to work as expected.

Using Factories as Fixtures

Complicated setup logic makes it harder to write and maintain tests, making the entire suite fragile and less resilient to change. So far, you’ve addressed this issue by creating fixtures, creating dependencies between fixtures, and using a factory to abstract as much of the setup logic as possible.

But there is still some setup logic left in your test fixtures:

@pytest.fixture def user_A(db, app_user_group: Group) -> User: return create_user(username="A", groups=[app_user_group]) @pytest.fixture def user_B(db, app_user_group: Group) -> User: return create_user(username="B", groups=[app_user_group])

Both fixtures are injected with app_user_group. This is currently necessary because the factory function create_user() does not have access to the app_user_group fixture. Having this setup logic in each test makes it harder to make changes, and it’s more likely to be overlooked in future tests. Instead, you want to encapsulate the entire process of creating a user and abstract it from the tests. This way, you can focus on the scenario at hand rather than setting up unique test data.

To provide the user factory with access to the app_user_group fixture, you can use a pattern called factory as fixture:

from typing import List, Optional import pytest from django.contrib.auth.models import User, Group, Permission @pytest.fixture def app_user_group(db) -> Group: group = Group.objects.create(name="app_user") change_user_permissions = Permission.objects.filter( codename__in=["change_user", "view_user"], ) group.permissions.add(*change_user_permissions) return group @pytest.fixture def app_user_factory(db, app_user_group: Group): # Closure def create_app_user( username: str, password: Optional[str] = None, first_name: Optional[str] = "first name", last_name: Optional[str] = "last name", email: Optional[str] = "foo@bar.com", is_staff: str = False, is_superuser: str = False, is_active: str = True, groups: List[Group] = [], ) -> User: user = User.objects.create_user( username=username, password=password, first_name=first_name, last_name=last_name, email=email, is_staff=is_staff, is_superuser=is_superuser, is_active=is_active, ) user.groups.add(app_user_group) # Add additional groups, if provided. user.groups.add(*groups) return user return create_app_user

This is not far from what you’ve already done, so let’s break it down:

  • The app_user_group fixture remains the same. It creates the special "app user" group with all the necessary permissions.

  • A new fixture called app_user_factory is added, and it is injected with the app_user_group fixture.

  • The fixture app_user_factory creates a closure and returns an inner function called create_app_user().

  • create_app_user() is similar to the function you previously implemented, but now it has access to the fixture app_user_group. With access to the group, you can now add users to app_user_group in the factory function.

To use the app_user_factory fixture, inject it into another fixture and use it to create a user instance:

@pytest.fixture def user_A(db, app_user_factory) -> User: return app_user_factory("A") @pytest.fixture def user_B(db, app_user_factory) -> User: return app_user_factory("B") def test_should_create_user_in_app_user_group( user_A: User, app_user_group: Group, ) -> None: assert user_A.groups.filter(pk=app_user_group.pk).exists() def test_should_create_two_users(user_A: User, user_B: User) -> None: assert user_A.pk != user_B.pk

Notice that, unlike before, the fixture you created is providing a function rather than an object. This is the main concept behind the factory as fixture pattern: The factory fixture creates a closure, which provides the inner function with access to fixtures.

For more about closures in Python, check out Python Inner Functions — What Are They Good For?

Now that you have your factories and fixtures, this is the complete code for your test:

from typing import List, Optional import pytest from django.contrib.auth.models import User, Group, Permission @pytest.fixture def app_user_group(db) -> Group: group = Group.objects.create(name="app_user") change_user_permissions = Permission.objects.filter( codename__in=["change_user", "view_user"], ) group.permissions.add(*change_user_permissions) return group @pytest.fixture def app_user_factory(db, app_user_group: Group): # Closure def create_app_user( username: str, password: Optional[str] = None, first_name: Optional[str] = "first name", last_name: Optional[str] = "last name", email: Optional[str] = "foo@bar.com", is_staff: str = False, is_superuser: str = False, is_active: str = True, groups: List[Group] = [], ) -> User: user = User.objects.create_user( username=username, password=password, first_name=first_name, last_name=last_name, email=email, is_staff=is_staff, is_superuser=is_superuser, is_active=is_active, ) user.groups.add(app_user_group) # Add additional groups, if provided. user.groups.add(*groups) return user return create_app_user @pytest.fixture def user_A(db, app_user_factory) -> User: return app_user_factory("A") @pytest.fixture def user_B(db, app_user_factory) -> User: return app_user_factory("B") def test_should_create_user_in_app_user_group( user_A: User, app_user_group: Group, ) -> None: assert user_A.groups.filter(pk=app_user_group.pk).exists() def test_should_create_two_users(user_A: User, user_B: User) -> None: assert user_A.pk != user_B.pk

Open the terminal and run the test:

$ pytest test.py ======================== test session starts ======================== platform linux -- Python 3.8.1, pytest-5.3.3, py-1.8.1, pluggy-0.13.1 django: settings: django_fixtures.settings (from ini) rootdir: /django_fixtures/django_fixtures, inifile: pytest.ini plugins: django-3.8.0 collected 2 items test.py .. [100%] ======================== 2 passed in 0.17s ==========================

Great job! You’ve successfully implemented the factory as fixture pattern in your tests.

Factories as Fixtures in Practice

The factory as fixture pattern is very useful. So useful, in fact, that you can find it in the fixtures provided by pytest itself. For example, the tmp_path fixture provided by pytest is created by the fixture factory tmp_path_factory. Likewise, the tmpdir fixture is created by the fixture factory tmpdir_factory.

Mastering the factory as fixture pattern can eliminate many of the headaches associated with writing and maintaining tests.

Conclusion

You’ve successfully implemented a fixture factory that provides Django model instances. You’ve also maintained and implemented dependencies between fixtures in a way that takes some of the hassle out of writing and maintaining tests.

In this tutorial, you’ve learned:

  • How to create and load fixtures in Django
  • How to provide test fixtures for Django models in pytest
  • How to use factories to create fixtures for Django models in pytest
  • How to implement the factory as fixture pattern to create dependencies between test fixtures

You’re now able to implement and maintain a solid test suite that will help you produce better and more reliable code, faster!

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Stack Abuse: The Singleton Design Pattern in Python

Planet Python - Wed, 2020-04-08 08:39
Introduction

In this article, we'll be diving into the Singleton Design Pattern, implemented in Python.

As time progresses, software gets more tailored to solving specific problems in different domains. While there are many difference in the application-level of our software, some aspects of software design remain largely the same. These aspects might not remain the same for all software out there but will hold true for a lot of scenarios. Therefore, learning and understanding them will be highly beneficial in helping us build resilient programs.

This is the first in a series about Design Patterns in Python and the different patterns that we can utilize to build software.

What is a Design Pattern?

A design pattern is a particular approach to solving a recurring problem in software development and is also used to represent good practices. It is not actual code, but it represents a particular way of organizing code to produce the desired solution in a recommended way. This means that we do not have to strictly follow set design patterns for all of our code, but analyze our needs and apply the best-suited design pattern for our needs.

Design patterns arise from a wealth of experience in solving problems in software development and are tried and tested to fit particular scenarios. Another reason to learn about design patterns is that, while a pattern may not work for a particular scenario, it can offer a base off which a solution can be formulated.

However, as much as design patterns can help us out of various situations, it is important to assess the current situation and explore all the options available before jumping directly into a pattern because there may be a better solution out there.

Design patterns are divided into a few broad categories, though mainly Creational patterns, Structural patterns, and Behavioral patterns.

Creational Patterns

There are several aspects of design patterns that set them apart from each other, including the complexity of the design pattern, the application level within a system, and the amount of detail.

Creational patterns include those that define ways to create objects that contribute to increased flexibility and reusability of code throughout the entire application.

Examples of creational patterns include the Singleton pattern, Factory Method, Abstract Factory, Builder and Prototype patterns.

The Singleton Pattern Definition

The singleton pattern is a common creational pattern that is used to define the creation of a single instance of a class while providing a single global access point to that object.

This pattern restricts the number of objects that can be created from a class to one single object that will often-times be shared globally in an application.

Motivation

This pattern is commonly implemented in features that require control over access to a shared resource such as a database connection or a file. By ensuring that a class can only be used to create a single instance and providing a single global access point, access to the shared resource can be restricted and integrity can be maintained.

The creation of single instances also helps ensure that some aspects of our programs cannot be overwritten by other classes resulting in unsafe or inefficient code. This also allows us to access the same object in multiple points of our programs without the fear that it may be overwritten at some point in our program.

For instance, database connections are done once in our programs, and the same object is used to perform operations on our database all over the application. If different parts of our application could create their own database connections, integrity issues may arise over time as each part tries to access the database on their own.

Implementation

The Singleton pattern requires that the instantiation of a class is restricted to one object only. The control of object creation is achieved by implementing a creation method that saves the created object in a static field.

All calls to this creation method either return the original singleton object or an error signaling the existence of an instantiated object. This prevents the creation of more than one object for our class and maintains the singleton property.

A good analogy of a singleton pattern is that a country can have a single government that controls access and operations within the country. Any attempts to create another government are forbidden.

We can implement this government analogy in a singleton class as follows in Python:

class SingletonGovt: __instance__ = None def __init__(self): """ Constructor. """ if SingletonGovt.__instance__ is None: SingletonGovt.__instance__ = self else: raise Exception("You cannot create another SingletonGovt class") @staticmethod def get_instance(): """ Static method to fetch the current instance. """ if not SingletonGovt.__instance__: SingletonGovt() return SingletonGovt.__instance__

In our example, we define the variable that will hold the single object to be instantiated. Our constructor checks if there is an existing class and raises an error.

When fetching the object using the get_instance() method, we check whether an existing instance is available and return it. If not, we create one and return it.

Our SingletonGovt in action:

government = SingletonGovt() print(government) same_government = SingletonGovt.get_instance() print(same_government) another_government = SingletonGovt.get_instance() print(another_government) new_government = SingletonGovt() print(new_government)

When we run our script, we can see that we have just one SingletonGovt instance stored at a single point in memory. Any attempts to create another government are thwarted by the exception that we raise:

Pros and Cons Pros
  • The Singleton pattern offers the guarantee that only one instance of our class exists and reduces the risk of unexpected behavior in our program.
  • Since the creation of the class is controlled by a single class, this offers flexibility since changes only need to be made to one class and object.
Cons
  • A class created using the singleton pattern violates the Single Responsibility Principle since the class may have to handle more than one responsibility at a given time.
  • Life cycle management can pose problems in other areas such as testing since the singleton class is kept alive during the lifespan of the application and different test cases might require new versions of the class.
Conclusion

In this post, we have introduced, discussed and implemented the Singleton Design Pattern.

Just like any other design pattern, it has its pros and cons and while it may be suitable for some situations, it might not apply to all our development needs. It is therefore up to us to analyze the problem at hand and make the decision whether or not the singleton pattern will make our work easier.

Categories: FLOSS Project Planets

Anwesha Das: Analysis of Aarogya Setu

Planet Python - Wed, 2020-04-08 07:29

The COVID 19 pandemic has changed the world we knew. Individuals, Government, everybody is trying to fight it out, cope with it. But to deal with the immediate problem, we often overlook the issues which are brought by the solutions.

The Government of India has developed an app called Aarogya Setu. This is an app to trace out the people the user came in contact with. If there is someone who is or can be a Covid19 positive among those people, the app notifies the user and authorities. So they can take action. The app has already gained massive popularity among people and has already been downloaded over 10 million times.

At a glance, the app and the aim behind it seem innocent and right. But if we dig deeper, we will be able to find out the real intention of the app, which is massive blanket data collection. ‘Data Retention’ and not ‘Data Deletion” is at the very core of the privacy policy of the app.

There is a different prominent organization that has laid down some essential points to keep in mind while developing such apps. EFF says “our advice to organizations that consider sharing aggregate location data: Get consent from the users who supply the data. Be cautious about the details. Aggregate on the highest level of generality that will be useful. Share your plans with the public before you release the data. And avoid sharing “deidentified” or “anonymized” location data that is not aggregated—it doesn’t work.” Chaos Computer Club defines contact tracing as an inherently risky technology. They further say that privacy should be at the fundamental base of any such app. Here are some notable work done in the field by :

It is not only India but many such Governments all around the world have developed such apps. The similar app developed by the Singapore Government is called TraceTogether. But what is at the very core of the app is the user’s privacy and security. Whereas the Indian app takes a bunch of information from the user. Such as - name, sex, age, profession, if or not you are a smoker. The only information TraceTogether takes is the phone number of the user, that is it. Also, where TraceTogether deletes the data after 21 days. But Aarogya Setu does not mention any specific time.

We at HasGeek along with Kaarana community organizing an event on “Analysis of Aarogya Setu Mobile Application. An in-depth discussion of how the official COVID19 disease surveillance app works” today at 6 pm. We have speakers from different fields and expertise starting from security, law, data science, policy, and activism. Also, we will have a speaker from GovTech Singapore to give their view and intent to create Trace Together.

See you all there.

Categories: FLOSS Project Planets

The Digital Cat: Multiple inheritance and mixin classes in Python

Planet Python - Wed, 2020-04-08 06:47

I recently revisited three old posts on Django class-based views that I wrote for this blog, updating them to Django 3.0 (you can find them here) and noticed once again that the code base uses mixin classes to increase code reuse. I also realised that mixins are not very popular in Python, so I decided to explore them, brushing up my knowledge of the OOP theory in the meanwhile.

To fully appreciate the content of the post, be sure you grasp two pillars of the OOP approach: delegation, in particular how it is implemented through inheritance, and polymorphism. This post about delegation and this post about polymorphism contain all you need to understand how Python implements those concepts.

Multiple inheritance: blessing and curse General concepts

To discuss mixins we need to start from one of the most controversial subjects in the whole OOP world: multiple inheritance. This is a natural extension of the concept of simple inheritance, where a class automatically delegates method and attribute resolution to another class (the parent class).

Let me state it again, as it is important for the rest of the discussion: inheritance is just an automatic delegation mechanism.

Delegation was introduced in OOP as a way to reduce code duplication. When an object needs a specific feature it just delegates it to another class (either explicitly or implicitly), so the code is written just once.

Let's consider the example of code management website, clearly completely fictional and not inspired by any existing product. Let's assume we created the following hierarchy

assignable reviewable item (assign_to_user, ask_review_to_user) ^ | | | pull request

which allows us to put in pull request only the specific code required by that element. This is a great achievement, as it is what libraries do for code, but on live objects. Method calls and delegation are nothing more than messages between objects, so the delegation hierarchy is just a simple networked system.

Unfortunately, the use of inheritance over composition often leads to systems that, paradoxically, increase code duplication. The main problem lies in the fact that inheritance can directly delegate to only one other class (the parent class), as opposed to composition, where the object can delegate to any number of other ones. This limitation of inheritance means that we might have a class that inherits from another one because it needs some of its features, but doing this receives features it doesn't want, or shouldn't have.

Let's continue the example of the code management portal, and consider an issue, which is an item that we want to store in the system, but cannot be reviewed by a user. If we create a hierarchy like this

assignable reviewable item (assign_to_user, ask_review_to_user) ^ | | | | +--------+--------+ | | | | | | issue pull request (not reviewable)

we end up putting the features related to the review process in an object that shouldn't have them. The standard solution to this problem is that of increasing the depth of the inheritance hierarchy and to derive from the new simpler ancestor.

assignable item (assign_to_user) ^ | | | | +------+--------------+ | | | | | | | reviewable assignable item | (ask_review_to_user) | ^ | | | | | | issue pull request

However, this approach stops being viable as soon as an object needs to inherit from a given class but not from the parent of that class. For example, an element that has to be reviewable but not assignable, like a best practice that we want to add to the site. If we want to keep using inheritance, the only solution at this point is to duplicate the code that implements the reviewable nature of the item (or the code that implements the assignable feature) and create two different class hierarchies.

assignable item +--------> reviewable item (assign_to_user) | (ask_review_to_user) ^ | ^ | | | | | | | CODE DUPLICATION | | | | +------+--------------+ | | | | | | | | | | | | V | | reviewable assignable item | | (ask_review_to_user) | | ^ | | | | | | | | | | issue pull request best practice

Please note that this doesn't even take into account that the new reviewable item might need attributes from assignable item, which prompts for another level of depth in the hierarchy, where we isolate those features in a more generic class. So, unfortunately, chances are that this is only the first of many compromises we will have to accept to keep the system in a stable state if we can't change our approach.

Multiple inheritance was then introduced in OOP, as it was clear that an object might want to delegate certain actions to a given class, and other actions to a different one, mimicking what life forms do when they inherit traits from multiple ancestors (parents, grandparents, etc.).

The above situation can then be solved having pull request inherit from both the class that provides the assign feature and from the one that implements the reviewable nature.

assignable item reviewable item (assign_to_user) (ask_review_to_user) ^ ^ ^ | | | | | | | | | | | | +------+-------------+ +----------------------+ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | issue pull request best practice

Generally speaking, then, multiple inheritance is introduced to give the programmer a way to keep using inheritance without introducing code duplication, keeping the class hierarchy simpler and cleaner. Eventually, everything we do in software design is to try and separate concerns, that is, to isolate features, and multiple inheritance can help to do this.

These are just examples and might be valid or not, depending on the concrete case, but they clearly show the issues that we can have even with a very simple hierarchy of 4 classes. Many of these problems clearly arise from the fact that we wanted to implement delegation only through inheritance, and I dare to say that 80% of the architectural errors in OOP projects come from using inheritance instead of composition and from using god objects, that is classes that have responsibilities over too many different parts of the system. Always remember that OOP was born with the idea of small objects interacting through messages, so the considerations we make for monolithic architectures are valid even here.

That said, as inheritance and composition implement two different types of delegation (to be and to have), they are both valuable, and multiple inheritance is the way to remove the single provider limitation that comes from having only one parent class.

Why is it controversial?

Given what I just said, multiple inheritance seems to be a blessing. When an object can inherit from multiple parents, we can easily spread responsibilities among different classes and use only the ones we need, promoting code reuse and avoiding god objects.

Unfortunately, things are not that simple. First of all, we face the issue that every microservice-oriented architecture faces, that is the risk of going from god objects (the extreme monolithic architecture) to almost empty objects (the extreme distributed approach), burdening the programmer with too a fine-grained control that eventually results in a system where relationships between objects are so complicated that it becomes impossible to grasp the effect of a change in the code.

There is a more immediate problem in multiple inheritance, though. As it happens with the natural inheritance, parents can provide the same "genetic trait" in two different flavours, but the resulting individual will have only one. Leaving aside genetics (which is incredibly more complicated than programming) and going back to OOP, we face a problem when an object inherits from two other objects that provide the same attribute.

So, if your class Child inherits from parents Parent1 and Parent2, and both provide the __init__ method, which one should your object use?

class Parent1(): def __init__(self): [...] class Parent2(): def __init__(self): [...] class Child(Parent1, Parent2): # This inherits from both Parent1 and Parent2, which __init__ does it use? pass

Things can even get worse, as parents can have different signatures of the common method, for example

class Parent1: # This inherits from Ancestor but redefines __init__ def __init__(self, status): [...] class Parent2: # This inherits from Ancestor but redefines __init__ def __init__(self, name): [...] class Child(Parent1, Parent2): # This inherits from both Parent1 and Parent2, which __init__ does it use? pass

The problem can be extended even further, introducing a common ancestor above Parent1 and Parent2.

class Ancestor: # The common ancestor, defines its own __init__ method def __init__(self): [...] class Parent1(Ancestor): # This inherits from Ancestor but redefines __init__ def __init__(self, status): [...] class Parent2(Ancestor): # This inherits from Ancestor but redefines __init__ def __init__(self, name): [...] class Child(Parent1, Parent2): # This inherits from both Parent1 and Parent2, which __init__ does it use? pass

As you can see, we already have a problem when we introduce multiple parents, and a common ancestor just adds a new level of complexity. The ancestor class can clearly be at any point of the inheritance tree (grandparent, grand-grandparent, etc.), the important part is that it is shared between Parent1 and Parent2. This is the so-called diamond problem, as the inheritance graph has the shape of a diamond

Ancestor ^ ^ / \ / \ Parent1 Parent2 ^ ^ \ / \ / Child

So, while with single-parent inheritance the rules are straightforward, with multiple inheritance we immediately have a more complex situation that doesn't have a trivial solution. Does all this prevent multiple inheritance from being implemented?

Not at all! There are solutions to this problem, as we will see shortly, but this further level of intricacy makes multiple inheritance something that doesn't fit easily in a design and has to be implemented carefully to avoid subtle bugs. Remember that inheritance is an automatic delegation mechanism, as this makes what happens in the code less evident. For these reasons, multiple inheritance is often depicted as scary and convoluted, and usually given some space only in the advanced OOP courses, at least in the Python world. I believe every Python programmer, instead, should familiarise with it and learn how to take advantage of it.

Multiple inheritance: the Python way

Let's see how it is possible to solve the diamond problem. Unlike genetics, we programmers can't afford any level of uncertainty or randomness in our processes, so in the presence of a possible ambiguity as the one created by multiple inheritance, we need to write down a rule that will be strictly followed in every case. In Python, this rule goes by the name of MRO (Method Resolution Order), which was introduced in Python 2.3 and is described in this document by Michele Simionato.

There is a lot to say about MRO and the underlying C3 linearisation algorithm, but for the scope of this post, it is enough to see how it solves the diamond problem. In case of multiple inheritance, Python follows the usual inheritance rules (automatic delegation to an ancestor if the attribute is not present locally), but the order followed to traverse the inheritance tree now includes all the classes that are specified in the class signature. In the example above, Python would look for attributes in the following order: Child, Parent1, Parent2, Ancestor.

So, as in the case of standard inheritance, this means that the first class in the list that implements a specific attribute will be the selected provider for that resolution. An example might clarify the matter

class Ancestor: def rewind(self): [...] class Parent1(Ancestor): def open(self): [...] class Parent2(Ancestor): def open(self): [...] def close(self): [...] def flush(self): [...] class Child(Parent1, Parent2): def flush(self): [...]

In this case an instance c of Child would provide rewind, open, close, and flush. When c.rewind is called, the code in Ancestor is executed, as this is the first class in the MRO list that provides that method. The method open is provided by Parent1, while close is provided by Parent2. If the method c.flush is called, the code is provided by the Child class itself, that redefines it overriding the one provided by Parent2.

As we see with the flush method, Python doesn't change its behaviour when it comes to method overriding with multiple parents. The first implementation of a method with that name is executed, and the parent's implementation is not automatically called. As in the case of standard inheritance, then, it's up to us to design classes with matching method signatures.

Under the bonnet

How does multiple inheritance work internally? How does Python create the MRO list?

Python has a very simple approach to OOP (even though it ultimately ends with a mind-blowing ouroboros, see here). Classes are objects themselves, so they contain data structures that are used by the language to provide features, and delegation makes no exception. When we run a method on an object, Python silently uses the __getattribute__ method (provided by object), which uses __class__ to reach the class from the instance, and __bases__ to find the parent classes. The latter, in particular, is a tuple, so it is ordered, and it contains all the classes that the current class inherits from.

The MRO is created using only __bases__, but the underlying algorithm is not that trivial and has to with the monotonicity of the resulting class linearisation. It is less scary than it sounds, but not something you want to read while suntanning, probably. If that's the case, the aforementioned document by Michele Simionato contains all the gory details on class linearisation that you always wanted to explore while lying on the beach.

Inheritance and interfaces

To approach mixins, we need to discuss inheritance in detail, and specifically the role of method signatures.

In Python, when you override a method provided by an ancestor class, you have to decide if and when to call its original implementation. This gives the programmer the freedom to decide whether they need to just augment a method or to replace it completely. Remember that the only thing Python does when a class inherits from another is to automatically delegate methods that are not implemented.

When a class inherits from another we are ideally creating objects that keep the backward compatibility with the interface of the parent class, to allow a polymorphic use of them. This means that when we inherit from a class and override a method changing its signature we are doing something that is dangerous and, at least from the point of view of polymorphism, wrong. Have a look at this example

class GraphicalEntity: def __init__(self, pos_x, pos_y, size_x, size_y): self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y def move(self, pos_x, pos_y): self.pos_x = pos_x self.pos_y = pos_y def resize(self, size_x, size_y): self.size_x = size_x self.size_y = size_y class Rectangle(GraphicalEntity): pass class Square(GraphicalEntity): def __init__(self, pos_x, pos_y, size): super().__init__(pos_x, pos_y, size, size) def resize(self, size): super().resize(size, size)

Please note that Square changes the signature of both __init__ and resize. Now, when we instantiate those classes we need to keep in mind the different signature of __init__ in Square

r1 = Rectangle(100, 200, 15, 30) r2 = Rectangle(150, 280, 23, 55) q1 = Square(300, 400, 50)

We usually accept that an enhanced version of a class accepts more parameters when it is initialized, as we do not expect it to be polymorphic on __init__. Problems arise when we try to leverage polymorphism on other methods, for example resizing all GraphicalEntity objects in a list

for shape in [r1, r2, q1]: size_x = shape.size_x size_y = shape.size_y shape.resize(size_x*2, size_y*2)

Since r1, r2, and q1 are all objects that inherit from GraphicalEntity we expect them to provide the interface provided by that class, but this fails, because Square changed the signature of resize. The same would happen if we instantiated them in a for loop from a list of classes, but as I said it is generally accepted that child classes change the signature of the __init__ method. This is not true, for example, in a plugin-based system, where all plugins shall be initialized the same way.

This is a classic problem in OOP. While we, as humans, perceive a square just as a slightly special rectangle, from the interface point of view the two classes are different, and thus should not be in the same inheritance tree when we are dealing with dimensions. This is an important consideration: Rectangle and Square are polymorphic on the move method, but not on __init__ and resize. So, the question is if we could somehow separate the two natures of being movable and resizable.

Now, discussing interfaces, polymorphism, and the reasons behind them would require an entirely separate post, so in the following sections, I'm going to ignore the matter and just consider the object interface optional. You will thus find examples of objects that break the interface of the parent, and objects that keep it. Just remember: whenever you change the signature of a method you change the (implicit) interface of the object, and thus you stop polymorphism. I'll discuss another time if I consider this right or wrong.

Mixin classes

MRO is a good solution that prevents ambiguity, but it leaves programmers with the responsibility of creating sensible inheritance trees. The algorithm helps to resolve complicated situations, but this doesn't mean we should create them in the first place. So, how can we leverage multiple inheritance without creating systems that are too complicated to grasp? Moreover, is it possible to use multiple inheritance to solve the problem of managing the double (or multiple) nature of an object, as in the previous example of a movable and resizable shape?

The solution comes from mixin classes: those are small classes that provide attributes but are not included in the standard inheritance tree, working more as "additions" to the current class than as proper ancestors. Mixins originate in the LISP programming language, and specifically in what could be considered the first version of the Common Lisp Object System, the Flavors extension. Modern OOP languages implement mixins in many different ways: Scala, for example, has a feature called traits, which live in their own space with a specific hierarchy that doesn't interfere with the proper class inheritance.

Mixin classes in Python

Python doesn't provide support for mixins with any dedicated language feature, so we use multiple inheritance to implement them. This clearly requires great discipline from the programmer, as it violates one of the main assumptions for mixins: their orthogonality to the inheritance tree. In Python, so-called mixins are classes that live in the normal inheritance tree, but they are kept small to avoid creating hierarchies that are too complicated for the programmer to grasp. In particular, mixins shouldn't have common ancestors other than object with the other parent classes.

Let's have a look at a simple example

class GraphicalEntity: def __init__(self, pos_x, pos_y, size_x, size_y): self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y class ResizableMixin: def resize(self, size_x, size_y): self.size_x = size_x self.size_y = size_y class ResizableGraphicalEntity(GraphicalEntity, ResizableMixin): pass

Here, the class ResizableMixin doesn't inherit from GraphicalEntity, but directly from object, so ResizableGraphicalEntity gets from it just the resize method. As we said before, this simplifies the inheritance tree of ResizableGraphicalEntity and helps to reduce the risk of the diamond problem. It leaves us free to use GraphicalEntity as a parent for other classes without having to inherit methods that we don't want. Please remember that this happens because the classes are designed to avoid it, and not because of language features: the MRO algorithm just ensures that there will always be an unambiguous choice in case of multiple ancestors.

Mixins cannot usually be too generic. After all, they are designed to add features to classes, but these new features often interact with other pre-existing features of the augmented class. In this case, the resize method interacts with the attributes size_x and size_y that have to be present in the object. Obviously, there are obviously examples of pure mixins, but since they would require no initialization their scope is definitely limited.

Using mixins to hijack inheritance

Thanks to the MRO, Python programmers can leverage multiple inheritance to override methods that objects inherit from their parents, allowing them to customise classes without code duplication. Let's have a look at this example

class GraphicalEntity: def __init__(self, pos_x, pos_y, size_x, size_y): self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y class Button(GraphicalEntity): def __init__(self, pos_x, pos_y, size_x, size_y): super().__init__(pos_x, pos_y, size_x, size_y) self.status = False def toggle(self): self.status = not self.status b = Button(10, 20, 200, 100)

As you can see the Button class extends the GraphicalEntity one in a classic way, using super to call the parent's __init__ method before adding the new status attribute. Now, if I wanted to create a SquareButton class I have two choices.

I might just override __init__ in the new class

class GraphicalEntity: def __init__(self, pos_x, pos_y, size_x, size_y): self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y class Button(GraphicalEntity): def __init__(self, pos_x, pos_y, size_x, size_y): super().__init__(pos_x, pos_y, size_x, size_y) self.status = False def toggle(self): self.status = not self.status class SquareButton(Button): def __init__(self, pos_x, pos_y, size): super().__init__(pos_x, pos_y, size, size) b = SquareButton(10, 20, 200)

which performs the requested job, but strongly connects the feature of having a single dimension with the Button nature. If we wanted to create a circular image we could not inherit from SquareButton, as the image has a different nature.

The second option is that of isolating the features connected with having a single dimension in a mixin class, and add it as a parent for the new class

class GraphicalEntity: def __init__(self, pos_x, pos_y, size_x, size_y): self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y class Button(GraphicalEntity): def __init__(self, pos_x, pos_y, size_x, size_y): super().__init__(pos_x, pos_y, size_x, size_y) self.status = False def toggle(self): self.status = not self.status class SingleDimensionMixin: def __init__(self, pos_x, pos_y, size): super().__init__(pos_x, pos_y, size, size) class SquareButton(SingleDimensionMixin, Button): pass b = SquareButton(10, 20, 200)

The second solution gives the same final result, but promotes code reuse, as now the SingleDimensionMixin class can be applied to other classes derived from GraphicalEntity and make them accept only one size, while in the first solution that feature was tightly connected with the Button ancestor class.

Please note that the position of the mixin is important. As super follows the MRO, the called method is dispatched to the nearest class in the linearisation. If you put SingleDimensionMixin after Button in the definition of SquareButton, Python would complain. In that case the call b = SquareButton(10, 20, 200) and the method signature __init__(self, pos_x, pos_y, size_x, size_y) would not match.

Mixins are not used only when you want to change the object's interface, though. Leveraging super we can achieve interesting designs like

class GraphicalEntity: def __init__(self, pos_x, pos_y, size_x, size_y): self.pos_x = pos_x self.pos_y = pos_y self.size_x = size_x self.size_y = size_y class Button(GraphicalEntity): def __init__(self, pos_x, pos_y, size_x, size_y): super().__init__(pos_x, pos_y, size_x, size_y) self.status = False def toggle(self): self.status = not self.status class LimitSizeMixin: def __init__(self, pos_x, pos_y, size): size_x = min(size_x, 500) size_y = min(size_y, 400) super().__init__(pos_x, pos_y, size_x, size_y) class LimitSizeButton(Button, LimitSizeMixin): pass b = LimitSizeButton(10, 20, 200, 100)

Here, LimitSizeButton calls __init__ of its first parent, which is Button. This, however, delegates the call to the next class in the MRO before initialising self.status, so the call is dispatched to LimitSizeMixin, that first operates some changes and eventually dispatches it to the original recipient, GraphicalEntity.

Remember that in Python, you are never forced to call the parent's implementation of a method, so the mixin here might also stop the dispatching mechanism if that is the requirement of the business logic of the new object.

A real example: Django class-based views

Finally, let's get to the original source of inspiration for this post: the Django codebase. I will show you here how the Django programmers used multiple inheritance and mixin classes to promote code reuse, and you will now hopefully grasp all the reasons behind them.

The example I chose can be found in the code of generic views, and in particular in two classes: TemplateResponseMixin and TemplateView.

As you might know, Django View class is the ancestor of all class-based views and provides a dispatch method that converts HTTP request methods into Python function calls (CODE). Now, the TemplateView is a view that answers to a GET request rendering a template with the data coming from a context passed when the view is called. Given the mechanism behind Django views, then, TemplateView should implement a get method and return the content of the HTTP response. The code of the class is

class TemplateView(TemplateResponseMixin, ContextMixin, View): """ Render a template. Pass keyword arguments from the URLconf to the context. """ def get(self, request, *args, **kwargs): context = self.get_context_data(**kwargs) return self.render_to_response(context)

As you can see TemplateView is a View, but it uses two mixins to inject features. Let's have a look at TemplateResponseMixin

class TemplateResponseMixin: [...] def render_to_response(self, context, **response_kwargs): [...] def get_template_names(self): [...]

[I removed the code of the class as it is not crucial for the present discussion, you can see the full class here]

It is clear that TemplateResponseMixin just adds to any class the two methods get_template_names and render_to_response. The latter is called in the get method of TemplateView to create the response. Let's have a look at a simplified schema of the calls:

GET request --> TemplateView.dispatch --> View.dispatch --> TemplateView.get --> TemplateResponseMixin.render_to_response

It might look complicated, but try to follow the code a couple of times and the whole picture will start to make sense. The important thing I want to stress is that the code in TemplateResponseMixin is available for any class that wants to have the feature of rendering a template, for example DetailView (CODE), which receives the feature of showing the details of a single object by SingleObjectTemplateResponseMixin, which inherits from TemplateResponseMixin, overriding its method get_template_names (CODE).

As we discussed before, mixins cannot be too generic, and here we see a good example of a mixin designed to work on specific classes. TemplateResponseMixin has to be applied to classes that contain self.request (CODE), and while this doesn't mean exclusively classes derived from View, it is clear that it has been designed to augment that specific type.

Takeaway points
  • Inheritance is designed to promote code reuse but can lead to the opposite result
  • Multiple inheritance allows us to keep the inheritance tree simple
  • Multiple inheritance leads to possible problems that are solved in Python through the MRO
  • Interfaces (either implicit or explicit) should be part of your design
  • Mixin classes are used to add simple changes to classes
  • Mixins are implemented in Python using multiple inheritance: they have great expressive power but require careful design.
Final words

I hope this post helped you to understand a bit more how multiple inheritance works, and to be less scared by it. I also hope I managed to show you that classes have to be carefully designed and that there is a lot to consider when you create a class system. Once again, please don't forget composition, it's a powerful and too often forgotten tool.

Feedback

Feel free to reach me on Twitter if you have questions. The GitHub issues page is the best place to submit corrections.

Categories: FLOSS Project Planets

Qt 5.12.8 Released

Planet KDE - Wed, 2020-04-08 06:34

I am happy to inform you we have released Ot 5.12.8 today.

Categories: FLOSS Project Planets

Mediacurrent: Open Waters Ep 12: Project Managing an Open Source Project

Planet Drupal - Wed, 2020-04-08 06:11

In this episode, we welcome a new host, Christine Bryan. And as a sort of initiation, we interview her about project managing an open source project.

About Christine

Christine has been working in marketing and technology for over fifteen years. She’s worked in government, non-profit, and for-profit industries. Her first Drupal project was on Mass.gov, the state website for Massachusetts, and since then she’s worked with clients including universities, manufacturing, and financial services. In her spare time she parents a pretty awesome one year old kiddo, plays video games, and likes to walk around the world at Epcot.

 


Audio Download Link


Tool Pick

Let’s talk about Jira.

Interview
  1. Tell us about yourself and your role as a project manager (PM) at Mediacurrent.
  2. How did you get started in Project Management?
  3. What are some qualities you need to be an effective PMs?
  4. Are there specific challenges an Open Source project presents from a PM point of view?
  5. When we asked you what you wanted to talk about on your first time out on the podcast, why did you choose Jira?
  6. As a PM, do you interact with Marketers of the project you are managing?  How do marketers get involved with your projects?
  7. What could potentially make a project fail if not managed properly? And how would you avoid failure?
  8. A project, although managed by a PM, depends on the execution of many people (devs, stakeholders, etc.), what do you think others can do to increase the chances of success?
  9. How do you define success when managing a project?
Categories: FLOSS Project Planets

Andy Wingo: multi-value webassembly in firefox: a binary interface

GNU Planet! - Wed, 2020-04-08 05:02

Hey hey hey! Hope everyone is staying safe at home in these weird times. Today I have a final dispatch on the implementation of the multi-value feature for WebAssembly in Firefox. Last week I wrote about multi-value in blocks; this week I cover function calls.

on the boundaries between things

In my article on Firefox's baseline compiler, I mentioned that all WebAssembly engines in web browsers treat the function as the unit of compilation. This facilitates streaming, parallel compilation of WebAssembly modules, by farming out compilation of individual functions to worker threads. It also allows for easy tier-up from quick-and-dirty code generated by the low-latency baseline compiler to the faster code produced by the optimizing compiler.

There are some interesting Conway's Law implications of this choice. One is that division of compilation tasks becomes an opportunity for division of human labor; there is a whole team working on the experimental Cranelift compiler that could replace the optimizing tier, and in my hackings on Firefox I have had minimal interaction with them. To my detriment, of course; they are fine people doing interesting things. But the code boundary means that we don't need to communicate as we work on different parts of the same system.

Boundaries are where places touch, and sometimes for fluid crossing we have to consider boundaries as places in their own right. Functions compiled with the baseline compiler, with Ion (the production optimizing compiler), and with Cranelift (the experimental optimizing compiler) are all able to call each other because they actively maintain a common boundary, a binary interface (ABI). (Incidentally the A originally stands for "application", essentially reflecting division of labor between groups of people making different components of a software system; Conway's Law again.) Let's look closer at this boundary-place, with an eye to how it changes with multi-value.

what's in an ABI?

Among other things, an ABI specifies a calling convention: which arguments go in registers, which on the stack, how the stack values are represented, how results are returned to the callers, which registers are preserved over calls, and so on. Intra-WebAssembly calls are a closed world, so we can design a custom ABI if we like; that's what V8 does. Sometimes WebAssembly may call functions from the run-time, though, and so it may be useful to be closer to the C++ ABI on that platform (the "native" ABI); that's what Firefox does. (Incidentally here I think Firefox is probably leaving a bit of performance on the table on Windows by using the inefficient native ABI that only allows four register parameters. I haven't measured though so perhaps it doesn't matter.) Using something closer to the native ABI makes debugging easier as well, as native debugger tools can apply more easily.

One thing that most native ABIs have in common is that they are really only optimized for a single result. This reflects their heritage as artifacts from a world built with C and C++ compilers, where there isn't a concept of a function with more than one result. If multiple results are required, they are represented instead as arguments, typically as pointers to memory somewhere. Consider the AMD64 SysV ABI, used on Unix-derived systems, which carefully specifies how to pass arbitrary numbers of arbitrary-sized data structures to a function (§3.2.3), while only specifying what to do for a single return value. If the return value is too big for registers, the ABI specifies that a pointer to result memory be passed as an argument instead.

So in a multi-result WebAssembly world, what are we to do? How should a function return multiple results to its caller? Let's assume that there are some finite number of general-purpose and floating-point registers devoted to return values, and that if the return values will fit into those registers, then that's where they go. The problem is then to determine which results will go there, and if there are remaining results that don't fit, then we have to put them in memory. The ABI should indicate how to address that memory.

When looking into a design, I considered three possibilities.

first thought: stack results precede stack arguments

When a function needs some of its arguments passed on the stack, it doesn't receive a pointer to those arguments; rather, the arguments are placed at a well-known offset to the stack pointer.

We could do the same thing with stack results, either reserving space deeper on the stack than stack arguments, or closer to the stack pointer. With the advent of tail calls, it would make more sense to place them deeper on the stack. Like this:

The diagram above shows the ordering of stack arguments as implemented by Firefox's WebAssembly compilers: later arguments are deeper (farther from the stack pointer). It's an arbitrary choice that happens to match up with what the native ABIs do, as it was easier to re-use bits of the already-existing optimizing compiler that way. (Native ABIs use this stack argument ordering because of sloppiness in a version of C from before I was born. If you were starting over from scratch, probably you wouldn't do things this way.)

Stack result order does matter to the baseline compiler, though. It's easier if the stack results are placed in the same order in which they would be pushed on the virtual stack, so that when the function completes, the results can just be memmove'd down into place (if needed). The same concern dictates another aspect of our ABI: unlike calls, registers are allocated to the last results rather than the first results. This is to make it easy to preserve stack invariant (1) from the previous article.

At first I thought this was the obvious option, but I ran into problems. It turns out that stack arguments are fundamentally unlike stack results in some important ways.

While a stack argument is logically consumed by a call, a stack result starts life with a call. As such, if you reserve space for stack results just by decrementing the stack pointer before a call, probably you will need to load the results eagerly into registers thereafter or shuffle them into other positions to be able to free the allocated stack space.

Eager shuffling is busy-work that should be avoided if possible. It's hard to avoid in the baseline compiler. For example, a call to a function with 10 arguments will consume 10 values from the temporary stack; any results will be pushed on after removing argument values from the stack. If there any stack results, it's almost impossible to avoid a post-call memmove, to move stack results to where they should be before the 10 argument values were pushed on (and probably spilled). So the baseline compiler case is not optimal.

However, things get gnarlier with the Ion optimizing compiler. Like many other optimizing compilers, Ion is designed to compute the necessary stack frame size ahead of time, and to never move the stack pointer during an activation. The only exception is for pushing on any needed stack arguments for nested calls (which are popped directly after the nested call). So in that case, assuming there are a number of multi-value calls in a stack frame, we'll be shuffling in the optimizing compiler as well. Not great.

Besides the need to shuffle, stack arguments and stack results differ as regards ownership and garbage collection. A callee "owns" the memory for its stack arguments; it is responsible for them. The caller can't assume anything about the contents of that memory after a call, especially if the WebAssembly implementation supports tail calls (a whole 'nother blog post, that). If the values being passed are just bits, that's one thing, but with the reference types proposal, some result values may be managed by the garbage collector. The callee is responsible for making stack arguments visible to the garbage collector; the caller is responsible for the results. The caller will need to emit metadata to allow the garbage collector to see stack result references. For this reason, a stack result actually starts life just before a call, because it can become initialized at any point and thus needs to be traced during the entire callee activation. Not all callers can easily add garbage collection roots for writable stack slots, so the need to place stack results in a fixed position complicates calling multi-value WebAssembly functions in some cases (e.g. from C++).

second thought: pointers to individual stack results

Surely there are more well-trodden solutions to the multiple-result problem. If we encoded a multi-value return in C, how would we do it? Consider a function in C that has three 64-bit integer results. The idiomatic way to encode it would be to have one of the results be the return value of the function, and the two others to be passed "by reference":

int64_t foo(int64_t* a, int64_t* b) { *a = 1; *b = 2; return 3; } void call_foo(void) { int64 a, b, c; c = foo(&a, &b); }

This program shows us a possibility for encoding WebAssembly's multiple return values: pass an additional argument for each stack result, pointing to the location to which to write the stack result. Like this:

The result pointers are normal arguments, subject to normal argument allocation. In the above example, given that there are already stack arguments, they will probably be passed on the stack, but in many cases the stack result pointers may be passed in registers.

The result locations themselves don't even need to be on the stack, though they certainly will be in intra-WebAssembly calls. However the ability to write to any memory is a useful form of flexibility when e.g. calling into WebAssembly from C++.

The advantage of this approach is that we eliminate post-call shuffles, at least in optimizing compilers. But, having to make an argument for each stack result, each of which might itself become a stack argument, seems a bit offensive. I thought we might be able to do a little better.

third thought: stack result area, passed as pointer

Given that stack results are going to be written to memory, it doesn't really matter where they will be written, from the perspective of the optimizing compiler at least. What if we allocated them all in a block and just passed one pointer to the block? Like this:

Here there's just one additional argument, no matter how many stack results. While we're at it, we can specify that the layout of the stack arguments should be the same as how they would be written to the baseline stack, to make the baseline compiler's job easier.

As I started implementation with the baseline compiler, I chose this third approach, essentially because I was already allocating space for the results in a block in this way by bumping the stack pointer.

When I got to the optimizing compiler, however, it was quite difficult to convince Ion to allocate an area on the stack of the right shape.

Looking back on it now, I am not sure that I made the right choice. The thing is, the IonMonkey compiler started life as an optimizing compiler for JavaScript. It can represent unboxed values, which is how it came to be used as a compiler for asm.js and later WebAssembly, and it does a good job on them. However it has never had to represent aggregate data structures like a C++ class, so it didn't have support for spilling arbitrary-sized data to the stack. It took a while staring at the register allocator to convince it to allocate arbitrary-sized stack regions, and then to allocate component scalar values out of those regions. If I had just asked the register allocator to give me one appropriate-sized stack slot for each scalar, and hacked out the ability to pass separate pointers to the stack slots to WebAssembly calls with stack results, then I would have had an easier time of it, and perhaps stack slot allocation could be more dense because multiple results wouldn't need to be allocated contiguously.

As it is, I did manage to hack it in, and I think in a way that doesn't regress. I added a layer over an argument type vector that adds a synthetic stack results pointer argument, if the function returns stack results; iterating over this type with ABIArgIter will allocate a stack result area pointer, either as a register argument or a stack argument. In the optimizing compiler, I added add a kind of value allocation coresponding to a variable-sized stack area, (using pointer tagging again!), and extended the register allocator to allocate LStackArea, and the component stack results. Interestingly, I had to add a kind of definition that starts life on the stack; previously all Ion results started life in registers and were only spilled if needed.

In the end, a function will capture the incoming stack result area argument, either as a normal SSA value (for Ion) or stored to a stack slot (baseline), and when returning will write stack results to that pointer as appropriate. Passing in a pointer as an argument did make it relatively easy to implement calls from WebAssembly to and from C++, getting the variable-shape result area to be known to the garbage collector for C++-to-WebAssembly calls was simple in the end but took me a while to figure out.

Finally I was a bit exhausted from multi-value work and ready to walk away from the "JS API", the bit that allows multi-value WebAssembly functions to be called from JavaScript (they return an array) or for a JavaScript function to return multiple values to WebAssembly (via an iterable) -- but then when I got to thinking about this blog post I preferred to implement the feature rather than document its lack. Avoidance-of-document-driven development: it's a thing!

towards deployment

As I said in the last article, the multi-value feature is about improved code generation and also making a more capable base for expressing further developments in the WebAssembly language.

As far as code generation goes, things are progressing but it is still early days. Thomas Lively has implemented support in LLVM for emitting return of C++ aggregates via multiple results, which is enabled via the -experimental-multivalue-abi cc1 flag. Thomas has also been implementing multi-value support in the binaryen WebAssembly toolchain component, used by the emscripten C++-to-WebAssembly toolchain. I think it will be a few months though before everything lands in a way that end users can take advantage of.

On the specification side, the multi-value feature is now at phase 4 since January, which basically means things are all done there.

Implementation-wise, V8 has had experimental support since 2017 or so, and the feature was staged last fall, although V8 doesn't yet support multi-value in their baseline compiler. WebKit also landed support last fall.

Unlike V8 and SpiderMonkey, JavaScriptCore (the JS and wasm engine in WebKit) actually implements a WebAssembly interpreter as their solution to the one-pass streaming compilation problem. Then on the compiler side, there are two tiers that both operate on basic block graphs (OMG and BBQ; I just puked a little in my mouth typing that). This strategy makes the compiler implementation quite straightforward. It's also an interesting design point because JavaScriptCore's garbage collector scans the stack conservatively; there's no need for the compiler to do bookkeeping on the GC's behalf, which I'm sure was a relief to the hacker. Anyway, multi-value in WebKit is done too.

The new thing of course is that finally, in Firefox, the feature is now fully implemented (woo) and enabled by default on Nightly builds (woo!). I did that! It took me a while! Perhaps too long? Anyway it's done. Thanks again to Bloomberg for supporting this work; large ups to y'all for helping the web move forward.

See you next time with a more general article rounding up compile-time benchmarks on a variety of WebAssembly implementations. Until then, happy hacking!

Categories: FLOSS Project Planets

Programiz: Python Docstrings

Planet Python - Wed, 2020-04-08 03:52
In this tutorial, we will learn about Python docstrings. More specifically, we will learn how and why docstrings are used with the help of examples.
Categories: FLOSS Project Planets

Pages