Planet Python

While list comprehesions in Python don't support the else keyword directly, conditional expressions can be embedded within list comprehension.

Table of contents

1. Do list comprehensions support else?
2. But else does work in comprehensions...
3. Python's ternary operator
4. Conditional expressions in list comprehension
5. Making more readable list comprehensions
6. Avoid complex comprehensions
7. Python's list comprehensions and if-else

Do list comprehensions support else?

This list comprehension isn't valid:

>>> counts = [2, -1, 4, 7, -3, 6] >>> sanitized_counts = [n for n in counts if n > 0 else 0] File "<stdin>", line 1 sanitized_counts = [n for n in counts if n > 0 else 0] ^^^^ SyntaxError: invalid syntax

Python's list comprehensions don't have any support for an else keyword.

But else does work in comprehensions...

But wait! I've seen else …

Read the full article: https://www.pythonmorsels.com/comprehension-with-else/

In my last post I've described how to use Z3 to find simple local peephole optimization patterns for the integer operations in PyPy's JIT. An example is int_and(x, 0) -> 0. In this post I want to scale up the problem of identifying possible optimizations to much bigger instruction sequences, also using Z3. For that, I am starting with the JIT traces of real benchmarks, after they have been optimized by the optimizer of PyPy's JIT. Then we can ask Z3 to find inefficient integer operations in those traces.

Starting from the optimized traces of real programs has some big advantages over the "classical" superoptimization approach of generating and then trying all possible sequences of instructions. It avoids the combinatorical explosion that happens with the latter approach. Also, starting from the traces of benchmarks or (even better) actual programs makes sure that we actually care about the missing optimizations that are found in this way. And because the traces are analyzed after they have been optimized by PyPy's optimizer, we only get reports for missing optimizations, that the JIT isn't able to do (yet).

The techniques and experiments I describe in this post are again the result of a bunch of discussions with John Regehr at a conference a few weeks ago, as well as reading his blog posts and papers. Thanks John! Also thanks to Max Bernstein for super helpful feedback on the drafts of this blog post (and for poking me to write things in general).

High-Level Approach

The approach that I took works as follows:

• Run benchmarks or other interesting programs and then dump the IR of the JIT traces into a file. The traces have at that point been already optimized by the PyPy JIT's optimizer.
• For every trace, ignore all the operations on non-integer variables.
• Translate every integer operation into a Z3 formula.
• For every operation, use Z3 to find out whether the operation is redundant (how that is done is described below).
• If the operation is redundant, the trace is less efficient than it could have been, because the optimizer could also have removed the operation. Report the inefficiency.
• Minimize the inefficient programs by removing as many operations as possible to make the problem easier to understand.

In the post I will describe the details and show some pseudocode of the approach. I'll also make the proper code public eventually (but it needs a healthy dose of cleanups first).

Dumping PyPy Traces

PyPy will write its JIT traces into the file out if the environment variable PYPYLOG is set as follows:

PYPYLOG=jit-log-opt:out pypy <program.py>

This environment variable works for PyPy, but also for other virtual machines built with RPython.

(This is really a side point for the rest of the blog post, but since the question came up I wanted to clarify it: Operations on integers in the Python program that the JIT is running don't all correspond 1-to-1 with the int_... operations in the traces. The int_... trace operations always operate on machine words. The Python int type supports arbitrarily large integers. PyPy will optimistically try to lower the operations on Python integers into machine word operations, but adds the necessary guards into the trace to make sure that overflow outside of the range of machine words is caught. In case one of these guards fails the interpreter switches to a big integer heap-allocated representation.)

Encoding Traces as Z3 formulas

The last blog post already contained the code to encode the results of individual trace operations into Z3 formulas, so we don't need to repeat that here. To encode traces of operations we introduce a Z3 variable for every operation in the trace and then call the z3_expression function for every single one of the operations in the trace.

For example, for the following trace:

[i1] i2 = uint_rshift(i1, 32) i3 = int_and(i2, 65535) i4 = uint_rshift(i1, 48) i5 = int_lshift(i4, 16) i6 = int_or(i5, i3) jump(i6, i2) # equal

We would get the Z3 formula:

z3.And(i2 == LShR(i1, 32), i3 == i2 & 65535, i4 == LShR(i1, 48), i5 == i4 << 16)

Usually we won't ask for the formula of the whole trace at once. Instead we go through the trace operation by operation and try to find inefficiencies in the current one we are looking at. Roughly like this (pseudo-)code:

def newvar(name): return z3.BitVec(name, INTEGER_WIDTH) def find_inefficiencies(trace): solver = z3.Solver() var_to_z3var = {} for input_argument in trace.inputargs: var_to_z3var[input_argument] = newz3var(input_argument) for op in trace: var_to_z3var[op] = z3resultvar = newz3var(op.resultvarname) arg0 = op.args[0] z3arg0 = var_to_z3var[arg0] if len(op.args) == 2: arg1 = op.args[1] z3arg1 = var_to_z3var[arg1] else: z3arg1 = None res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # checking for inefficiencies, see the next sections ... if ...: return "inefficient", op # not inefficient, assert op into the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Identifying constant booleans with Z3

To get started finding inefficiencies in a trace, we can first focus on boolean variables. For every operation in the trace that returns a bool we can ask Z3 to prove that this variable must be always True or always False. Most of the time, neither of these proofs will succeed. But if Z3 manages to prove one of them, we know have found an ineffiency: instead of computing the boolean result (eg by executing a comparison) the JIT's optimizer could have replaced the operation with the corresponding boolean constant.

Here's an example of an inefficiency found that way: if x < y and y < z are both true, PyPy's JIT could conclude that x < z must also be true. However, currently the JIT cannot make that conclusion because it only reasons about the concrete ranges (lower and upper bounds) for every integer variable, but it has no way to remember anything about relationships between different variables. This kind of reasoning would quite often be useful to remove list/string bounds checks. Here's a talk about how LLVM does this (but it might be too heavyweight for a JIT setting).

Here are some more examples found that way:

• x - 1 == x is always False
• x - (x == -1) == -1 is always False. The pattern x - (x == -1) happens a lot in PyPy's hash computations: To be compatible with the CPython hashes we need to make sure that no object's hash is -1 (CPython uses -1 as an error value on the C level).

Here's pseudo-code for how to implement checking boolean operations for inefficiencies:

def find_inefficiencies(trace): ... for op in trace: ... res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # check for boolean constant result if op.has_boolean_result(): if prove(solver, res == 0): return "inefficient", op, 0 if prove(solver, res == 1): return "inefficient", op, 1 # checking for other inefficiencies, see the next sections ... # not inefficient, add op to the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Identifying redundant operations

A more interesting class of redundancy is to try to find two operations in a trace that compute the same result. We can do that by asking Z3 to prove for each pair of different operations in the trace to prove that the result is always the same. If a previous operation returns the same result, the JIT could have re-used that result instead of re-computing it, saving time. Doing this search for equivalent operations with Z3 is quadratic in the number of operations, but since traces have a maximum length it is not too bad in practice.

This is the real workhorse of my script so far, it's what finds most of the inefficiencies. Here's a few examples:

• The very first and super useful example the script found is int_eq(b, 1) == b if b is known to be a boolean (ie and integer 0 or 1). I have already implemented this optimization in the JIT.
• Similarly, int_and(b, 1) == b for booleans.
• (x << 4) & -0xf == x << 4
• ((x >> 63) << 1) << 2) >> 3 == x >> 63. In general the JIT is quite bad at optimizing repeated shifts (the infrastructure for doing better with that is already in place, so this will be a relatively easy fix).
• (x & 0xffffffff) | ((x >> 32) << 32) == x. Having the JIT optimize this would maybe require first recognizing that (x >> 32) << 32 can be expressed as a mask: (x & 0xffffffff00000000), and then using (x & c1) | (x & c1) == x & (c1 | c2)
• A commonly occurring pattern is variations of this one: ((x & 1345) ^ 2048) - 2048 == x & 1345 (with different constants, of course). xor is add without carry, and x & 1345 does not have the bit 2048 set. Therefore the ^ 2048 is equivalent to + 2048, which the - 2048 cancels. More generally, if a & b == 0, then a + b == a | b == a ^ b. I don't understand at all why this appears so often in the traces, but I see variations of it a lot. LLVM can optimize this, but GCC can't, thanks to Andrew Pinski for filing the bug!

And here's some implementation pseudo-code again:

def find_inefficiencies(trace): ... for op in trace: ... res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # check for boolean constant result ... # searching for redundant operations for previous_op in trace: if previous_op is op: break # done, reached the current op previous_op_z3var = var_to_z3var[previous_op] if prove(solver, previous_op_z3var == res): return "inefficient", op, previous_op ... # more code here later ... # not inefficient, add op to the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Synthesizing more complicated constants with exists-forall

To find out whether some integer operations always return a constant result, we can't simply use the same trick as for those operations that return boolean results, because enumerating 2⁶⁴ possible constants and checking them all would take too long. Like in the last post, we can use z3.ForAll to find out whether Z3 can synthesize a constant for the result of an operation for us. If such a constant exists, the JIT could have removed the operation, and replaced it with the constant that Z3 provides.

Here a few examples of inefficiencies found this way:

• (x ^ 1) ^ x == 1 (or, more generally: (x ^ y) ^ x == y)
• if x | y == 0, it follows that x == 0 and y == 0
• if x != MAXINT, then x + 1 > x

Implementing this is actually slightly annoying. The solver.add calls for non-inefficient ops add assertions to the solver, which are now confusing the z3.ForAll query. We could remove all assertion from the solver, then do the ForAll query, then add the assertions back. What I ended doing instead was instantiating a second solver object that I'm using for the ForAll queries, that remains empty the whole time.

def find_inefficiencies(trace): solver = z3.Solver() empty_solver = z3.Solver() var_to_z3var = {} ... for op in trace: ... res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # check for boolean constant result ... # searching for redundant operations ... # checking for constant results constvar = z3.BitVec('find_const', INTEGER_WIDTH) condition = z3.ForAll( var_to_z3var.values(), z3.Implies( *solver.assertions(), expr == constvar ) ) if empty_solver.check(condition) == z3.sat: model = empty_solver.model() const = model[constvar].as_signed_long() return "inefficient", op, const # not inefficient, add op to the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Minimization

Analyzing an inefficiency by hand in the context of a larger trace is quite tedious. Therefore I've implemented a (super inefficient) script to try to make the examples smaller. Here's how that works:

• First throw out all the operations that occur after the inefficient operation in the trace.
• Then we remove all "dead" operations, ie operations that don't have their results used (all the operations that we can analyze with Z3 are without side effects).
• Now we try to remove every guard in the trace one by one and check afterwards, whether the resulting trace still has an inefficiency.
• We also try to replace every single operation with a new argument to the trace, to see whether the inefficiency is still present.

The minimization process is sort of inefficient and I should probably be using shrinkray or C-Reduce instead. However, it seems to work well in practice and the runtime isn't too bad.

Results

So far I am using the JIT traces of three programs: 1) Booting Linux on the Pydrofoil RISC-V emulator, 2) booting Linux on the Pydrofoil ARM emulator, and 3) running the PyPy bootstrap process on top of PyPy.

I picked these programs because most Python programs don't contain interesting amounts of integer operations, and the traces of the emulators contain a lot of them. I also used the bootstrap process because I still wanted to try a big Python program and personally care about the runtime of this program a lot.

The script identifies 94 inefficiencies in the traces, a lot of them come from repeating patterns. My next steps will be to manually inspect them all, categorize them, and implement easy optimizations identified that way. I also want a way to sort the examples by execution count in the benchmarks, to get a feeling for which of them are most important.

I didn't investigate the full set of Python benchmarks that PyPy uses yet, because I don't expect them to contain interesting amounts of integer operations, but maybe I am wrong about that? Will have to try eventually.

Conclusion

This was again much easier to do than I would have expected! Given that I had the translation of trace ops to Z3 already in place, it was a matter of about a day's of programming to use this infrastructure to find the first problems and minimizing them.

Reusing the results of existing operations or replacing operations by constants can be seen as "zero-instruction superoptimization". I'll probably be rather busy for a while to add the missing optimizations identified by my simple script. But later extensions to actually synthesize one or several operations in the attempt to optimize the traces more and find more opportunities should be possible.

Finding inefficiencies in traces with Z3 is significantly less annoying and also less error-prone than just manually inspecting traces and trying to spot optimization opportunities.

Random Notes and Sources

Again, John's blog posts:

• Let’s Work on an LLVM Superoptimizer
• Early Superoptimizer Results
• A Few Synthesizing Superoptimizer Results
• Synthesizing Constants

and papers:

• A Synthesizing Superoptimizer
• Hydra: Generalizing Peephole Optimizations with Program Synthesis

I remembered recently that I had seen the approach of optimizing the traces of a tracing JIT with Z3 a long time ago, as part of the (now long dead, I think) SPUR project. There's a workshop paper from 2010 about this. SPUR was trying to use Z3 built into the actual JIT (as opposed to using Z3 only to find places where the regular optimizers could be improved). In addition to bitvectors, SPUR also used the Z3 support for arrays to model the C# heap and remove redundant stores. This is still another future extension for all the Z3 work I've been doing in the context of the PyPy JIT.

What are discrete optimization problems? How do you solve them with constraint programming in Python? Christopher Trudeau is back on the show this week, bringing another batch of PyCoder's Weekly articles and projects

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

This quiz will evaluate your understanding of Python’s string data type and test your knowledge about manipulating textual data with string objects. You’ll cover the basics of creating strings using literals and the str() function, applying string methods, using operators and built-in functions with strings, indexing and slicing strings, and more!

Take this quiz after reading our Strings and Character Data in Python tutorial.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

So, you’re a Python developer or you just use Python to make your life easier by writing utility scripts to automate repetitive tasks and boost your productivity.
Such Python scripts are usually local to your machine, run from the command line and require some Python skills to use including for instance setting up a virtual environment and knowing the basics of using the CLI.
You may have wondered what it involves to convert a script like this into a web app to make it available for other users such as your family members and co-workers.

In this article, I’ll walk through the general process I followed to “convert” a Python utility script into a full-blown web app by building a backend API using Django and Django Rest Framework and a single-page-app frontend using React.js.

But first, here is a bit of a background…

My family often needed me to merge (combine) PDF files for them. When they apply for online services they’re often required to upload all documents as a single PDF file.

As they only use Windows, they couldn’t find a reliable tool of merging PDFs. The apps they found worked only sometimes, they also didn’t want to upload sensitive docs to random PDF merging websites.
Being the family tech guy, they would ask me for help. They would send me the files to merge and I’d send them back a single PDF file; merged and ready to upload.

I used pdftk for a while. It’s available only on Linux (as far as I know) and the command looks like this:

pdftk 1.pdf 2.pdf 3.pdf cat output out.pdf

I then upgraded to a tiny Python script using PyPDF. Something like this:

from pathlib import Path from pypdf import PdfWriter merger = PdfWriter() path = Path.home() / "Documents" / "uni-application" with merger as PdfWriter: for file in path.glob('*.pdf'): merger.append(file) merger.write("merged.pdf")

I had to set the path manually every time and rename the PDFs so that they are merged in the desired order.

At some point, I felt I was merging PDFs for the family too often that I needed a way to expose this script to them somehow. I thought I should just build a web interface for it, host it on my local home server and let them merge as many files as they want without needing my help.

There are many ways to package and distribute a Python script including making a stand-alone executable. However, unless you build an intuitive GUI for it, average users may not be able to benefit from it.
From all GUI options out there, I chose to make it a web app mainly because web apps have a common look and an expected functionality, users are used to interacting with them almost everyday and they run (hopefully) everywhere.

So, as a side little project over months, I built a basic REST-ish API for the backend that allows you to create a merge request, add PDF files to it, merge the files and get back a single PDF file to download and use.
I also built a simple frontend for it using React.js.

For deployment, I used docker to run the app on my home server and expose it on the LAN with a custom “.home” local domain. Now, my family use it almost everyday to merge PDFs on their own and everyone is happy.

I’ll go over the general process I followed to build this small web app that serves only one purpose: merging PDF files.
You can swap PDF merging for any functionality like: converting audio/video files, resizing images, extracting text from PDFs or images, etc. The process is pretty much the same.

This is not a step-by-step guide. I’ll just explain the overall process. Tutorials on any technology mentioned here can be easily found online. Also, I won’t clutter the article with code examples as you can refer to the source code in the Github repo.

Backend API and Auth

I chose to go with Django and Django Rest Framework for building the backend as I’m more familiar with these two than other Python options like Flask and FastAPI. In addition, being a battery-included framework, Django comes with a lot of nice features I needed for this specific app, the most important of which are authentication and ORM.

Why the API? Why not just full-stack Django?

I could have just built a typical Django app that works as the backend and frontend at the same time. But I knew for sure I’m going to build a REST API alongside it at some point down the road, so why not just start by an API?
Moreover, the API-based backend is frontend-agnostic. It doesn’t assume a specific client-side technology. I can swap React for Vue for example. I can also build more clients to consume the API like a mobile or desktop app.

Building the API:

I started by thinking what the original Python script does and what the user usually wants to get back using it. In this case the user wants a way to:

• upload at least 2 PDF files,
• merge the PDF files on the server,
• and download the single merged PDF file.

Models

To satisfy these requirements, I started a new Django project with a “merger” app, installed Django Rest Framework, added the models to save everything in the DB. Meaning, when a user creates a merge request, it’s saved to the DB to track various things about it like how many files it has and whether it’s been merged or not. Later, we can utilize this data and build upon it if we decide for example to put each user on a plan/package with a limited number of merges/files per merge etc.

I used the name “order” for the merge request effectively treating it as a transaction. My idea is to make things easier later if the app turns into a paid service.

Serializers

Then I created DRF serializers which are responsible for converting data from Python to JSON format and back (APIs expect and return JSON, not Python data types like dicts).

They say: don’t trust user uploads. So, in serializers also, I validate the uploaded PDF file to make sure they’re indeed a PDF file by verifying that the content type is ‘pdf’ and checking MIME type by reading the first 2084 bits of the file.
This is not a bullet-proof validation system. It may keep away casual malicious users but it won’t probably protect against determined attackers. Nevertheless, it prevents common user errors like uploading the wrong file format.

The validation also checks the file size to make sure it doesn’t exceed the limit set in settings.py, otherwise users can intentionally or naively upload huge files and block server response while uploading.
If the file fails to validate, the API will return a message containing the reason for rejection.

Although you can leave some of this validation for the frontend, you should always expect that some advanced users may go around frontend validation by trying to send requests directly to the API or even build their own frontend. After all, the API is open and can be consumed from any client. This off course doesn’t mean that they can bypass auth or permissions but by having validation in the backend, if they use a custom-built client, they can’t bypass the backend validation.

Views & Endpoints

I then switched to building the views which are essentially the endpoints the frontend will call to interact with the backend.
Based on the requirements of this specific app, following are the API endpoints with a short description of what each does:

GET: /orders/

when a GET request is sent to this endpoint, it returns back a list all orders that belong to the currently authenticated user.

POST: /orders/

A POST request with a payload containing the merge name like: {“name”: “my merge”} creates a new order (merge request).

GET: /orders/{uuid}/

returns details of an order identified by its id

DELETE /orders/{uuid}/

deletes an order

POST /orders/{uuid}/add_files

A POST request with a payload containing a file add files to an order after validating file type and max files limit. It also checks to see if the order has already been merged in which case no files can be added.

GET /orders/{uuid}/files/

lists files of an order

GET /orders/{uuid}/merge/

merges order identified by its id.
This is where the core feature of the app lives. This view/endpoint does some initial checks to verify that the order has at least 2 files to merge and has not been previously merged then it hands work over to a utility function that preforms the actual merging of all PDF files associated with the order:

import os import uuid from pypdf import PdfWriter from pypdf.errors import PdfReadError, PyPdfError from django.conf import settings from .exceptions import MergeException def merge_pdf_files(pdf_files): merger = PdfWriter() for f in pdf_files: try: merger.append(f.file) except FileNotFoundError: raise MergeException merged_path = os.path.join(settings.MEDIA_ROOT, f"merged_pdfs/merged_pdf_{uuid.uuid4()}.pdf") try: merger.write(merged_path) except FileNotFoundError: raise MergeException except PdfReadError: raise MergeException except PyPdfError: raise MergeException merger.close() if os.path.isfile(merged_path): return merged_path return None

Nothing fancy here, the function takes a list of PDF files, merges them using PyPDF and returns the path for the merged PDF.

On a successful merge, the view takes the path and sets it as the value for download_url property of the order instance for the next endpoint to use. It also marks the order and all of its files as merged. This can be used to cleanup all merged orders and their associated files to save server space.

GET /orders/{uuid}/download/

download the merged PDF of an order after verifying that it has been merged and ready for download. The API allows a max number of downloads of each order and max time on server. This prevents users from keeping merged files on the server forever and sharing the links, turning the app basically into a free file sharing service.

DELETE /files/{uuid}/

delete a file identified by its id

I then wired the urls to the views connecting each endpoint to the corresponding view.

Auth and Permissions:

To insure privacy (not necessarily security), and to allow the app to be used as a paid service later, I decided to require an account for every user to use the app. Thanks to Django’s built-in features, this can be done fairly easily with the auth module. However, I didn’t want to use the traditional session-based authentication flow. I just prefer the token-based auth as it’s more suitable for API-based apps.

I chose to use JWT (JSON Web Token). Basically it’s a way to allow the user to exchange their credentials for a set of tokens: an access token and a refresh token.
The access token will be attached in the header of every request (prefixed with “Bearer “) that requires authentication. Otherwise the request will fail. This token has shorter life span (usually hours or even minutes) and when it expires, the user can send the refresh token to get a new access token and continue using the app.

There are a number of packages that can add JWT auth flow to Django. Most of them use simple-jwt which you can use directly but it requires you to write more code yourself to implement the minimum register/login/out flow.
I went with Djoser which is a REST implementation of Django’s auth system.
It allowed me to use a custom user model to have extra fields in the user table and most importantly, utilize email/password for registration and login instead of Django’s default username/password although I had to tweak the models in the example project to work as intended.
Djoser also gave me the following endpoints for free:

• GET /auth/users/ : lists all users if you’re an admin. Only returns your user info if you aren’t admin.
• POST /auth/users/ : a POST request with a payload containing: name, email and password will register a new user.
• POST /auth/jwt/create: a POST request with a payload containing: email and password will return access and refresh tokens to use for authenticating subsequent requests.
• POST /auth/refresh : a POST request with a payload containing: the refresh token will return a new access token.
  as well as some other useful endpoints for changing and resetting passwords.

On top of Django’s auth, DRF uses permissions to decide what API resources a user can access. In short, in the API, I check if the user is logged in first. Then I have only two permissions:

• check if the user is the owner of the order before allowing them to access it, upload files to it, merge it or download it.
• check if the user is the owner of the order that a file is associated with before allowing them to view the file details or delete it.
  Failure to meet required permissions causes the API to raise a permission error and return a descriptive message.

During development of the backend, I used the browser extension “ModHeader” to include the access token in all requests I made through the browser (for testing via DRF built-in web UI).

Frontend

I chose React.js for building the frontend as a single-page-app (SPA) because it’s relatively easy to use for small apps and has a huge community. I won’t go into much details of how the frontend is built (this is a Python blog after all) but I will touch on the main points.

Auth in the frontend

First, here is a brief description of the auth workflow in the frontend:

• when a user first visits the app, they’re asked to login or signup to continue.
• the user can signup by filling out their name, email, password. This form data will be sent via an API “POST” request to the backend endpoint /auth/users and register the user.
• for existing users, email and password are sent in a “POST” request to /auth/jwt/create which will return a pair of tokens: refresh and access. These tokens are saved in browser cookies. The access token will be sent in the header of all subsequent request to authenticate the user. When it expires, the frontend will request a new token on behalf of the user by sending the refresh token to /auth/refresh. If both expire, the user will be redirected to login again to obtain a new set of tokens.
• when a user navigates to /logout, all tokens are cleared from browser cookies and the user is redirected to /login route.

Routes:

I built the necessary React component and routes for the app to roughly match the backend endpoints I discussed earlier. These are the routes the app has for authenticated users:

• / => home screen to choose to create a new merge or list previous merges.
• /orders => list all merges for the currently logged in user with links to edit or delete any order.
• /create => create new merge.
• /order/{id} =>details of a merge by id.
• /logout => logs the user out.

Merging workflow:

The workflow for merging a PDF file from the frontend is as follows:

• when the user creates a new merge, they’re redirected to the merge detail route showing the merge they’ve just created with three buttons: add PDFs, merge, download.
• merge detail route allows the user to upload PDF files to the merge. If the files are less than 2, only “Add PDF files” is enabled. When the user adds 2 files the “merge” button is activated. When the files reach the max number of files allowed in a single merge (5 currently) the “Add PDFs” button is disabled and the upload form is hidden.
• When the user is done adding PDFs, they can click “merge” which will merge the files on the server and activate only the download button.
• clicking the download button opens a new browser tab with the merged PDF.

UI & CSS:

For UI, the app uses daisyUI, a free component library that makes it easier to use TailwindCSS. The latter is super popular in the frontend world as a utility-first CSS framework.

Deployment:

I’ve not deployed the app to a real production server yet as the home server environment is very forgiving and you can skip some steps that you wouldn’t skip in a production deployment.
For now, I just have a basic Dockerfile and a docker-compose file to spin up the backend API (a regular Django project) and have it ready to accept calls from the frontend.

Likewise, a set of docker files is used to spin up the frontend. After building it using “npm run build”, the docker file copies the deployable app from the “dist” folder to the Nginx document root folder inside the docker container and just runs it as any other website hosted on a web server.
This setup is probably enough for development and hosting locally. When it comes time to publish the on the web, “real” deployment considerations must be taken.

It’s worth noting that I have a separate repo for backend and frontend to keep both ends decoupled from each other. The backend API can be consumed from any frontend be it a web, mobile or desktop app.

Further improvements:

The app in its current state works and does the intended job. It’s far from perfect and can use some improvements. I’ll include these in the README of the repos.

Source code:

• Backend: https://github.com/ahmedlemine/pdf-merger
• Frontend: https://github.com/ahmedlemine/pdfmerger-frontend

Conclusion

In this article I walked through the general process of how I built a web app for a Python script to make it available for use by average end users.
Python scripts are great starting points for apps. They’re also a source of inspiration for app ideas. If you have a script that performs a common daily-life task, consider building an app for it. The process will teach you a ton on the lifecycle of app development. It forces you to think of and account for new aspects you don’t usually consider when writing a stand-alone script.
As you build the app though, always remember to:

• keep it simple. Don’t over complicate things.
• ship fast. Aim at building an MVP (Minimum Viable Product) with the necessary functionality. Don’t wait until you’ve built every feature you think the app should have. Instead, ship it and then iterate on it and add features slowly as they’re needed.
• not to feel intimidated by other mature projects out there. They’ve been built most likely over a long period of time and have been iterated over tens or even thousands of times before they reached their mature state they are in today.

I hope you found this article helpful and I look forward to seeing you in a future one.