FLOSS Project Planets

When you switch from PaaS to IaaS, suddenly you have a series of new responsibilities. There is a lot to learn and also a lot to mimic from existing PaaS providers. We would like to share our experience after successfully migrating four larger Drupal sites from Pantheon to Azure cloud. The overview is in chronological order, detailing how we proceeded with the implementation. Infrastructure Basics We worked with an excellent infrastructure team who provisioned the following architecture for us:

Early promising results with SBOMs and Python packages About • Blog • Cool URLs Early promising results with SBOMs and Python packages

Published 2024-11-14 by Seth Larson
Reading time: minutes

This critical role would not be possible without funding from the Alpha-Omega project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem!

I've kicked off a project to reduce the "phantom dependency" problem for Python. The phantom dependency problem is where distinct software (sometimes written in Python, but often C, C++, Rust, etc) is included in a Python package but then isn't recorded anywhere in the package metadata.

These distinct pieces of software aren't not recorded because of lack of time or awareness, there is no standardized method to record this information in Python package metadata.

This means that when a software composition analysis (SCA) tool looks at the Python package the tool will "miss" all the software that's included in the package aside from the top-level package itself.

For example, the popular Python image manipulation library "Pillow" is not only "Pillow", the wheel files contain many more libraries to comply with the "manylinux" package platform:

# (the below libraries are bundled by auditwheel) $ unzip -l pillow-11.0.0-cp312-cp312-manylinux_2_28_x86_64.whl | grep 'pillow.libs' pillow.libs/ pillow.libs/libharfbuzz-144af51e.so.0 pillow.libs/libxcb-b8a56d01.so.1.1.0 pillow.libs/libpng16-4cc6a9fc.so.16.44.0 pillow.libs/libXau-154567c4.so.6.0.0 pillow.libs/libbrotlicommon-3ecfe81c.so.1 pillow.libs/liblzma-c9407571.so.5.6.3 pillow.libs/libfreetype-e7d5437d.so.6.20.1 pillow.libs/liblcms2-e69eef39.so.2.0.16 pillow.libs/libopenjp2-05423b53.so pillow.libs/libtiff-0a86184d.so.6.0.2 pillow.libs/libjpeg-45e70d75.so.62.4.0 pillow.libs/libbrotlidec-ba690955.so.1 pillow.libs/libwebp-2fd3cdca.so.7.1.9 pillow.libs/libsharpyuv-898c0cb5.so.0.1.0 pillow.libs/libwebpdemux-f2642bcc.so.2.0.15 pillow.libs/libwebpmux-d524b4d5.so.3.1.0

I see many recognizable projects in the list of shared objects, like libjpeg, libwebp, libpng, xz-utils (liblzma), etc. If we try to scan this installed wheel with a tool like Syft we receive this report:

$ syft dir:venv ✔ Indexed file system venv ✔ Cataloged packages [2 packages] NAME VERSION TYPE pillow 11.0.0 python pip 24.2 python

Syft isn't able to find any of the compiled libraries! So if we were to run a vulnerability scanner we would only receive vulnerability records for Pillow and pip. My plan is to help fix this problem with Software Bill-of-Materials documents (SBOMs) included in a standardized way inside of Python packages.

To test how well this proposal works with today's tools, I forked auditwheel and created a rudimentary patch which:

• For each shared library which is being bundled into a wheel, record the original file path and checksum. Bundle the shared libraries into the wheel as normal.
• Using platform-specific manager query each file path back to the package that provides the file. In this specific case rpm was used (rpm -qf <path>) because manylinux_2_28_x86_64 uses AlmaLinux 8 as the distribution.
• Gather information about that package using rpm, such as the name, version, etc.
• For each package, create the intrinsic "package URL" (PURL) software identifier for later use. This includes information about the packaging format, package name, version, but also the distro and architecture. For example, the PURL for the copy of libwebp used by the wheel is: pkg:rpm/almalinux/libwebp@1.0.0-9.el8_9.1?arch=x86_64&distro=almalinux-8
• Generate a CycloneDX SBOM file containing the above gathered information split into components and with relationship links between the top-level component (Pillow) and the bundled libraries.
• Embed that generated SBOM file into the wheel.

Let's run through building Pillow from source and using our forked auditwheel:

# The manylinux image may differ depending on your platform. $ docker run --rm -it -v.:/tmp/wheelhouse \ quay.io/pypa/manylinux_2_28_x86_64 # Install dependencies for Pillow $ yum install --nogpgcheck libtiff-devel \ libjpeg-devel openjpeg2-devel zlib-devel \ freetype-devel lcms2-devel libwebp-devel \ tcl-devel tk-devel harfbuzz-devel \ fribidi-devel libxcb-devel # Create a virtualenv and install auditwheel fork $ /usr/local/bin/python3.12 -m venv venv $ source venv/bin/activate $ python -m pip install build $ python -m pip install git+https://github.com/sethmlarson/auditwheel@sboms # Download the Pillow source from PyPI $ python -m pip download --no-binary=pillow pillow==11.0.0 $ tar -xzvf pillow-11.0.0.tar.gz # Build a non-manylinux wheel for Pillow $ python -m build ./pillow-11.0.0/ # Repair the wheel using auditwheel $ auditwheel repair ./pillow-11.0.0/dist/ ... Fixed-up wheel written to /wheelhouse/pillow-11.0.0-cp312-cp312-manylinux_2_28_x86_64.whl # Inspect the wheel for our SBOM, there it is! $ unzip -l /wheelhouse/pillow-11.0.0-*.whl | grep '.cdx.json' 5703 11-14-2024 19:39 pillow.libs/auditwheel.cdx.json # Move the wheel outside our container $ mv /wheelhouse/pillow-11.0.0-cp312-cp312-manylinux_2_28_x86_64.whl /tmp/wheelhouse/

So now we have a wheel file that contains an SBOM partially describing its contents. Let's try installing that wheel and running Syft:

$ syft dir:venv ✔ Indexed file system /tmp/venv-pillow ✔ Cataloged packages [13 packages] NAME VERSION TYPE Pillow 11.0.0 python bzip2-libs 1.0.6-26.el8 rpm freetype 2.9.1-9.el8 rpm jbigkit-libs 2.1-14.el8 rpm lcms2 2.9-2.el8 rpm libXau 1.0.9-3.el8 rpm libjpeg-turbo 1.5.3-12.el8 rpm libpng 1.6.34-5.el8 rpm libtiff 4.0.9-33.el8_10 rpm libwebp 1.0.0-9.el8_9.1 rpm libxcb 1.13.1-1.el8 rpm openjpeg2 2.4.0-5.el8 rpm pip 24.2 python

Woo hoo! Now the proper libraries are showing up in Syft. That means we'll be able to get vulnerability information from all the contained software components. This isn't the end, there are many many MANY ways that software ends up in a Python package. This quick validation test only shows that even with today's SBOM and SCA tools that embedding SBOM documents into wheels can be useful for downstream tools. Onwards to even more! 🚀

If you're interested in this project, follow the repository on GitHub and participate in the kick-off discussion on Python Discourse.

That's all for this post! 👋 If you're interested in more you can read the last report.

Have thoughts or questions? Let's chat over email or social:

sethmichaellarson@gmail.com
@sethmlarson@fosstodon.org

Want more articles like this one? Get notified of new posts by subscribing to the RSS feed or the email newsletter. I won't share your email or send spam, only whatever this is!

Want more content now? This blog's archive has ready-to-read articles. I also curate a list of cool URLs I find on the internet.

Find a typo? This blog is open source, pull requests are appreciated.

Thanks for reading! ♡ This work is licensed under CC BY-SA 4.0

︎

What DrupalCamp Berlin taught me about "volunteering" Alte Münze arthur.lorenz Thu, 11/14/2024 - 09:15 From hesitation to enthusiasm: when I decided to volunteer at DrupalCamp Berlin, I didn't know what to expect. In hindsight, it was an unforgettable experience that allowed me to immerse myself deeply in the community. In this post, I share behind-the-scenes insights and what it really means to help shape an event like this.

The Summary

To ensure Drupal’s stability and independence, the project is managed through a well-established, transparent governance system. Dries Buytaert, the Founder and Project Lead, helped design a model that distributes power and prevents any single person or entity — even himself — from making unilateral decisions that could alter the project unexpectedly. The independent Drupal Association oversees Drupal.org and other key infrastructure, free from commercial pressures. This approach ensures that Drupal.org is reliable and creates a fair playing field for all contributors, embodying true open-source leadership.

Just as the Drupal software has grown and changed significantly over its 23-year history, so has its governance. And, while there’s always room for improvement, it is safe to say that Drupal’s seasoned governance is what allows it to be one of the largest, independent open source projects in the world. 

The Detail

Dries Buytaert, as the founder and project lead, ultimately guides the direction of Drupal, and is responsible for shaping the project’s philosophy and core principles. 

While Dries started Drupal on his own, he has helped evolve the governance model over the years to be mature and resilient.  To help govern the project's technical aspects, Dries established the core committer team and other supporting groups. To  oversee non-technical areas, he co-founded the Drupal Association. These initiatives were intentional efforts to scale and strengthen Drupal’s governance.

On the technical side, the governance model for Drupal core is very mature, as described in the Drupal Project Governance. Technical decision-making is distributed among the core committers and other maintainers, promoting a transparent, structured, and collaborative approach to managing Drupal core.     

Many other aspects of Drupal governance are managed by the Drupal Association, which is a U.S. 501(c)3 nonprofit organization formed in 2008 to support the Drupal project and the Drupal community.  I am currently the Chief Executive Officer of the Association.  Our mission is to drive innovation and adoption of Drupal as a high-impact digital public good, hand-in-hand with our open source community.  A fundamental obligation of the Drupal Association is to ensure that Drupal is available to anyone, anywhere in the world free of charge.  We primarily accomplish this task through Drupal.org.

The Drupal Association is a bona fide non-profit organization (not a pass-through), with assets of just over $3 million and an operating budget of over $4 million. We publish our finances annually (see: Find the reports in the Accountability section of D.org).  The Association is not controlled or funded by any single entity nor does it pass revenues onto another entity.  The Association’s revenue comes from hundreds of organizations and thousands of individuals.  No single financial contributor accounts for more than 10% of our revenue. This diverse support base prevents any one entity from having too much influence.

The Drupal Association employs a full-time team of 19 professionals located throughout the world.  These people include engineers, marketers, accountants, communication staff, and program administration team members.  I say all this to demonstrate that we have the capacity to legitimately, and independently, carry out our mission.

The Drupal Association owns and controls important components of the Drupal ecosystem that allow Drupal to be one of the largest independent FOSS projects in the world.

The Drupal Association owns and/or controls the infrastructure that powers Drupal.org.  The Drupal Association has complete control over who accesses Drupal.org, how they access it, and what they can do when accessing it.  These are covered by our Terms of Service.

In administering Drupal.org, the Drupal Association controls a number of services, including:

• The database of Drupal.org users/project contributors
• A self-hosted GitLab instance that includes all of the Drupal code repositories for core and contrib, testing with GitLab CI and documentation through GitLab Pages
• Drupal software packaging (the actual .zip and .tar.gz files containing Drupal code)
• Drupal Updates (the Updates.xml feed, Automatic Updates endpoint, Secure Signing server, and Packages.Drupal.org- the composer endpoint for Drupal projects).
• The Drupal namespace on GitHub
• The Drupal namespace on Packagist
• The Drupal namespace on NPM
• The Drupal Infrastructure namespace on gitlab.com (separate from our self-hosted instance)
• The contribution credit system
• Usage data about Drupal core and extensions

The Drupal Association also owns and controls the primary means by which the community communicates and gathers.  We organize DrupalCons and manage Drupal Slack.  We issue The Drupal Association Newsletter and TheWeeklyDrop (together with Bob Kepford).  We control and manage Mastodon, X/Twitter, YouTube, LinkedIn (Drupal, Drupal Association, Drupal Jobs), Facebook, Instagram, and TikTok.

Drupal has the Maker/Taker Problem that nearly all open source projects face.  There are companies that profit off Drupal who don’t give back to help maintain the project.  The Drupal Association has chosen to address this issue by restructuring our Drupal Certified Partner program to focus exclusively on those companies that give back to the community.  The goal is to incentivize the creation of a culture of contribution within companies that work in Drupal that provide the Drupal Project with sufficient resources to innovate and grow.  There is always work to be done in creating a more equitable program, but it is beginning to work as we have more than doubled the number of Drupal Certified Partners in the past 15 months.

The Drupal Association is governed by a 12-person Board of Directors that meets several times a year, including two public meetings at DrupalCons.  Nine directors are selected by a Nominating Committee of the board and two directors are elected by members of the Drupal Association.  The final seat is the “Founding Director”.  This is a voting seat that can only be filled by Dries Buytaert.  Like all board seats, this is an unpaid, voluntary role that carries with it a single vote on the board.  It has to be approved annually by the Board of Directors. Except for the trademark licensing, the Drupal Association has no contracts or agreements with Dries Buytaert or the Drupal Project, and Dries receives no funding from the Drupal Association or its operation of Drupal.org.

Dries Buytaert owns the trademark “Drupal”.  He has transparently communicated the Drupal Trademark and Logo Policy by which these are governed.  Under the policy, any changes to the policy go into effect sixty (60) days after publication.  Dries Buytaert also owns the domain names “drupal.org”, “drupal.com” and “drupalcon.org”.

Dries has granted the Drupal Association an exclusive license to use “Drupal”, “Drupal.org”, and “DrupalCon” and a non-exclusive license to use Drupal for non-commercial uses.  This license allows the Drupal Association to support the Drupal Project by providing the infrastructure to host and maintain the official version of Drupal and to organize its contributors.  It also allows the Association to support the Drupal Community in their work with Drupal.

The net effect of this arrangement is that Dries Buytaert retains ultimate control over what software can be named “Drupal” and what website can be named “Drupal.org.”  He can thus ensure that any software that calls itself “Drupal” or website that uses “Drupal.org” conforms with his vision.  This would likely cause the Drupal Association to fork the software and maintain it under a new name and url.  The high cost of such an action to both parties makes this option highly unlikely and unable to execute quickly.

What the trademark does not allow him to do is to block any person or organization from using any component of Drupal core or any modules housed on Drupal.org.  Those decisions are the sole discretion of the Drupal Association.  To date, we have exercised this authority in a very limited manner to protect and safeguard the website and its content from attacks and misuse.

Twenty-three years ago, Dries chose to release Drupal under an open-source license, inspiring tens of thousands to build careers and champion an Open Web. However, fulfilling this vision required more than just a General Public License. By creating the Drupal Association, setting up Drupal core's governance, and licensing the trademark, Dries ensured Drupal remained open-source without commercial entanglements, securing a strong, independent foundation.

Along with Dries Buytaert and many contributors, the Drupal Association is focused on the future of Drupal (see: Starshot Initiative). How can we support its adoption through marketing and create sustainable revenue streams for Drupal to flourish?  These are tough questions that confront many open source projects.  Our governance allows us to move forward in this work with great certainty.

Installing private python packages into Docker container can be tricky because container does not have access to private repositories and you do not want to leave trace of private ssh key in docker…

"Django app name translation in admin" is small drop-in django application that overrides few admin templates thus allowing app names in Django admin to be translated.

create documentation for django website

Telling Django to rename all uploaded files in ASCII encoding is easy and takes only two steps.

Dictionary comprehensions are a concise and quick way to create, transform, and filter dictionaries in Python. They can significantly enhance your code’s conciseness and readability compared to using regular for loops to process your dictionaries.

Understanding dictionary comprehensions is crucial for you as a Python developer because they’re a Pythonic tool for dictionary manipulation and can be a valuable addition to your programming toolkit.

In this tutorial, you’ll learn how to:

• Create dictionaries using dictionary comprehensions
• Transform existing dictionaries with comprehensions
• Filter key-value pairs from dictionaries using conditionals
• Decide when to use dictionary comprehensions

To get the most out of this tutorial, you should be familiar with basic Python concepts, such as for loops, iterables, and dictionaries, as well as list comprehensions.

Get Your Code: Click here to download the free sample code that you’ll use to learn about dictionary comprehensions in Python.

Creating and Transforming Dictionaries in Python

In Python programming, you’ll often need to create, populate, and transform dictionaries. To do this, you can use dictionary literals, the dict() constructor, and for loops. In the following sections, you’ll take a quick look at how to use these tools. You’ll also learn about dictionary comprehensions, which are a powerful way to manipulate dictionaries in Python.

Creating Dictionaries With Literals and dict()

To create new dictionaries, you can use literals. A dictionary literal is a series of key-value pairs enclosed in curly braces. The syntax of a dictionary literal is shown below:

Python Syntax {key_1: value_1, key_2: value_2,..., key_N: value_N} Copied!

The keys must be hashable objects and are commonly strings. The values can be any Python object, including other dictionaries. Here’s a quick example of a dictionary:

Python >>> likes = {"color": "blue", "fruit": "apple", "pet": "dog"} >>> likes {'color': 'blue', 'fruit': 'apple', 'pet': 'dog'} >>> likes["hobby"] = "guitar" >>> likes {'color': 'blue', 'fruit': 'apple', 'pet': 'dog', 'hobby': 'guitar'} Copied!

In this example, you create dictionary key-value pairs that describe things people often like. The keys and values of your dictionary are string objects. You can add new pairs to the dictionary using the dict[key] = value syntax.

Note: To learn more about dictionaries, check out the Dictionaries in Python tutorial.

You can also create new dictionaries using the dict() constructor:

Python >>> dict(apple=0.40, orange=0.35, banana=0.25) {'apple': 0.4, 'orange': 0.35, 'banana': 0.25} Copied!

In this example, you create a new dictionary using dict() with keyword arguments. In this case, the keys are strings and the values are floating-point numbers. It’s important to note that the dict() constructor is only suitable for those cases where the dictionary keys can be strings that are valid Python identifiers.

Using for Loops to Populate Dictionaries

Sometimes, you need to start with an empty dictionary and populate it with key-value pairs dynamically. To do this, you can use a for loop. For example, say that you want to create a dictionary in which keys are integer numbers and values are powers of 2.

Here’s how you can do this with a for loop:

Python >>> powers_of_two = {} >>> for integer in range(1, 10): ... powers_of_two[integer] = 2**integer ... >>> powers_of_two {1: 2, 2: 4, 3: 8, 4: 16, 5: 32, 6: 64, 7: 128, 8: 256, 9: 512} Copied!

In this example, you create an empty dictionary using an empty pair of curly braces. Then, you run a loop over a range of integer numbers from 1 to 9. Inside the loop, you populate the dictionary with the integer numbers as keys and powers of two as values.

The loop in this example is readable and clear. However, you can also use dictionary comprehension to create and populate a dictionary like the one shown above.

Introducing Dictionary Comprehensions Read the full article at https://realpython.com/python-dictionary-comprehension/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Textual-cogs is a collection of Textual dialogs that you can use in your Textual application. You can see a quick demo of the dialogs below:

Dialogs included so far:

• Generic MessageDialog – shows messages to the user
• SaveFileDialog – gives the user a way to select a location to save a file
• SingleChoiceDialog – gives the user a series of choices to pick from
• TextEntryDialog – ask the user a question and get their answer using an Input widget
• and more

You can check out textual-cogs on GitHub.

Installation

 

You can install textual-cog using pip:

python -m pip install textual-cog

You also need Textual to run these dialogs.

Example Usage

 

Here is an example of creating a small application that opens the MessageDialog immediately. You would normally open the dialog in response to a message or event that has occurred, such as when the application has an error or you need to tell the user something.

from textual.app import App from textual.app import App, ComposeResult from textual_cogs.dialogs import MessageDialog from textual_cogs import icons class DialogApp(App): def on_mount(self) -> ComposeResult: def my_callback(value: None | bool) -> None: self.exit() self.push_screen( MessageDialog( "What is your favorite language?", icon=icons.ICON_QUESTION, title="Warning", ), my_callback, ) if __name__ == "__main__": app = DialogApp() app.run()

When you run this code, you will get something like the following:

 

Creating a SaveFileDialog

 

The following code demonstrates how to create a SaveFileDialog:

from textual.app import App from textual.app import App, ComposeResult from textual_cogs.dialogs import SaveFileDialog class DialogApp(App): def on_mount(self) -> ComposeResult: self.push_screen(SaveFileDialog()) if __name__ == "__main__": app = DialogApp() app.run()

When you run this code, you will see the following:

Wrapping Up

The textual-cogs package is currently only a collection of reusable dialogs for your Textual application. However, this can help speed up your ability to add code to your TUI applications because the dialogs are taken care of for you.

Check it out on GitHub or the Python Package Index today.

The post ANN – The textual-cogs Package – Creating Reusable Dialogs for Textual appeared first on Mouse Vs Python.

Managing Multilingual Content in Drupal 10 Multisites kanapatrick Wed, 11/13/2024 - 13:51

In an increasingly globalized world, businesses are turning to multilingual solutions to reach an international audience. Drupal 10 offers a powerful multisite architecture that allows you to manage multiple sites from a single installation, ideal for organizations with a global reach.

Integrating Drupal with other systems is a common part of creating or developing a website or web application. Although Drupal offers many tools to facilitate this process, encountering minor or major difficulties is simply inevitable. Based on our knowledge from several hundred projects for clients, we’ve compiled a list of the common problems. It’s worth familiarizing yourself with them to effectively avoid them and speed up the implementation of integration projects.

In this quiz, you’ll test your understanding of how to use Python’s built-in functions input() and print() for basic input and output operations.

You’ll also revisit how to use readline to improve the user experience when collecting input, and how to format output using the sep and end keyword arguments of print().

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Julius wrote an insightful blog post about the “modern sleep” issue with Windows [1]. Basically Microsoft decided that the right way to run laptops is to never entirely sleep, which uses more battery but gives better options for waking up and doing things. I agree with Microsoft in concept and this is something that is a problem that can be solved. A phone can run for 24+ hours without ever fully sleeping, a laptop has a more power hungry CPU and peripherals but also has a much larger battery so it should be able to do the same. Some of the reviews for Snapdragon Windows laptops claim up to 22 hours of actual work without charging! So having suspend not really stop the system should be fine.

The ability of a phone to never fully sleep is a change in quality of the usage experience, it means that you can access it and immediately have it respond and it means that all manner of services can be checked for new updates which may require a notification to the user. The XMPP protocol (AKA Jabber) was invented in 1999 which was before laptops were common and Instant Message systems were common long before then. But using Jabber or another IM system on a desktop was a very different experience to using it on a laptop and using it on a phone is different again. The “modern sleep” allows laptops to act like phones in regard to such messaging services. Currently I have Matrix IM clients running on my Android phone and Linux laptop, if I get a notification that takes much typing for a response then I get out my laptop to respond. If I had an ARM based laptop that never fully shut down I would have much less need for Matrix on a phone.

Making “modern sleep” popular will lead to more development of OS software to work with it. For Linux this will hopefully mean that regular Linux distributions (as opposed to Android which while running a Linux kernel is very different to Debian etc) get better support for such things and therefore become more usable on phones. Debian on a Librem 5 or PinePhonePro isn’t very usable due to battery life issues.

A laptop with an LTE card can be used for full mobile phone functionality. With “modern sleep” this is a viable option. I am tempted to make a laptop with LTE card and bluetooth headset a replacement for my phone. Some people will say “what if someone tries to call you when it’s not convenient to have your laptop with you”, my response is “what if people learn to not expect me to answer the phone at any time as they managed that in the 90s”. Seriously SMS or Matrix me if you want an instant response and if you want a long chat schedule it via SMS or Matrix.

Dell has some useful advice about how to use their laptops (and probably most laptops from recent times) in this regard [2]. You can’t close the lid before unplugging the power cable you have to unplug first and then close. You shouldn’t put a laptop in a sealed bag for travel either. This is a terrible situation, you can put a tablet in a bag and don’t need to take any special precautions when unplugging and laptops should work the same. The end result of what Microsoft, Dell, Intel, and others are doing will be good but they are making some silly design choices along the way! I blame Intel mostly for selling laptop CPUs with TDPs >40W!

For an amusing take on this Linus Tech Tips has a video about being forced to use MacBooks by Microsoft’s implementation of Modern Sleep [3].

I’ll try out some ARM laptops in the near future and blog about how well they work on Debian.

• [1] https://blog.jeujeus.de/blog/hardware/laptops-will-not-sleep-anymore/
• [2] https://tinyurl.com/2b7mxaj7
• [3] https://www.youtube.com/watch?v=OHKKcd3sx2c

Related posts:

1. Modern Laptops Suck One of the reasons why I’m moving from a laptop...
2. Cooling a Thinkpad Late last year I wrote about the way that modern...
3. Cheap Laptops for Children I was recently browsing an electronics store and noticed some...

The goal

When building C++ code with CMake, it is very common to want to set some pre-processor defines in the CMake code.

For instance, we might want to set the project’s version number in a single place, in CMake code like this:

project(MyApp VERSION 1.5)

This sets the CMake variable PROJECT_VERSION to 1.5, which we can then use to pass -DMYAPP_VERSION_STRING=1.5 to the C++ compiler. The about dialog of the application can then use this to show the application version number, like this:

const QString aboutString = QStringLiteral("My App version: %1").arg(MYAPP_VERSION_STRING); QMessageBox::information(this, "My App", aboutString);

Similarly, we might have a boolean CMake option like START_MAXIMIZED, which the user compiling the software can set to ON or OFF:

option(START_MAXIMIZED "Show the mainwindow maximized" OFF)

If it’s ON, you would pass -DSTART_MAXIMIZED, otherwise nothing. The C++ code will then use #ifdef. (We’ll see that there’s a better way.)

#ifdef START_MAXIMIZED w.showMaximized(); #else w.show(); #endif The common (but suboptimal) solution

A solution that many people use for this is the CMake function add_definitions. It would look like this:

add_definitions(-DMYAPP_VERSION_STRING="${PROJECT_VERSION}") if (START_MAXIMIZED) add_definitions(-DSTART_MAXIMIZED) endif()

Technically, this works but there are a number of issues.

First, add_definitions is deprecated since CMake 3.12 and add_compile_definitions should be used instead, which allows to remove the leading -D.

More importantly, there’s a major downside to this approach: changing the project version or the value of the boolean option will force CMake to rebuild every single .cpp file used in targets defined below these lines (including in subdirectories). This is because add_definitions and add_compile_definitions ask to pass -D to all cpp files, instead of only those that need it. CMake doesn’t know which ones need it, so it has to rebuild everything. On large real-world projects, this could take something like one hour, which is a major waste of time.

A first improvement we can do is to at least set the defines to all files in a single target (executable or library) instead of “all targets defined from now on”. This can be done like this:

target_compile_definitions(myapp PRIVATE MYAPP_VERSION_STRING="${PROJECT_VERSION}") if(START_MAXIMIZED) target_compile_definitions(myapp PRIVATE START_MAXIMIZED) endif()

We have narrowed the rebuilding effect a little bit, but are still rebuilding all cpp files in myapp, which could still take a long time.

The recommended solution

There is a proper way to do this, such that only the files that use these defines will be rebuilt; we simply have to ask CMake to generate a header with #define in it and include that header in the few cpp files that need it. Then, only those will be rebuilt when the generated header changes. This is very easy to do:

configure_file(myapp_config.h.in myapp_config.h)

We have to write the input file, myapp_config.h.in, and CMake will generate the output file, myapp_config.h, after expanding the values of CMake variables. Our input file would look like this:

#define MYAPP_VERSION_STRING "${PROJECT_VERSION}" #cmakedefine01 START_MAXIMIZED

A good thing about generated headers is that you can read them if you want to make sure they contain the right settings. For instance, myapp_config.h in your build directory might look like this:

#define MYAPP_VERSION_STRING "1.5" #define START_MAXIMIZED 1

For larger use cases, we can even make this more modular by moving the version number to another input file, say myapp_version.h.in, so that upgrading the version doesn’t rebuild the file with the showMaximized() code and changing the boolean option doesn’t rebuild the about dialog.

If you try this and you hit a “file not found” error about the generated header, that’s because the build directory (where headers get generated) is missing in the include path. You can solve this by adding set(CMAKE_INCLUDE_CURRENT_DIR TRUE) near the top of your CMakeLists.txt file. This is part of the CMake settings that I recommend should always be set; you can make it part of your new project template and never have to think about it again.

There’s just one thing left to explain: what’s this #cmakedefine01 thing?

If your C++ code uses #ifdef, you want to use #cmakedefine, which either sets or doesn’t set the define. But there’s a major downside of doing that — if you forget to include myapp_config.h, you won’t get a compile error; it will just always go to the #else code path.

We want a solution that gives an error if the #include is missing. The generated header should set the define to either 0 or 1 (but always set it), and the C++ code should use #if. Then, you get a warning if the define hasn’t been set and, because people tend to ignore warnings, I recommend that you upgrade it to an error by adding the compiler flag -Werror=undef, with gcc or clang.  Let me know if you are aware of an equivalent flag for MSVC.

if(CMAKE_COMPILER_IS_GNUCXX OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") target_compile_options(myapp PRIVATE -Werror=undef) endif()

And these are all the pieces we need. Never use add_definitions or add_compile_definitions again for things that are only used by a handful of files. Use configure_file instead, and include the generated header. You’ll save a lot of time compared to recompiling files unnecessarily.

I hope this tip was useful.

For more content on CMake, we curated a collection of resources about CMake with or without Qt. Check out the videos.

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post Setting C++ Defines with CMake appeared first on KDAB.

Introduction

eGenix PyRun™ is our open source, one file, no installation version of Python, making the distribution of a Python interpreter to run Python based scripts and applications to Unix based systems simple and efficient.

eGenix PyRun's executable only needs 4-6MB on disk, but still supports most Python applications and scripts.

Compared to a regular Python installation of typically 100MB on disk, eGenix PyRun is ideal for applications and scripts that need to be distributed to containers, VMs, clusters, client installations, customers or end-users.

It makes "installing" Python on a Unix based system as simple as copying a single file.

eGenix has been using eGenix PyRun as run-time for the Linux version of mxODBC Connect Server product since 2008 with great success and decided to make it available as a stand-alone open-source product.

We provide the source archive to build your own eGenix PyRun on Github, as well as a few binary distributions to get you started on Linux x86_64. In the future, we will set up automated builds for several other platforms.

Please see the product page for more details:

    >>> eGenix PyRun - One file Python Runtime

News

This major release of eGenix PyRun comes with the following enhancements:

Enhancements / Changes

• Added support for Python 3.12
• Added support for LTO release builds
• Added dev build targets for development; these don't use PGO and thus build faster

For a complete list of changes, please see the eGenix PyRun Changelog. Downloads

Please visit the eGenix PyRun product page for downloads, instructions on installation and documentation of the product.

Support

Commercial support for this product is available directly from eGenix.com.

Please see the support section of our website for details.

More Information

For more information on eGenix PyRun, licensing and download instructions, please write to sales@egenix.com.

Enjoy !

Marc-Andre Lemburg, eGenix.com