Feeds

Seth Michael Larson: Visualizing the Python package SBOM data flow

Planet Python - Thu, 2024-11-21 19:00

Visualizing the Python package SBOM data flow About • Blog • Cool URLs Visualizing the Python package SBOM data flow

Published 2024-11-22 by Seth Larson
Reading time: minutes

This critical role would not be possible without funding from the Alpha-Omega project.

TLDR: Skip intro, take me to the visualization!

I'm working on improving measurability of Python packages by allowing Software Bill-of-Materials documents (SBOM) to be included in Python packages so that projects and build tools can record information about a package for downstream use.

This is a cross-functional project where I need input from Python projects, Python packaging tools (build backends+tools and installers), but also from folks completely outside the Python community like SBOM tooling maintainers. With projects like this, it can be difficult to "see the forest through the trees". When you're reviewing the packaging PEP, it can be difficult to imagine how or who is using the new standard. This article is to help visualize the end-to-end data flow.

How SBOM data will be included in Python packages

In short, the proposal is:

Allow Python projects to manually specify SBOM documents in pyproject.toml with [project].sbom-files = ["..."]
Allow Python package archives to include self-describing SBOM documents and reference them in metadata via Sbom-File field.
Zero-or-more SBOM documents per Python package archive. Each tool adding SBOM data creates a new SBOM inside the archive to avoid conflicts. End-user SBOM tools need to handle multiple SBOMs to "stitch" them together.

End-to-end SBOM data flow

There are two Python packages being shown, Package A on the left and Package B on the right. Package A depends on Package B. Package A is a pure-Python package with no bundled dependencies. Package B uses binary extensions and uses auditwheel to bundle shared libraries.

@import url(https://fonts.googleapis.com/css2?family=Inter:wght@400;500);
AuditwheelAuditwheelPython EnvironmentPython EnvironmentBuild BackendBuild BackendPython
Package

Python...Python
Package B

Python...Source ForgeSource ForgeSource Code BSource Code BSBOM GeneratorSBOM GeneratorSrc
SBOMSrc...Src
SBOMSrc...Build
SBOMBuild...3rd P
Deps3rd P...SO /
DLLsSO /...Build
SBOMBuild...Src
SBOMSrc...Build
SBOMBuild...3rd P
Deps3rd P...Py
Pkg BPy...Build
SBOMBuild...Src
SBOMSrc...Build
SBOMBuild...METADATAMETADATAPython
Package B

Python...METADATAMETADATAOperational SBOM (OBOM)Operational SBOM (OBOM)1122335566Package BPackage BDataDataDataDataDataDataDataDataBuild BackendBuild BackendPython
Package A

Python...Source ForgeSource ForgeSource Code ASource Code AMETADATAMETADATAPackage APackage ADataDataPython
Package A

Python...METADATAMETADATAPython Package IndexPython Package Indexinstall_requiresinstall_re...44DEPENDS_ONDEPENDS_ONrefrefrefrefrefrefText is not SVG - cannot display
How SBOM data flows from Python package source code, build, to an SBOM generation tool

Stage 1: If the Python project bundles third-party software in their own source code then the project may specify one or more SBOM documents through project.sbom-files in pyproject.toml. Build backends copy these documents into source distributions and wheels.

Stage 2: If the Python build-backend pulls dependencies (like Maturin and Cargo) while building a wheel those dependencies can be recorded in another SBOM document in the wheel.

Stage 3: If a tool that modifies wheels by adding dependencies is used (like auditwheel) then that tool can record modifications in an SBOM document. At this point there are three separate SBOM documents included in the Package B archive.

Stage 4: Archives are uploaded to an index like PyPI. The index can do some validation of included SBOM documents, if any.

Stage 5: Installers download and install the Python package archives. The SBOM files are placed into the .dist-info/sboms/ directory in the Python environment and referenced in package metadata.

Stage 6: SBOM generation tools scan the Python environment and using existing Python package metadata and new SBOM documents with per-package data stitch together an Operational SBOM (OBOM) detailing the Python environment.

Who does what?

The plan is to allow each "actor" in the system adding SBOM data to a Python package to create their own SBOM document inside the Python package.

This means they can choose any SBOM standard (although we'll recommend sticking to a well-known one like CycloneDX and SPDX) and that intermediate tools won't need to "merge" SBOM data together. Avoiding this merging is extremely important, because cross-standard SBOM data merges are a very hard problem. This problem is deferred to SBOM generation tools which already need to support multiple SBOM standards.

Pure-Python projects that don't vendor software are easy, there's nothing to do here.
Python projects that vendor software can annotate that software using an SBOM and specify the SBOM in pyproject.toml. Keeping this up-to-date is a non-zero amount of work, but I am hoping that by providing this PEP it will enable these types of contributions. I'm also hoping to provide a lightweight pre-commit hook to help keeping these SBOM documents up-to-date, similar to what CPython already uses.
Python project which use a build backend that pull dependencies should be able to annotate what those dependencies are at build time. There will be exceptions, looking into tools like Meson and multibuild to see what can be done.
Python bundling tools like auditwheel, delocate, etc can annotate shared libraries and DLLs that are pulled into wheels.

My hope is that the most difficult part of this work (manually annotating a package if automatic tools can't) will enable a new type of contribution from users of Python packages to provide SBOM data. Previously there was no standardized method to have SBOM data propagate through Python packages, thus discouraged this type of contribution.

If you're interested in having your use-case covered or you have concerns about the approach, please open a GitHub issue on the project tracker.

That's all for this post! 👋 If you're interested in more you can read the last report.

Have thoughts or questions? Let's chat over email or social:

sethmichaellarson@gmail.com
@sethmlarson@fosstodon.org

Want more articles like this one? Get notified of new posts by subscribing to the RSS feed or the email newsletter. I won't share your email or send spam, only whatever this is!

Want more content now? This blog's archive has ready-to-read articles. I also curate a list of cool URLs I find on the internet.

Find a typo? This blog is open source, pull requests are appreciated.

Thanks for reading! ♡ This work is licensed under CC BY-SA 4.0

︎

Search form

Tag cloud

You are here

Feeds

Pages

Recent Publications

FLOSS Project Planets

FLOSS Research