GNU Planet!

Subscribe to GNU Planet! feed
Planet GNU - https://planet.gnu.org/
Updated: 1 hour 6 min ago

libtool @ Savannah: libtool-2.5.0 released [alpha]

Mon, 2024-05-13 15:06

Libtoolers!

The Libtool Team is pleased to announce the release of libtool 2.5.0, a alpha release.

GNU Libtool hides the complexity of using shared libraries behind a
consistent, portable interface. GNU Libtool ships with GNU libltdl, which
hides the complexity of loading dynamic runtime libraries (modules)
behind a consistent, portable interface.

There have been 91 commits by 29 people in the 113 weeks since 2.4.7.

See the NEWS below for a brief summary.

Thanks to everyone who has contributed!
The following people contributed changes to this release:

  Albert Chu (1)
  Alex Ameen (3)
  Antonin Décimo (3)
  Brad Smith (2)
  Bruno Haible (2)
  Dmitry Antipov (1)
  Florian Weimer (1)
  Gilles Gouaillardet (1)
  Ileana Dumitrescu (24)
  Jakub Wilk (1)
  Jonathan Wakely (2)
  Manoj Gupta (1)
  Mike Frysinger (23)
  Mingli Yu (2)
  Oliver Kiddle (1)
  Olly Betts (1)
  Ozkan Sezer (2)
  Paul Eggert (2)
  Paul Green (1)
  Raul E Rangel (1)
  Richard Purdie (5)
  Sam James (4)
  Samuel Thibault (1)
  Stephen Webb (1)
  Tijl Coosemans (1)
  Tim Rice (1)
  Uwe Kleine-König (1)
  Vadim Zeitlin (1)
  Xiang.Lin (1)

Ileana
 [on behalf of the libtool maintainers]
==================================================================

Here is the GNU libtool home page:
    https://gnu.org/s/libtool/

For a summary of changes and contributors, see:
  https://git.sv.gnu.org/gitweb/?p=libtool.git;a=shortlog;h=v2.5.0
or run this command from a git-cloned libtool directory:
  git shortlog v2.4.7..v2.5.0

Here are the compressed sources:
  https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.gz   (1.9MB)
  https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.xz   (1008KB)

Here are the GPG detached signatures:
  https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.gz.sig
  https://alpha.gnu.org/gnu/libtool/libtool-2.5.0.tar.xz.sig

Use a mirror for higher download bandwidth:
  https://www.gnu.org/order/ftp.html

Here are the SHA1 and SHA256 checksums:

  fb3ab5907115b16bf12a0d3d424c79cb0003d02e  libtool-2.5.0.tar.gz
  1DjDF0VdhVVM4vmYvkiGb9QM/L+DTWCzAm9PwO1YPSM=  libtool-2.5.0.tar.gz
  70e2dd113a9460c279df01b2eee319adb99ee998  libtool-2.5.0.tar.xz
  fhDMhjgj1AjsX/6kHUPDckqgiBZldXljydsL77LIecw=  libtool-2.5.0.tar.xz

Verify the base64 SHA256 checksum with cksum -a sha256 --check
from coreutils-9.2 or OpenBSD's cksum since 2007.

Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact.  First, be sure to download both the .sig file
and the corresponding tarball.  Then, run a command like this:

  gpg --verify libtool-2.5.0.tar.gz.sig

The signature should match the fingerprint of the following key:

  pub   rsa4096 2021-09-23 [SC]
        FA26 CA78 4BE1 8892 7F22  B99F 6570 EA01 146F 7354
  uid   Ileana Dumitrescu <ileanadumi95@protonmail.com>
  uid   Ileana Dumitrescu <ileanadumitrescu95@gmail.com>

If that command fails because you don't have the required public key,
or that public key has expired, try the following commands to retrieve
or refresh it, and then rerun the 'gpg --verify' command.

  gpg --locate-external-key ileanadumi95@protonmail.com

  gpg --recv-keys 6570EA01146F7354

  wget -q -O- 'https://savannah.gnu.org/project/release-gpgkeys.php?group=libtool&download=1' | gpg --import -

As a last resort to find the key, you can try the official GNU
keyring:

  wget -q https://ftp.gnu.org/gnu/gnu-keyring.gpg
  gpg --keyring gnu-keyring.gpg --verify libtool-2.5.0.tar.gz.sig

This release was bootstrapped with the following tools:
  Autoconf 2.72e
  Automake 1.16.5
  Gnulib v0.1-6995-g29d705ead1

NEWS

  • Noteworthy changes in release 2.5.0 (2024-05-13) [alpha]


** New features:

  - Pass '-fdiagnostics-color', '-frecord-gcc-switches',
    '-fno-sanitize*', '-Werror', and 'prefix-map' flags.

  - Pass the '-no-canonical-prefixes' linker flag.

  - Pass '-fopenmp=*' for Clang to allow choosing between libgomp and
    libomp.

  - Pass '-shared-libsan', '-static-libsan', 'rtlib=*', and
    'unwindlib=*' for Clang.

  - Expanded process.h inclusion on Windows for more than the
    proprietary MSVC compiler. Other alternative Windows compilers
    also require process.h.

  - Pass 'elf32_x86_64' and 'elf64_x86_64' to the linker on hurd-amd64.

  - Recognize --windows* config triplets.

** Important incompatible changes:

  - Removed test_compile from command line options.

  - By default executables are created with the RUNPATH property for
    the Android linker. RUNPATH works for libraries which are not
    installed in system locations.

  - Removed AC_PROG_SED fallback, as the macro has been supported
    in Autoconf since the 90's.

** Bug fixes:

  - Check for space after -l, -L, and -R linker flags.

  - Updated documentation for tests, the demo directory, and
    elsewhere.

  - Fixed Solaris 11 builds.

  - Clean trailing "/" from sysroot path.

  - Fixed shared library builds for System V.

  - Added mingw to the list of systems not requiring libm.

  - Fixed support for nios2 systems.

  - Fixed linker check for '--whole-archive' support for linkers other
    than ld.

  - Use -Fe instead of -o with MSVC to avoid deprecation warnings.

  - Improved reproducibility of libtool scripts.

  - Avoided MinGW warning by adding CRTIMP.

  - Improved grep portability.

  - Fixed cross-building warnings when checking for file.


** Changes in supported systems or compilers:

  - Removed support for bitrig (--bitrig*).

  - Added support for flang (Fortran LLVM-based) compilers.


Enjoy!

Categories: FLOSS Project Planets

GNU Guix: Authenticate your Git checkouts!

Tue, 2024-05-07 08:08

You clone a Git repository, then pull from it. How can you tell its contents are “authentic”—i.e., coming from the “genuine” project you think you’re pulling from, written by the fine human beings you’ve been working with? With commit signatures and “verified” badges ✅ flourishing, you’d think this has long been solved—but nope!

Four years after Guix deployed its own tool to allow users to authenticate updates fetched with guix pull (which uses Git under the hood), the situation hasn’t changed all that much: the vast majority of developers using Git simply do not authenticate the code they pull. That’s pretty bad. It’s the modern-day equivalent of sharing unsigned tarballs and packages like we’d blissfully do in the past century.

The authentication mechanism Guix uses for channels is available to any Git user through the guix git authenticate command. This post is a guide for Git users who are not necessarily Guix users but are interested in using this command for their own repositories. Before looking into the command-line interface and how we improved it to make it more convenient, let’s dispel any misunderstandings or misconceptions.

Why you should care

When you run git pull, you’re fetching a bunch of commits from a server. If it’s over HTTPS, you’re authenticating the server itself, which is nice, but that does not tell you who the code actually comes from—the server might be compromised and an attacker pushed code to the repository. Not helpful. At all.

But hey, maybe you think you’re good because everyone on your project is signing commits and tags, and because you’re disciplined, you routinely run git log --show-signature and check those “Good signature” GPG messages. Maybe you even have those fancy “✅ verified” badges as found on GitLab and on GitHub.

Signing commits is part of the solution, but it’s not enough to authenticate a set of commits that you pull; all it shows is that, well, those commits are signed. Badges aren’t much better: the presence of a “verified” badge only shows that the commit is signed by the OpenPGP key currently registered for the corresponding GitLab/GitHub account. It’s another source of lock-in and makes the hosting platform a trusted third-party. Worse, there’s no notion of authorization (which keys are authorized), let alone tracking of the history of authorization changes (which keys were authorized at the time a given commit was made). Not helpful either.

Being able to ensure that when you run git pull, you’re getting code that genuinely comes from authorized developers of the project is basic security hygiene. Obviously it cannot protect against efforts to infiltrate a project to eventually get commit access and insert malicious code—the kind of multi-year plot that led to the xz backdoor—but if you don’t even protect against unauthorized commits, then all bets are off.

Authentication is something we naturally expect from apt update, pip, guix pull, and similar tools; why not treat git pull to the same standard?

Initial setup

The guix git authenticate command authenticates Git checkouts, unsurprisingly. It’s currently part of Guix because that’s where it was brought to life, but it can be used on any Git repository. This section focuses on how to use it; you can learn about the motivation, its design, and its implementation in the 2020 blog post, in the 2022 peer-reviewed academic paper entitled Building a Secure Software Supply Chain with GNU Guix, or in this 20mn presentation.

To support authentication of your repository with guix git authenticate, you need to follow these steps:

  1. Enable commit signing on your repo: git config commit.gpgSign true. (Git now supports other signing methods but here we need OpenPGP signatures.)

  2. Create a keyring branch containing all the OpenPGP keys of all the committers, along these lines:

    git checkout --orphan keyring git reset --hard gpg --export alice@example.org > alice.key gpg --export bob@example.org > bob.key … git add *.key git commit -m "Add committer keys."

    All the files must end in .key. You must never remove keys from that branch: keys of users who left the project are necessary to authenticate past commits.

  3. Back to the main branch, add a .guix-authorizations file, listing the OpenPGP keys of authorized committers—we’ll get back to its format below.

  4. Commit! This becomes the introductory commit from which authentication can proceed. The introduction of your repository is the ID of this commit and the OpenPGP fingerprint of the key used to sign it.

That’s it. From now on, anyone who clones the repository can authenticate it. The first time, run:

guix git authenticate COMMIT SIGNER

… where COMMIT is the commit ID of the introductory commit, and SIGNER is the OpenPGP fingerprint of the key used to sign that commit (make sure to enclose it in double quotes if there are spaces!). As a repo maintainer, you must advertise this introductory commit ID and fingerprint on a web page or in a README file so others know what to pass to guix git authenticate.

The commit and signer are now recorded on the first run in .git/config; next time, you can run it without any arguments:

guix git authenticate

The other new feature is that the first time you run it, the command installs pre-push and pre-merge hooks (unless preexisting hooks are found) such that your repository is automatically authenticated from there on every time you run git pull or git push.

guix git authenticate exits with a non-zero code and an error message when it stumbles upon a commit that lacks a signature, that is signed by a key not in the keyring branch, or that is signed by a key not listed in .guix-authorizations.

Maintaining the list of authorized committers

The .guix-authorizations file in the repository is central: it lists the OpenPGP fingerprints of authorized committers. Any commit that is not signed by a key listed in the .guix-authorizations file of its parent commit(s) is considered inauthentic—and an error is reported. The format of .guix-authorizations is based on S-expressions and looks like this:

;; Example ‘.guix-authorizations’ file. (authorizations (version 0) ;current file format version (("AD17 A21E F8AE D8F1 CC02 DBD9 F8AE D8F1 765C 61E3" (name "alice")) ("2A39 3FFF 68F4 EF7A 3D29 12AF 68F4 EF7A 22FB B2D5" (name "bob")) ("CABB A931 C0FF EEC6 900D 0CFB 090B 1199 3D9A EBB5" (name "charlie"))))

The name bits are hints and do not have any effect; what matters is the fingerprints that are listed. You can obtain them with GnuPG by running commands like:

gpg --fingerprint charlie@example.org

At any time you can add or remove keys from .guix-authorizations and commit the changes; those changes take effect for child commits. For example, if we add Billie’s fingerprint to the file in commit A, then Billie becomes an authorized committer in descendants of commit A (we must make sure to add Billie’s key as a file in the keyring branch, too, as we saw above); Billie is still unauthorized in branches that lack A. If we remove Charlie’s key from the file in commit B, then Charlie is no longer an authorized committer, except in branches that start before B. This should feel rather natural.

That’s pretty much all you need to know to get started! Check the manual for more info.

All the information needed to authenticate the repository is contained in the repository itself—it does not depend on a forge or key server. That’s a good property to allow anyone to authenticate it, to ensure determinism and transparency, and to avoid lock-in.

Interested? You can help!

guix git authenticate is a great tool that you can start using today so you and fellow co-workers can be sure you’re getting the right code! It solves an important problem that, to my knowledge, hasn’t really been addressed by any other tool.

Maybe you’re interested but don’t feel like installing Guix “just” for this tool. Maybe you’re not into Scheme and Lisp and would rather use a tool written in your favorite language. Or maybe you think—and rightfully so—that such a tool ought to be part of Git proper.

That’s OK, we can talk! We’re open to discussing with folks who’d like to come up with alternative implementations—check out the articles mentioned above if you’d like to take that route. And we’re open to contributing to a standardization effort. Let’s get in touch!

Acknowledgments

Thanks to Florian Pelz and Simon Tournier for their insightful comments on an earlier draft of this post.

Categories: FLOSS Project Planets

Gary Benson: git submodule forgetting

Thu, 2024-05-02 11:11

Did you forget the -r when cloning a git repo with submodules? The command you’re looking for is git submodule update --init

Categories: FLOSS Project Planets

gnulib @ Savannah: GNU gnulib: gnulib-tool has become much faster

Fri, 2024-04-26 06:12

If you are developer on a package that uses GNU gnulib as part of its build system:

gnulib-tool has been known for being slow for many years. We have listened to your complaints. We have rewritten gnulib-tool in another programming language (Python). It is between 8 times and 100 times faster than the previous implementation.

Both implementations behave identically, that is, produce the same generated files and the same output. Nothing changes in your way to use Gnulib; it's only faster.

In order to reap the new speed:

1. Make sure you have Python (version 3.7 or newer) installed on your machine.

2. Update your gnulib checkout. (For some packages, it comes as a git submodule named 'gnulib'.) Like this:

  $ git checkout master
  $ git pull

  Set the environment variable GNULIB_SRCDIR, pointing to this checkout.

  If the package is using a git submodule named 'gnulib', it is also advisable to do

  $ git commit -m 'build: Update gnulib submodule to latest.' gnulib

  (as a preparation for step 4, because the --no-git option does not work as expected in all variants of 'bootstrap').

3. Clean the built files of your package:

  $ make -k distclean


4. Regenerate the fetched and generated files of your package. Depending on the package, this may be a command such as

  $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR

  or

  $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh

  or, if no such script is available:

  $ $GNULIB_SRCDIR/gnulib-tool --update


5. Continue with

  $ ./configure
  $ make

  as usual.

Enjoy! The rewritten gnulib-tool was implemented by Dmitry Selyutin, Collin Funk, and me.

Categories: FLOSS Project Planets

FSF Blogs: What role community plays in free software and more -- Interview with David Wilson

Wed, 2024-04-24 12:35
Can't wait for the start of LibrePlanet 2024: Cultivating Community? Same here. To sweeten the wait, we have interviewed David Wilson, one of the keynote speakers and the creator of the System Crafters channel and community.
Categories: FLOSS Project Planets

parallel @ Savannah: GNU Parallel 20240422 ('Børsen') [stable]

Mon, 2024-04-22 17:12

GNU Parallel 20240422 ('Børsen') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  I’m a big fan of GNU parallel!
    -- Scott Cain @scottjcain@twitter
 
New in this release:

  • Bug fixes and man page updates.


GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.


About GNU Parallel


GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference


If you use programs that use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --citation)


If GNU Parallel saves you money:



About GNU SQL


GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.


About GNU Niceload


GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

Categories: FLOSS Project Planets

www-zh-cn @ Savannah: Welcome our new member - integral

Sun, 2024-04-21 20:56

Hi, All:

Please join me in welcoming our new member:

 User Details:
-------------
Name:
Login:   integral
Email:   integral@member.fsf.org

I wish integral a wonderful journey in GNU CTT.

Happy Hacking
wxie

Categories: FLOSS Project Planets

gnulib @ Savannah: GNU gnulib: calling for beta-testers

Sun, 2024-04-21 06:47

If you are developer on a package that uses GNU gnulib as part of its build system:

gnulib-tool has been known for being slow for many years. We have listened to your complaints. A rewrite of gnulib-tool in another programming language (Python) is ready for beta-testing. It is between 8 times and 100 times faster than the original gnulib-tool.

Both implementations should behave identically, that is, produce the same generated files and the same output. You can help us ensure this, through the following steps:

1. Make sure you have Python (version 3.7 or newer) installed on your machine.

2. Update your gnulib checkout. (For some packages, it comes as a git submodule named 'gnulib'.) Like this:

  $ git checkout master
  $ git pull

     Set the environment variable GNULIB_SRCDIR, pointing to this checkout.

     If the package is using a git submodule named 'gnulib', it is also advisable to do

  $ git commit -m 'build: Update gnulib submodule to latest.' gnulib

     (as a preparation for step 5, because the --no-git option does not work as expected in all variants of 'bootstrap').

3. Set an environment variable that enables checking that the two implementations behave the same:

  $ export GNULIB_TOOL_IMPL=sh+py


4. Clean the built files of your package:

  $ make -k distclean


5. Regenerate the fetched and generated files of your package. Depending on the package, this may be a command such as

  $ ./bootstrap --no-git --gnulib-srcdir=$GNULIB_SRCDIR

     or

  $ export GNULIB_SRCDIR; ./autopull.sh; ./autogen.sh

     or, if no such script is available:

  $ $GNULIB_SRCDIR/gnulib-tool --update

     If there is a failure, due to differences between the 'sh' and 'py' results, please report it to <bug-gnulib@gnu.org>.

6. If this invocation was successful, you can trust the rewritten gnulib-tool and use it from now on, by setting the environment variable

  $ export GNULIB_TOOL_IMPL=py


7. Continue with

  $ ./configure
  $ make

     as usual.

And enjoy the speed! The rewritten gnulib-tool was implemented by Dmitry Selyutin, Collin Funk, and me.

Categories: FLOSS Project Planets

www-zh-cn @ Savannah: It is easy to contribute to GNU

Wed, 2024-04-17 19:30

I will be delivering my talk, "It is easy to contribute to GNU," Saturday, May 4, 2024, 12:15--13:00 EDT (16:00 UTC), at the LibrePlanet 2024 conference, and I hope you’ll check it out!

LibrePlanet is a conference about software freedom, happening on May 4 & 5, 2024. The event is hosted by the Free Software Foundation (FSF), and brings together software developers, law and policy experts, activists, students, and computer users to learn skills, celebrate free software accomplishments, and face upcoming challenges. Newcomers are always welcome, and LibrePlanet 2024 will feature programming for all ages and experience levels.

*Please register in advance at <https://libreplanet.org/2024/>.*

wxie

Categories: FLOSS Project Planets

GNU Taler news: GNU Taler v0.10 released

Sun, 2024-04-14 18:00
We are happy to announce the release of GNU Taler v0.10.
Categories: FLOSS Project Planets

Simon Josefsson: Reproducible and minimal source-only tarballs

Sat, 2024-04-13 12:44

With the release of Libntlm version 1.8 the release tarball can be reproduced on several distributions. We also publish a signed minimal source-only tarball, produced by git-archive which is the same format used by Savannah, Codeberg, GitLab, GitHub and others. Reproducibility of both tarballs are tested continuously for regressions on GitLab through a CI/CD pipeline. If that wasn’t enough to excite you, the Debian packages of Libntlm are now built from the reproducible minimal source-only tarball. The resulting binaries are hopefully reproducible on several architectures.

What does that even mean? Why should you care? How you can do the same for your project? What are the open issues? Read on, dear reader…

This article describes my practical experiments with reproducible release artifacts, following up on my earlier thoughts that lead to discussion on Fosstodon and a patch by Janneke Nieuwenhuizen to make Guix tarballs reproducible that inspired me to some practical work.

Let’s look at how a maintainer release some software, and how a user can reproduce the released artifacts from the source code. Libntlm provides a shared library written in C and uses GNU Make, GNU Autoconf, GNU Automake, GNU Libtool and gnulib for build management, but these ideas should apply to most project and build system. The following illustrate the steps a maintainer would take to prepare a release:

git clone https://gitlab.com/gsasl/libntlm.git cd libntlm git checkout v1.8 ./bootstrap ./configure make distcheck gpg -b libntlm-1.8.tar.gz

The generated files libntlm-1.8.tar.gz and libntlm-1.8.tar.gz.sig are published, and users download and use them. This is how the GNU project have been doing releases since the late 1980’s. That is a testament to how successful this pattern has been! These tarballs contain source code and some generated files, typically shell scripts generated by autoconf, makefile templates generated by automake, documentation in formats like Info, HTML, or PDF. Rarely do they contain binary object code, but historically that happened.

The XZUtils incident illustrate that tarballs with files that are not included in the git archive offer an opportunity to disguise malicious backdoors. I blogged earlier how to mitigate this risk by using signed minimal source-only tarballs.

The risk of hiding malware is not the only motivation to publish signed minimal source-only tarballs. With pre-generated content in tarballs, there is a risk that GNU/Linux distributions such as Trisquel, Guix, Debian/Ubuntu or Fedora ship generated files coming from the tarball into the binary *.deb or *.rpm package file. Typically the person packaging the upstream project never realized that some installed artifacts was not re-built through a typical autoconf -fi && ./configure && make install sequence, and never wrote the code to rebuild everything. This can also happen if the build rules are written but are buggy, shipping the old artifact. When a security problem is found, this can lead to time-consuming situations, as it may be that patching the relevant source code and rebuilding the package is not sufficient: the vulnerable generated object from the tarball would be shipped into the binary package instead of a rebuilt artifact. For architecture-specific binaries this rarely happens, since object code is usually not included in tarballs — although for 10+ years I shipped the binary Java JAR file in the GNU Libidn release tarball, until I stopped shipping it. For interpreted languages and especially for generated content such as HTML, PDF, shell scripts this happens more than you would like.

Publishing minimal source-only tarballs enable easier auditing of a project’s code, to avoid the need to read through all generated files looking for malicious content. I have taken care to generate the source-only minimal tarball using git-archive. This is the same format that GitLab, GitHub etc offer for the automated download links on git tags. The minimal source-only tarballs can thus serve as a way to audit GitLab and GitHub download material! Consider if/when hosting sites like GitLab or GitHub has a security incident that cause generated tarballs to include a backdoor that is not present in the git repository. If people rely on the tag download artifact without verifying the maintainer PGP signature using GnuPG, this can lead to similar backdoor scenarios that we had for XZUtils but originated with the hosting provider instead of the release manager. This is even more concerning, since this attack can be mounted for some selected IP address that you want to target and not on everyone, thereby making it harder to discover.

With all that discussion and rationale out of the way, let’s return to the release process. I have added another step here:

make srcdist gpg -b libntlm-1.8-src.tar.gz

Now the release is ready. I publish these four files in the Libntlm’s Savannah Download area, but they can be uploaded to a GitLab/GitHub release area as well. These are the SHA256 checksums I got after building the tarballs on my Trisquel 11 aramo laptop:

91de864224913b9493c7a6cec2890e6eded3610d34c3d983132823de348ec2ca libntlm-1.8-src.tar.gz ce6569a47a21173ba69c990965f73eb82d9a093eb871f935ab64ee13df47fda1 libntlm-1.8.tar.gz

So how can you reproduce my artifacts? Here is how to reproduce them in a Ubuntu 22.04 container:

podman run -it --rm ubuntu:22.04 apt-get update apt-get install -y --no-install-recommends autoconf automake libtool make git ca-certificates git clone https://gitlab.com/gsasl/libntlm.git cd libntlm git checkout v1.8 ./bootstrap ./configure make dist srcdist sha256sum libntlm-*.tar.gz

You should see the exact same SHA256 checksum values. Hooray!

This works because Trisquel 11 and Ubuntu 22.04 uses the same version of git, autoconf, automake, and libtool. These tools do not guarantee the same output content for all versions, similar to how GNU GCC does not generate the same binary output for all versions. So there is still some delicate version pairing needed.

Ideally, the artifacts should be possible to reproduce from the release artifacts themselves, and not only directly from git. It is possible to reproduce the full tarball in a AlmaLinux 8 container – replace almalinux:8 with rockylinux:8 if you prefer RockyLinux:

podman run -it --rm almalinux:8 dnf update -y dnf install -y make wget gcc wget https://download.savannah.nongnu.org/releases/libntlm/libntlm-1.8.tar.gz tar xfa libntlm-1.8.tar.gz cd libntlm-1.8 ./configure make dist sha256sum libntlm-1.8.tar.gz

The source-only minimal tarball can be regenerated on Debian 11:

podman run -it --rm debian:11 apt-get update apt-get install -y --no-install-recommends make git ca-certificates git clone https://gitlab.com/gsasl/libntlm.git cd libntlm git checkout v1.8 make -f cfg.mk srcdist sha256sum libntlm-1.8-src.tar.gz

As the Magnus Opus or chef-d’œuvre, let’s recreate the full tarball directly from the minimal source-only tarball on Trisquel 11 – replace docker.io/kpengboy/trisquel:11.0 with ubuntu:22.04 if you prefer.

podman run -it --rm docker.io/kpengboy/trisquel:11.0 apt-get update apt-get install -y --no-install-recommends autoconf automake libtool make wget git ca-certificates wget https://download.savannah.nongnu.org/releases/libntlm/libntlm-1.8-src.tar.gz tar xfa libntlm-1.8-src.tar.gz cd libntlm-v1.8 ./bootstrap ./configure make dist sha256sum libntlm-1.8.tar.gz

Yay! You should now have great confidence in that the release artifacts correspond to what’s in version control and also to what the maintainer intended to release. Your remaining job is to audit the source code for vulnerabilities, including the source code of the dependencies used in the build. You no longer have to worry about auditing the release artifacts.

I find it somewhat amusing that the build infrastructure for Libntlm is now in a significantly better place than the code itself. Libntlm is written in old C style with plenty of string manipulation and uses broken cryptographic algorithms such as MD4 and single-DES. Remember folks: solving supply chain security issues has no bearing on what kind of code you eventually run. A clean gun can still shoot you in the foot.

Side note on naming: GitLab exports tarballs with pathnames libntlm-v1.8/ (i.e.., PROJECT-TAG/) and I’ve adopted the same pathnames, which means my libntlm-1.8-src.tar.gz tarballs are bit-by-bit identical to GitLab’s exports and you can verify this with tools like diffoscope. GitLab name the tarball libntlm-v1.8.tar.gz (i.e., PROJECT-TAG.ARCHIVE) which I find too similar to the libntlm-1.8.tar.gz that we also publish. GitHub uses the same git archive style, but unfortunately they have logic that removes the ‘v’ in the pathname so you will get a tarball with pathname libntlm-1.8/ instead of libntlm-v1.8/ that GitLab and I use. The content of the tarball is bit-by-bit identical, but the pathname and archive differs. Codeberg (running Forgejo) uses another approach: the tarball is called libntlm-v1.8.tar.gz (after the tag) just like GitLab, but the pathname inside the archive is libntlm/, otherwise the produced archive is bit-by-bit identical including timestamps. Savannah’s CGIT interface uses archive name libntlm-1.8.tar.gz with pathname libntlm-1.8/, but otherwise file content is identical. Savannah’s GitWeb interface provides snapshot links that are named after the git commit (e.g., libntlm-a812c2ca.tar.gz with libntlm-a812c2ca/) and I cannot find any tag-based download links at all. Overall, we are so close to get SHA256 checksum to match, but fail on pathname within the archive. I’ve chosen to be compatible with GitLab regarding the content of tarballs but not on archive naming. From a simplicity point of view, it would be nice if everyone used PROJECT-TAG.ARCHIVE for the archive filename and PROJECT-TAG/ for the pathname within the archive. This aspect will probably need more discussion.

Side note on git archive output: It seems different versions of git archive produce different results for the same repository. The version of git in Debian 11, Trisquel 11 and Ubuntu 22.04 behave the same. The version of git in Debian 12, AlmaLinux/RockyLinux 8/9, Alpine, ArchLinux, macOS homebrew, and upcoming Ubuntu 24.04 behave in another way. Hopefully this will not change that often, but this would invalidate reproducibility of these tarballs in the future, forcing you to use an old git release to reproduce the source-only tarball. Alas, GitLab and most other sites appears to be using modern git so the download tarballs from them would not match my tarballs – even though the content would.

Side note on ChangeLog: ChangeLog files were traditionally manually curated files with version history for a package. In recent years, several projects moved to dynamically generate them from git history (using tools like git2cl or gitlog-to-changelog). This has consequences for reproducibility of tarballs: you need to have the entire git history available! The gitlog-to-changelog tool also output different outputs depending on the time zone of the person using it, which arguable is a simple bug that can be fixed. However this entire approach is incompatible with rebuilding the full tarball from the minimal source-only tarball. It seems Libntlm’s ChangeLog file died on the surgery table here.

So how would a distribution build these minimal source-only tarballs? I happen to help on the libntlm package in Debian. It has historically used the generated tarballs as the source code to build from. This means that code coming from gnulib is vendored in the tarball. When a security problem is discovered in gnulib code, the security team needs to patch all packages that include that vendored code and rebuild them, instead of merely patching the gnulib package and rebuild all packages that rely on that particular code. To change this, the Debian libntlm package needs to Build-Depends on Debian’s gnulib package. But there was one problem: similar to most projects that use gnulib, Libntlm depend on a particular git commit of gnulib, and Debian only ship one commit. There is no coordination about which commit to use. I have adopted gnulib in Debian, and add a git bundle to the *_all.deb binary package so that projects that rely on gnulib can pick whatever commit they need. This allow an no-network GNULIB_URL and GNULIB_REVISION approach when running Libntlm’s ./bootstrap with the Debian gnulib package installed. Otherwise libntlm would pick up whatever latest version of gnulib that Debian happened to have in the gnulib package, which is not what the Libntlm maintainer intended to be used, and can lead to all sorts of version mismatches (and consequently security problems) over time. Libntlm in Debian is developed and tested on Salsa and there is continuous integration testing of it as well, thanks to the Salsa CI team.

Side note on git bundles: unfortunately there appears to be no reproducible way to export a git repository into one or more files. So one unfortunate consequence of all this work is that the gnulib *.orig.tar.gz tarball in Debian is not reproducible any more. I have tried to get Git bundles to be reproducible but I never got it to work — see my notes in gnulib’s debian/README.source on this aspect. Of course, source tarball reproducibility has nothing to do with binary reproducibility of gnulib in Debian itself, fortunately.

One open question is how to deal with the increased build dependencies that is triggered by this approach. Some people are surprised by this but I don’t see how to get around it: if you depend on source code for tools in another package to build your package, it is a bad idea to hide that dependency. We’ve done it for a long time through vendored code in non-minimal tarballs. Libntlm isn’t the most critical project from a bootstrapping perspective, so adding git and gnulib as Build-Depends to it will probably be fine. However, consider if this pattern was used for other packages that uses gnulib such as coreutils, gzip, tar, bison etc (all are using gnulib) then they would all Build-Depends on git and gnulib. Cross-building those packages for a new architecture will therefor require git on that architecture first, which gets circular quick. The dependency on gnulib is real so I don’t see that going away, and gnulib is a Architecture:all package. However, the dependency on git is merely a consequence of how the Debian gnulib package chose to make all gnulib git commits available to projects: through a git bundle. There are other ways to do this that doesn’t require the git tool to extract the necessary files, but none that I found practical — ideas welcome!

Finally some brief notes on how this was implementated. Enabling bootstrappable source-only minimal tarballs via gnulib’s ./bootstrap is achieved by using the GNULIB_REVISION mechanism, locking down the gnulib commit used. I have always disliked git submodules because they add extra steps and has complicated interaction with CI/CD. The reason why I gave up git submodules now is because the particular commit to use is not recorded in the git archive output when git submodules is used. So the particular gnulib commit has to be mentioned explicitly in some source code that goes into the git archive tarball. Colin Watson added the GNULIB_REVISION approach to ./bootstrap back in 2018, and now it no longer made sense to continue to use a gnulib git submodule. One alternative is to use ./bootstrap with --gnulib-srcdir or --gnulib-refdir if there is some practical problem with the GNULIB_URL towards a git bundle the GNULIB_REVISION in bootstrap.conf.

The srcdist make rule is simple:

git archive --prefix=libntlm-v1.8/ -o libntlm-v1.8.tar.gz HEAD

Making the make dist generated tarball reproducible can be more complicated, however for Libntlm it was sufficient to make sure the modification times of all files were set deterministically to a timestamp found in the git repository. Interestingly there seems to be a couple of different ways to accomplish this, Guix doesn’t support minimal source-only tarballs but rely on a .tarball-timestamp file inside the tarball. Paul Eggert explained what TZDB is using some time ago. The approach I’m using now is fairly similar to the one I suggested over a year ago.

Doing continous testing of all this is critical to make sure things don’t regress. Libntlm’s pipeline definition now produce the generated libntlm-*.tar.gz tarballs and a checksum as a build artifact. Then I added the 000-reproducability job which compares the checksums and fails on mismatches. You can read its delicate output in the job for the v1.8 release. Right now we insists that builds on Trisquel 11 match Ubuntu 22.04, that PureOS 10 builds match Debian 11 builds, that AlmaLinux 8 builds match RockyLinux 8 builds, and AlmaLinux 9 builds match RockyLinux 9 builds. As you can see in pipeline job output, not all platforms lead to the same tarballs, but hopefully this state can be improved over time. There is also partial reproducibility, where the full tarball is reproducible across two distributions but not the minimal tarball, or vice versa.

If this way of working plays out well, I hope to implement it in other projects too.

What do you think? Happy Hacking!

Categories: FLOSS Project Planets

FSF Blogs: Meet the locals: Come to LibrePlanet and connect with free software supporters in New England

Wed, 2024-04-10 14:11
New England free software supporters: we invite you to come socialize with other local free software supporters at LibrePlanet 2024.
Categories: FLOSS Project Planets

stow @ Savannah: GNU Stow 2.4.0 released

Sun, 2024-04-07 19:22

Stow 2.4.0 has been released. This release contains some much-wanted bug-fixes — specifically, fixing the --dotfiles option to work with dot-foo directories, and avoiding a spurious warning when unstowing. There were also very many clean-ups and improvements, mostly internal and not visible to users. See http://git.savannah.gnu.org/cgit/stow.git/tree/NEWS for more details.

Categories: FLOSS Project Planets

Pages