GNU Planet!

https://www.bloomberg.com/news/features/2025-01-03/chinese-cyber-hackers-terrify-us-intelligence-after-infiltrating-guam

Version 3.18 of GNU Mailutils is available for download.
A short summary of changes follows.

Using all in mailutils debug level specification enables all debugging categories.  Syntactically, all can be used wherever an actual category name is allowed, thus, e.g., all.!=prot enables all levels except prot in all debugging categories.

In particular, mail variables that correspond to some mailutils configuration settings, now correctly reflect their value.

• Minor fix in handling of the EHLO command in smtp client. 
• Improve docs.
• Minor fix in mhn and related tests.
• mail utility: use the mailer configuration capability.

Dear GNU CTT:

Thank you for your contribution and effort.

I am very proud of the performance in 2024 for this team.

Here is summary from GNU translation team for 2024.

Dear GNU translators!

2024 repeated the general traits of 2023: most active teams kept doing
a good job updating the translations, and a few new translations were
made.  Currently, the total amount of translations is over 3350.

      General Statistics

Most new translations were made by the Chinese (zh-cn) team this year;
then the Polish and French teams follow.  The Turkish team, although
it published no new translations this year, made a notable progress
in terms of keeping its translation up-to-date.

The table below shows the number and size of newly translated
articles in important directories and typical number of outdated
GNUNified translations throughout the year.

+-team--+-----new-----+--outdated--+
|  de   | 1 (9.7Ki) * |  124 (61%) |
+-------+-------------+------------+
|  es   | 1 (  5.2Ki) | 0.5 (0.2%) |
+-------+-------------+------------+
|  fr   | 4 ( 42.0Ki) | 0.5 (0.1%) |
+-------+-------------+------------+
|  ja   | 2 (  9.9Ki) |  48 ( 34%) |
+-------+-------------+------------+
|  pl   | 6 ( 85.4Ki) |  54 ( 37%) |
+-------+-------------+------------+
|  ru   | 2 ( 20.7Ki) | 0.3 (0.1%) |
+-------+-------------+------------+
|  sq   | 2 ( 17.1Ki) | 2.3 (2.9%) |
+-------+-------------+------------+
|  tr   | 0 (  0.0Ki) | 0.1 (0.1%) |
+-------+-------------+------------+
| zh-cn | 23 ( 543Ki) |     0 &    |
+-------+-------------+------------+
+-------+-------------+
| total | 39 ( 723Ki) |
+-------+-------------+

I wish you all a freer, healthier, and more peaceful 2025.

Happy hacking
wxie

Version 0.54 of GNU Collaborative International Dictionary of Englis is available for download.

The dictionary corpus underwent a thorough spell-checking.  A number of articles has been fixed or upgraded.  All files has been reformatted to limit physical line length to 72 characters.

If you are using GNU dico to consult the dictionary, please upgrade to version 2.12.

GNU dico version 2.12 is available for download.

This versions provides important improvements in the gcide module:

• idxgcide skips duplicated headwords.
• Fixed some gcide entities, introduced missing ones.
• Fixed Ancient Greek transliteration.

When given this option, the module will watch for modifications in the dictionary corpus files and rebuild the index file when necessary.

HTML output is enabled by the "html" module option.  It is produced only after "OPTION MIME" command command.

The argument to DEFINE or MATCH command can optionally be modified before being used in actual search.  This allows, for example, to input search terms in transliteration instead of in the actual script.

Conversions are implemented by loadable modules and are associated with individual databases.

This version is shipped with the new module greek_kbd, which implements Greek transliteration.

• Switch to Python 3.11+ with type hints
• Upgrade Django to version 4.2
• Improve desktop and mobile views using HTML5
• Implement a dark mode
• Use Poetry and pyproject.toml
• Integrate Pylint and Mypy for development
• Implement various fixes and improvements

Support for obsolescent libpcre has been withdrawn.

We used to have a Mono package. It was introduced on August 8 2016 by commit 763b3d50b6249b43fceda51445bbeb1f5f5fd7d0, at Mono version 4.4.1.0, but it was later discovered in April of 2022 that the release tarball that it was built from included prebuilt binaries. Further research revealed that these binaries were not optional. Due to this, a decision was made to remove the Mono package, carried out on September 1, 2022.

We now once again have a Mono package, due to a patch series I submitted on November 29, which after some revisions was committed on December 22. This patch series introduced a full, 17-mono-package sequence that takes us from a mono-1.2.6 built fully from source to mono-6.12.0 built fully from source, using only packages that already have full bootstrap paths. I make no promise that this is the shortest or most optimal path, but it exists and I have verified it works.

As I've spent what is probably an unreasonable amount of time working toward this, I thought I'd share some of my thoughts, experiences, and commentary. Sorry in advance if it gets a bit rambly or lecture-ish.

I started down this road because someone I'm working on a project with decided to depend on a C# package that requires C# 12.0 features, and my personal Mono package based on the tarball releases (which include bootstrap binaries) only went up to C# 7.0. This meant that the C# package in question de facto required strictly Microsoft's (er, I mean, "the .NET foundation"'s) .NET implementation — hereafter referred to as "dotnet" — and a very recent version no less. The bootstrapping story with dotnet is very bad; even beginning to untangle it would probably require a relatively modern C# compiler, and something that at least sort of understands MSBuild. And there's not much point to "bootstrapping" dotnet from something that isn't bootstrapped itself. So I figured I may as well start with Mono.

While Mono is today probably the most well-known alternative to Microsoft's .NET offerings, it is not the only one. Indeed, in the early 2000s there were at least 2 competing free software implementations: Mono, and DotGNU's Portable.NET (abbreviated pnet). They differed in goals, licenses, and methods: Portable.NET was a GNU project concerned with, among other things, limiting the ability of Microsoft to impose vendor lock-in via its proprietary .NET implementation and software patents. As a GNU project, it used the GPL for its runtime and compiler, and the GPL with a linking exception for its standard library, pnetlib. Mono, on the other hand, used a mix of many copyleft and permissive licenses: X11 for the standard library, GPL for the compiler (later dual-licensed to add an X11 option), and LGPL for the runtime, with GPL and LGPL code also offered "under commercial terms for when the GPL and the LGPL are not suitable". In 2016 after its acquisition by Microsoft, the runtime was relicensed to use the Expat (MIT) license.

But perhaps most importantly to us, while Mono opted to write its C# compiler, mcs, in... C#, Portable.NET's runtime and C# compiler were both written in C. Portable.NET along with the entire DotGNU project (except for LibJIT) was decommissioned in 2012, but the source is still available, and it still works fine (with a few modifications for compatibility with newer versions of its dependencies). In September of 2022, Adam Faiz submitted patches to package pnet and pnetlib, along with one of their dependencies named treecc. These packages were based on the last release of Portable.NET, version 0.8.0, released in 2007. I initially used these packages as the basis for my bootstrap efforts, and even managed to get mono-1.2.6 built using them, but later discovered that using a more recent version from git made it much easier. For example, while pnet-0.8.0 can do pointer arithmetic inside unsafe code blocks, it doesn't support the += or -= operators specifically, which requires lots of patching (after all, who would use x = x + y when you could do x += y?). There are many other similar improvements in the git version, so for this patch series I've decided to go with pnet-git.

After building mono-1.2.6, I tried a few later versions, and the third or fourth one would always fail with errors about missing methods. It turns out that the reason for this is that, contrary to what their marketing suggests, C# and Java are not "write once, run everywhere". This is because their compilers rely on the details of the libraries that the program will be run with at compile-time. This is used, for example, to do overload resolution. Suppose, for example, that a certain implementation of the == operator is present in version 1.0 of a library, and then in version 2.0 of a library a more specific implementation is introduced. Now code that is compiled against version 2.0 may instead automatically reference the more-specific implementation, as is in accordance with the rules of C#. But when it is run with version 1.0, it will fail because that implementation doesn't exist. In my case, for some reason the initial mcs and core libraries being built to compile the rest of Mono were being compiled against a 2.0 library and then run with a 1.0 library. It turns out that this was because mcs uses mono's code for producing assemblies (.NET dlls and exes), and mono decides which version to put in an assembly it writes based on "which runtime version" is being used, and that version is decided at startup based on... the version that was put in the assembly it is running. So for example, mono-1.9.1 would produce 2.0 assemblies because mono-1.2.6 produced 2.0 assemblies because pnet produced 2.0 assemblies. So I modified Mono's runtime in mono-1.9.1 to allow for this version to be overridden via environment variable, and set it to v1.1.4322, and things went a lot more smoothly after that.

From there on it was mostly the usual trial-and-error process of identifying where things had bitrotted. I made sure to unvendor libgc wherever possible, though eventually by mono-4.9.0 they explicitly dropped support in their configure script for using any libgc other than what was bundled, so at that point I switched to using their homebrewed sgen garbage collector.

Once I got to mono-2.11.4, though, things took a turn for the interesting: Mono started using git submodules, and the (recursive? #t) clones were all failing. It turns out that this is because their submodules reference github.com using the git:// protocol.

This is notable for a few reasons.

First, GitHub dropped support for the git:// protocol in 2021, so recursive clones won't work now. This means I have to explicitly list out every submodule, its commit, and its sha256 hash, for every Mono version until they switched to using http or https. mono-2.11.4 has only 4 submodules, but that doesn't last for long: by mono-4.9.0 it has 14 submodules. A significant portion of these patches is just listing these submodules and their hashes. It's a bit annoying.

The more concerning reason, though, is why GitHub dropped support for the git:// protocol: it is unencrypted and unauthenticated. This is mitigated somewhat by the use of sha-1 hashes to identify commits in the referenced submodules, putting a significant computational burden on anyone who would try to alter what was fetched corresponding to a given submodule. Significantly more risky, though, is the process of updating submodules that use git:// URLs. It is quite unlikely that a developer is going to independently clone one of the submodules over https, navigate to a desirable commit, copy the sha-1 hash, and manually update the submodule reference's commit. They're far more likely to run cd submodule; git pull; cd ..; git add submodule; git commit ... or an equivalent.

Of course, any changes a network man-in-the-middle might try to make here would still be reflected in the commit history, so even if a developer did that, they or any of their fellow committers could spot anything strange or malicious and point it out. Also, the changes couldn't be propagated to others trying to pull them who weren't on a network path containing the MITM because the potentially-malicious commit wouldn't be present in the real submodule's repository. So the transparency of git clearly showing changes to text files, combined with the fact that surely no git hosting platform would just allow arbitrary entities to make whatever commits they want accessible under any arbitrary repository URL, rather mitigate this security issue.

This usage of git:// URLs lasted all the way until September 28, 2021, when GitHub's removal of support for it forced the developers to change them to https.

On November 28, 2016, Mono added a submodule named roslyn-binaries. Unsurprisingly, it included binary blobs for Microsoft's Roslyn compiler (which I believe had been open-sourced shortly prior). From here on, Mono's build system would default to using these binaries for building on little-endian systems (though another compiler could be specified with the --with-csc configure flag). I happen to know that it is extremely unlikely that many Mono developers used this configure flag. I know this because the 5.0 series is an absolute pain in the neck to build from source, because they consistently depend on new C# features before they implement them.

To go on a brief tangent: does anyone remember back when youtube-dl was temporarily taken down from GitHub due to the RIAA's DMCA request? Many were unhappy about that. One such unhappy person made news when they made the full contents of youtube-dl's repository available to access through the DMCA request repository. It turns out that there are many actions that one can take on GitHub that will make arbitrary commits available under arbitrary repository URLs.

So, in reality, for the span of time from November 28, 2016 to September 28, 2021, anybody sitting on the network path between GitHub and any Mono developer updating the roslyn-binaries submodule could decide on any arbitrary new commit to be used. Of course, merely inspecting the diff for the commit will reveal nothing of use, because the contents are binary blobs. And not only are these blobs those of a compiler, they are the blobs of a compiler that is sure to be used to compile another compiler, which will then be redistributed as an opaque, non-bootstrappable binary blob to be used for compiling other compilers.

You would be hard-pressed to find a more fertile breeding ground for Ken Thompson / Trusting Trust attacks. If every agent of the NSA (and whatever other agencies, including those of other countries, had access to the appropriate network traffic) somehow failed to capitalize on 6 years of opportunity to compromise an entire software ecosystem using only a basic MITM of unencrypted traffic, they deserve to be sacked. Whether such an attack actually occurred or not, this is a case study in carelessness and why bootstrappability is so important; discovering all this made me quite worried about having used a Mono version built from blobs previously, and has convinced me that, as time-wasting and tedious as this project has been, it is nevertheless probably an important one.

If you're going to write a self-hosting compiler, the least you can do is keep it self-hosting. Deciding to write a self-hosting compiler is a valid choice, of course, with its own merits and demerits, but there is something bitterly poetic about Mono starting out requiring specifically Microsoft's C# compiler in order to build (Mono did its initial bootstrapping using Microsoft's proprietary csc), achieving independence through self-hosting, being acquired by Microsoft, and thereafter coming crawling back to Microsoft's C# compiler once more before eventually dying.

The funny thing is that it's not even necessary. The dependencies on new C# features are all in Mono's standard library (which increasingly borrowed code from Microsoft's corefx library), not in Mono's compiler.

Even before roslyn-binaries, there was binary-reference-assemblies, which contained prebuilt "reference" blobs for the various versions of the standard libraries. These exist, I assume, precisely because of the library incompatibility problems regarding overloading that I mentioned earlier. While later versions of Mono included sources and a build system for producing these reference binaries, mono-4.9.0 and earlier did not. Mono's build system still demanded something to install, though, so I told it to use the real standard library of the input Mono version. When I did get to a Mono version that at least claimed to support regenerating the reference binaries, I found that it didn't work with mcs due to differences in which libraries had to be referenced, so I had to patch it to add a bunch of references determined through trial and error.

The xunit-binaries submodule was also added sometime before mono-5.1.0. This dependency makes it impossible to run the full test suite without binary blobs. Presumably for this reason, Debian elects to only run tests within the mono/mini/ and mono/tests/ subdirectories. For my part, I've disabled all tests except for those of mono-6.12.0, the final version, limited to the two aforementioned subdirectories. This is because it would take extra time for the builds, because several of the tests depend on binary blobs bundled into the Mono repository itself (which my thorough cleaning of all dlls and exes from the sources removes), because a large chunk of the tests depend on binary blobs in xunit-binaries in later versions, and because "expect some test failures" is part of the Mono documentation and I don't have the time to figure out for the Mono developers every reason why each of 17 versions of their test suite is broken.

The 5.0 series was when Microsoft acquired Mono, and it shows. You'll notice I needed to introduce pre- packages for various versions because in several cases a tagged release could not build the following tagged release. For that matter, they couldn't build the pre- package either, but it at least took fewer patches to get them working. The reason for this is that Mono added a dependency on Microsoft's corefx library source code, and it usually started using C# features well before mcs was able to compile them. Because of this, despite taking 8 versions to get from 1.2.6 to 4.9.0, it took another 8 versions to get through the 5.0 series, and 5 of them required nontrivial patching to massage the source into a form compilable by mcs.

Eventually I realized that the dependencies on new features were all coming from corefx, not from Mono's compiler. Consequently, the only reason for this particular bootstrap-hostile ordering of builds is that it happened to be the order the Mono devs committed things. So I just cherry-picked every commit I could find touching mcs/mcs (magit was quite useful for this) and applied it to 5.10.0 to produce what is essentially the 6.12.0 compiler, then used it to jump straight to building 6.12.0.

Use of this technique earlier on in the bootstrap process may be of interest to anyone looking to shorten the chain of packages.

My initial goal was to package dotnet, and I had tried to progress toward that from mono-4.9.0 for a period, but with no success. During that time, though, I did encounter a bug in Mono's xbuild condition parser, which I wrote a patch for, and included in mono-6.12.0.

I also discovered that xbuild would wrongly complain about missing references even when the proper assemblies were in MONO_PATH or MONO_GAC_PREFIX, because xbuild would erroneously only consider the path /gnu/store/...mono-6.12.0/lib/mono/gac when looking for global assembly caches, completely ignoring MONO_GAC_PREFIX. So I wrote a patch to fix that, and included it in mono-6.12.0.

Having witnessed how much nicer it is to package things that use rpath / runpath than things that use environment variables (like python) and therefore require constant wrapping of executables and use of propagated-inputs, I devised a patch that would extend Mono's per-assembly config files to support a <runpath> element. For example, if you have a file /tmp/dir2/test2.exe, and there is also a file /tmp/dir2/test2.exe.config, and its contents are

and it references test1.dll, it will first look for it at /tmp/dir1/test1.dll. Note that, of course, test1.dll still needs to be accessible to the compiler at compile-time through MONO_PATH or an explicitly-specified path passed on the mcs command line.

It is my hope that this feature will be of use to anybody interested in developing a build system.

Mono had several difficult points in bootstrapping and packaging, but at the end of the day it still met the basic description of a software package: well-defined environment-supplied inputs and sources, a user-supplied install prefix, and files installed under that prefix.

The dotnet world is an entirely different beast. The first step of most build systems I have encountered from that realm is downloading an entire toolchain, among other dependencies, as a binary blob. They heavily depend on the exact packages they specify being available exactly where they say to install them. There is no "install", there are no "install directories" to my knowledge. A build that doesn't contact nuget.org is an aberration. I am at a loss how to build these things, much less package them. I badly need help.

There are also some portability issues with the current bootstrap path. While Portable.NET can fall back to an interpreter written in C where LibJIT isn't supported, old versions of Mono have no such capability. Strictly speaking, there is some bitrotted code for an interpreter that used to work, but has stopped working by mono-1.2.6. It was left unmaintained until it was eventually removed in 2014, only to be revived in 2017. This poses a dilemma for anybody wanting to bootstrap Mono on a platform that wasn't supported by mono-1.2.6's JIT compiler. There are a number of possible ways to try resolving this, ranging from backporting the new interpreter, to fixing up the old one for every version prior to the new interpreter, to forward-porting the old compiler and class libraries to the new interpreter, etc.

The most interesting option, though, in my opinion, would be to port mcs to Portable.NET. This would achieve the intended portability, while also allowing individual builds to be much faster since we're only building mcs, not the runtime and class library each time. It would also allow us to make much bigger version jumps, since, as we discovered earlier, many of the new C# feature dependencies in Mono come from the class library rather than the compiler. Such a shortened bootstrap could also make the bootstrap path more appealing for other distributions to use instead of binaries.

"You wish now that our places had been exchanged. That I had died, and DotGNU had lived?"

"... Yes. I wish that."

Maintenance of Mono was recently transferred over to WineHQ. With that announcement this statement was placed at https://www.mono-project.com:

"We want to recognize that the Mono Project was the first .NET implementation on Android, iOS, Linux, and other operating systems. The Mono Project was a trailblazer for the .NET platform across many operating systems. It helped make cross-platform .NET a reality and enabled .NET in many new places and we appreciate the work of those who came before us."

I would like to clarify that, according to Miguel de Icaza himself, DotGNU "started working on the system about the same time". According to this DotGNU newsletter, Portable.NET began "in January 2001". While it's unclear exactly when Portable.NET reached various milestones, and the significance of the various milestones varies somewhat (for example, Mono probably does not care that Portable.NET also includes a Java and C compiler), I think that there is cause to dispute the claim that Mono was "the first" .NET implementation on Linux.

On a related note, if we haven't looked at the possibility of using Portable.NET in the Java bootstrap process, it may be worth visiting at some point.

Thank you to the DotGNU project, for the .NET implementation that made this bootstrap possible, Adam Faiz, for the initial packaging of it that let me jump straight in, the Mono project, for... Mono, and you, for your time.

GNU Parallel 20241222 ('Bashar') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  "Do this with gnu parallel" is the Copilot hack of the day
    -- Chase Clark @chasingmicrobes.bsky.social
 
New in this release:

• No new features. This is a candidate for a stable release.
• Bug fixes and man page updates.

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

• Give a demo at your local user group/team/colleagues
• Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
• Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
• Request or write a review for your favourite blog or magazine
• Request or build a package for your favourite distribution (if it is not already there)
• Invite me for your next conference

If you use programs that use GNU Parallel for research:

• Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

• (Have your company) donate to FSF https://my.fsf.org/donate/

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.