FLOSS Project Planets

Evgeni Golov: A String is not a String, and that's Groovy!

Planet Debian - Fri, 2021-11-19 09:16

Halloween is over, but I still have some nightmares to share with you, so sit down, take some hot chocolate and enjoy :)

When working with Jenkins, there is almost no way to avoid writing Groovy. Well, unless you only do old style jobs with shell scripts, but y'all know what I think about shell scripts…

Anyways, Eric have been rewriting the jobs responsible for building Debian packages for Foreman to pipelines (and thus Groovy).

Our build process for pull requests is rather simple:

  1. Setup sources - get the orig tarball and adjust changelog to have an unique version for pull requests
  2. Call pbuilder
  3. Upload the built package to a staging archive for testing

For merges, it's identical, minus the changelog adjustment.

And if there are multiple packages changed in one go, it runs each step in parallel for each package.

Now I've been doing mass changes to our plugin packages, to move them to a shared postinst helper instead of having the same code over and over in every package. This required changes to many packages and sometimes I'd end up building multiple at once. That should be fine, right?

Well, yeah, it did build fine, but the upload only happened for the last package. This felt super weird, especially as I was absolutely sure we did test this scenario (multiple packages in one PR) and it worked just fine…

So I went on a ride though the internals of the job, trying to understand why it didn't work.

This requires a tad more information about the way we handle packages for Foreman:

  • the archive is handled by freight
  • it has suites like buster, focal and plugins (that one is a tad special)
  • each suite has components that match Foreman releases, so 2.5, 3.0, 3.1, nightly etc
  • core packages (Foreman etc) are built for all supported distributions (right now: buster and focal)
  • plugin packages are built only once and can be used on every distribution

As generating the package index isn't exactly fast in freight, we tried not not run it too often. The idea was that when we build two packages for the same target (suite/version combination), we upload both at once and run import only once for both. That means that when we build Foreman for buster and focal, this results in two parallel builds and then two parallel uploads (as they end up in different suites). But if we build Foreman and Foreman Installer, we have four parallel builds, but only two parallel uploads, as we can batch upload Foreman and Installer per suite. Well, or so was the theory.

The Groovy code, that was supposed to do this looked roughly like this:

def packages_to_build = find_changed_packages() def repos = [:] packages_to_build.each { pkg -> suite = 'buster' component = '3.0' target = "${suite}-${component}" if (!repos.containsKey(target)) { repos[target] = [] } repos[target].add(pkg) } do_the_build(packages_to_build) do_the_upload(repos)

That's pretty straight forward, no? We create an empty Map, loop over a list of packages and add them to an entry in the map which we pre-create as empty if it doesn't exist.

Well, no, the resulting map always ended with only having one element in each target list. And this is also why our original tests always worked: we tested with a PR containing changes to Foreman and a plugin, and plugins go to this special target we have…

So I started playing with the code (https://groovyide.com/playground is really great for that!), trying to understand why the heck it erases previous data.

The first finding was that it just always ended up jumping into the "if map entry not found" branch, even though the map very clearly had the correct entry after the first package was added.

The second one was weird. I was trying to minimize the reproducer code (IMHO always a good idea) and switched target = "${suite}-${component}" to target = "lol". Two entries in the list, only one jump into the "map entry not found" branch. What?! 🧐

So this is clearly related to the fact that we're using String interpolation here. But hey, that's a totally normal thing to do, isn't it?!

Admittedly, at this point, I was lost. I knew what breaks, but not why.

Luckily, I knew exactly who to ask: Jens.

After a brief "well, that's interesting", Jens quickly found the source of our griefs: Double-quoted strings are plain java.lang.String if there’s no interpolated expression, but are groovy.lang.GString instances if interpolation is present.. And when we do repos[target] the GString target gets converted to a String, but when we use repos.containsKey() it remains a GString. This is because GStrings get converted to Strings, if the method wants one, but containsKey takes any Object while the repos[target] notation for some reason converts it. Maybe this is because using GString as Map keys should be avoided.

We can reproduce this with simpler code:

def map = [:] def something = "something" def key = "${something}" map[key] = 1 println key.getClass() map.keySet().each {println it.getClass() } map.keySet().each {println it.equals(key)} map.keySet().each {println it.equals(key as String)}

Which results in the following output:

class org.codehaus.groovy.runtime.GStringImpl class java.lang.String false true

With that knowledge, the fix was to just use the same repos[target] notation also for checking for existence — Groovy helpfully returns null which is false-y when it can't find an entry in a Map absent.

So yeah, a String is not always a String, and it'll bite you!

Categories: FLOSS Project Planets

Neil Williams: git worktrees

Planet Debian - Fri, 2021-11-19 08:26

A few scenarios have been problematic with git and I've now discovered git worktrees which help with each.

  • If you've wanted to compare multiple files in different branches of the same tree - without needing to commit on either side.
  • If you want to work on two (or more) versions of the same file at the same time, again without needing to commit.
  • You have a file or a bunch of files that aren't ready to be committed, even locally.
  • You are working on a development branch and an urgent fix is required on an old git tag.
  • You have a large git repository which is a burden to clone (or has complex submodules).

You could go to the trouble of making a new directory and re-cloning the same tree. However, a local commit in one tree is then not accessible to the other tree.

You could commit everything every time, but with a dirty tree, that involves sorting out the .gitignore rules as well. That could well be pointless with an experimental change.

Git worktrees allow multiple filesystems from a single git tree. Commits on any branch are visible from other branches, even when the commit was on a different worktree. This makes things like cherry-picking easy, without needing to push pointless changes or branches.

Branches on a worktree can be rebased as normal, with the benefit that commit hashes from other local changes are available for reference and cherry-picks.

I'm sure git worktrees are not new. However, I've only started using them recently and others have asked about how the worktree operates.

Creating a new tree can be done with a new or existing branch. To make it easier, set the new directory at the same time, usually in ../

New branch (branched from the current branch):

git worktree add -b branch_name ../branch_name

Existing branch - note, slightly different syntax here, specify the commit-ish last (branch name, tag or hash):

git worktree add ../branch_name branch_name git worktree list /home/neil/Documents/testing/testrepo 0612677 [master] /home/neil/Documents/testing/testtree d38f5a3 [testtree]

Use git worktree remove <name> to drop the entire directory for that tree and the git tracking.

I'm using this for work on the Debian Security Tracker. I have two local branches and having two worktrees allows me to have three terminals open, using the same files and the same git repository.

One to run make serve and update the local SQLite database. One to access master to run git pull One to make local changes without risking collisions on master.

git add data/CVE/list git commit # pre commit hook runs here git log -n 1 # copy the hash # switch to master terminal git pull git cherry-pick <HASH> git push # switch to server terminal git rebase master # no git pull or fetch, it's all local make # switch back to changes terminal git rebase master

Sadly, one area where this isn't as easy is with importing a new DSC into Salsa with git-buildpackage as that uses several branches at the same time. It would be possible but you'll need to have a separate upstream and possibly pristine-tar branches and supply the relevant options. Possibly something git-buildpackage to adopt - it is common to need to make changes to the packaging with a new upstream release & a lot of those changes are currently done outside git.

For the rest of the support, see git worktree (1)

Categories: FLOSS Project Planets

Lucas Cimon: Hacktoberfest on fpdf2 &amp; v2.4.6

Planet Python - Fri, 2021-11-19 08:10

Last month, I realized late that October was hacktoberfest month!

This online event is a month-long celebration (October 1-31) of open source software run in partnership with different software companies, with a focus on encouraging contributions to open source projects.

While I participated in the 2019 edition as a contributor …

Categories: FLOSS Project Planets

Web Review, Week 2021-46

Planet KDE - Fri, 2021-11-19 07:01

Let’s go for my web review for the week 2021-46.

The Inner Osborne Effect

Tags: management, business, product-management

Interesting idea… Until that article, I didn’t fully realize the impact of the Osborne Effect inside of a company and not towards its customers.


Writing great software isn’t all about the software you write – Letters To A New Developer

Tags: tech, craftsmanship, product-management, empathy

Very good advises emphasizing empathy and holistic view of products.


It’s Now Possible To Sign Arbitrary Data With Your SSH Keys

Tags: tech, cryptography, ssh

Little known but potentially very useful SSH feature.


Highlights from Git 2.34 | The GitHub Blog

Tags: tech, git

Lots of good stuff in this release. Particularly looking forward to the new default merge strategy and the ability to sign commits using SSH keys.


String formatting comparison | Pydon’t 🐍

Tags: tech, python

Good summary about string formatting in Python. Clearly helps to decide between f-strings and the fomat function.


Bye for now!

Categories: FLOSS Project Planets

Real Python: The Real Python Podcast – Episode #87: Building a Content Aggregator and Working With RSS in Python

Planet Python - Fri, 2021-11-19 07:00

Have you wanted to work with RSS feeds in Python? Maybe you're looking for a new project to build for your portfolio that uses Django, unit tests, and custom commands. This week on the show, we have Real Python author Ricky White to talk about his recent step-by-step project titled, "Build a Content Aggregator in Python."

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Bits from Debian: New Debian Developers and Maintainers (September and October 2021)

Planet Debian - Fri, 2021-11-19 07:00

The following contributors got their Debian Developer accounts in the last two months:

  • Bastian Germann (bage)
  • Gürkan Myczko (tar)

The following contributors were added as Debian Maintainers in the last two months:

  • Clay Stan
  • Daniel Milde
  • David da Silva Polverari
  • Sunday Cletus Nkwuda
  • Ma Aiguo
  • Sakirnth Nagarasa


Categories: FLOSS Project Planets

Looking for Qt Champions - 2021!

Planet KDE - Fri, 2021-11-19 06:23

Who do you think should be a Qt Champion? Nominate the champions you know right now!

Categories: FLOSS Project Planets

KDE Floating Panels: ALMOST THERE!

Planet KDE - Fri, 2021-11-19 06:11
💸💸 Help me contribute to KDE and do these videos: 💸💸 Patreon: https://www.patreon.com/niccolove Youtube: https://www.youtube.com/channel/UCONH73CdRXUjlh3-DdLGCPw/join Paypal: https://paypal.me/niccolove Stay in the loop: https://t.me/veggeroblog My website is https://niccolo.venerandi.com and if you want to contact me, my telegram handle is [at] veggero.
Categories: FLOSS Project Planets

Agiledrop.com Blog: Decoupling Drupal: React & Vue

Planet Drupal - Fri, 2021-11-19 05:35

In this article, we take a look at decoupling Drupal with React or Vue, going through use cases, benefits and project examples.

Categories: FLOSS Project Planets

Mike Gabriel: Improbability of a million, lintian thinks...

Planet Debian - Fri, 2021-11-19 02:08

An interesting mindset overcome by reality...

Also, lintian does not differentiate between between 100.000 and 1.000.000.

W: ayatana-indicator-display: improbable-bug-number-in-closes 1000143 N: N: The most recent changelog closes a low-numbered bug number. While this is distantly possible, it's more likely a typo or N: a placeholder value that mistakenly wasn't filled in. N: N: Visibility: warning N: Show-Always: no N: Check: debian/changelog N: N:



Categories: FLOSS Project Planets

Dirk Eddelbuettel: RcppArmadillo on CRAN: Bugfix, New Features

Planet Debian - Thu, 2021-11-18 19:12

Armadillo is a powerful and expressive C++ template library for linear algebra aiming towards a good balance between speed and ease of use with a syntax deliberately close to a Matlab. RcppArmadillo integrates this library with the R environment and language–and is widely used by (currently) 928 other packages on CRAN.

I somehow missed to blog and tweet about the recent release based on the Armadillo 10.7.3 upstream release. Conrad is in “long-term support mode”, and 10.7.* is meant to provide fixes and stability relative to the most recent release which we did on September 30. We did actually find a regression when checking reverse-dependencies requiring an upstream move to 10.7.3. At the same time, we folded pull request #352 in. It addresses an old bug of ours where Armadillo fields types were not converted correctly in all dimensions.

So no we do have on CRAN, as well as with the (opt-in) field) fixes on the drat repo) repo. As the change perturbs a few existing packages, it is opt-in for now. We will likely aim at a proper deprecation of the old behaviour and give packages time to adjust. Stay tuned.

With that, big thanks to Jonathan Berrisch for filing issue #351 and basically addressing it in pull request #352. Very nice work, and I basically just wrapped a few more tests around it.

The full set of changes (since the last CRAN release follows.

Changes in RcppArmadillo version (2021-11-18)
  • Correct dimensions setting in import/export of arma::field types, protected by #define (Jonathan Berrisch in #352 fixing #351)

  • Add unit tests for fields both with and without new #define (Dirk)

Changes in RcppArmadillo version (2021-11-04)
  • Upgraded to Armadillo release 10.7.3

    • fix regression in alias handling by fliplr(), flipud(), reverse()
Changes in RcppArmadillo version (2021-11-02)
  • Upgraded to Armadillo release 10.7.2

    • more robust handling of diagonal matrices by pinv()
Changes in RcppArmadillo version (2021-10-08)
  • Upgraded to Armadillo release 10.7.1

    • fix regression in interactions between dense matrix subviews and sparse matrices

Courtesy of my CRANberries, there is a diffstat report relative to previous release. More detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Reproducible Builds (diffoscope): diffoscope 193 released

Planet Debian - Thu, 2021-11-18 19:00

The diffoscope maintainers are pleased to announce the release of diffoscope version 193. This version includes the following changes:

[ Chris Lamb ] * Don't duplicate file lists at each directory level. (Closes: #989192, reproducible-builds/diffoscope#263) * When pretty-printing JSON, mark the difference as such, additionally avoiding including the full path. (Closes: reproducible-builds/diffoscope#205) * Codebase improvements: - Update a bunch of %-style string interpolations into f-strings or str.format. - Import itertools top-level directly. - Drop some unused imports. - Use isinstance(...) over type(...) == - Avoid aliasing variables if we aren't going to use them. [ Brandon Maier ] * Fix missing diff output on large diffs. [ Mattia Rizzolo ] * Ignore a Python warning coming from a dependent library (triggered by supporting Python 3.10) * Document that support both Python 3.9 and 3.10.

You find out more by visiting the project homepage.

Categories: FLOSS Project Planets

"Mathspp Pydon'ts": String formatting comparison | Pydon't 🐍

Planet Python - Thu, 2021-11-18 18:00

This article compares the three main string formatting methods in Python and suggests which methods to use in each situation.

(If you are new here and have no idea what a Pydon't is, you may want to read the Pydon't Manifesto.)


The Zen of Python says that

“There should be one – and preferably only one – obvious way to do it.”

And yet, there are three main ways of doing string formatting in Python. This Pydon't will settle the score, comparing these three methods and helping you decide which one is the obvious one to use in each situation.

In this Pydon't, you will:

  • learn about the old C-style formatting with %;
  • learn about the string method .format;
  • learn about the Python 3.6+ feature of literal string interpolation and f-strings;
  • understand the key differences between each type of string formatting; and
  • see where each type of string formatting really shines.

You can now get your free copy of the ebook “Pydon'ts – Write beautiful Python code” on Gumroad.

String formatting rationale

Let's pretend, for a second, that Python had zero ways of doing string formatting.

Now, I have a task for you: write a function that accepts a programming language name and returns a string saying that said programming language rocks. Can you do it? Again, without any string formatting whatsoever!

Here is a possible solution:

def language_rocks(language): return language + " rocks!" # --- >>> language_rocks("Python") 'Python rocks!'

Great job!

Now, write a function that accepts a programming language name and its (estimated) number of users, and returns a string saying something along the lines of “ rocks! Did you know that has around users?”.

Can you do it? Recall that you are not supposed to use any string formatting facilities, whatsoever!

Here is a possible solution:

def language_info(language, users_estimate): return ( language + " rocks! Did you know that " + language + " has around " + str(users_estimate) + " users?!" ) # --- >>> language_info("Python", 10) 'Python rocks! Did you know that Python has around 10 users?!'

Notice how that escalated quite quickly: the purpose of our function is still very simple, and yet we have a bunch of string concatenations happening all over the place, just because we have some pieces of information that we want to merge into the string.

This is what string formatting is for: it's meant to make your life easier when you need to put information inside strings.

Three string formatting methods

Now that we've established that string formatting is useful, let's take a look at the three main ways of doing string formatting in Python.

First, here is how you would refactor the function above:

# Using C-style string formatting: def language_info_cstyle(language, users_estimate): return ( "%s rocks! Did you know that %s has around %d users?!" % (language, language, users_estimate) ) # Using the Python 3 `.format` method from strings: def language_info_format(language, users_estimate): return "{} rocks! Did you know that {} has around {}...
Categories: FLOSS Project Planets

Evennia: The Evennia blog has moved to evennia.com!

Planet Python - Thu, 2021-11-18 17:53

This dev blog has moved! All past and future posts will now be found here instead on evennia.com. The linked post discusses the move in more detail, including the little custom blog platform I wrote for it.

The old posts here on blogspot/bloggly will remain but won't be updated anymore.



Categories: FLOSS Project Planets

Luca Saiu: Global variable initialisation in C++

GNU Planet! - Thu, 2021-11-18 16:00
Today Volker Birk (https://fdik.org/) and I were speaking over lunch about object initialisation in C++ and about how weakly defined a program entry point is, because of objects with static storage duration. Volker wrote a short program whose output changes after reversing the order of two variable definitions, both out of a ‘main’ function whose entire body was ‘return 0;’. He discussed it in German (https://blog.fdik.org/2021-11/s1637238415), on his blog (https://blog.fdik.org). I was more annoyed by the fact that initialisation order is not guaranteed to respect functional dependency across compilation units. Here is my test case, where GCC and the GNU ... [Read more]
Categories: FLOSS Project Planets

PyCharm: PyCharm 2021.3 Release Candidate Is Out

Planet Python - Thu, 2021-11-18 08:29

PyCharm’s 2021.3 major release is right around the corner, and now the PyCharm team is fine-tuning the new features and fixing important bugs.

As we approach the end of our EAP – Early Access Program, we’d like to thank everyone who joined it, tested the new features, commented on Twitter, submitted bug reports, and etc.

Your contribution always makes all the difference!

Download PyCharm 2021.3 RC

Previously highlighted features:
  • Poetry Support
  • New FastAPI Project Type
  • New Jupyter Notebook Experience
  • Remote Development Support (Beta)

Read our previous EAP blog posts for more information on highlighted features.

  • This build requires an active JetBrains subscription.
  • If you find any bugs while exploring this release candidate, please submit them to the PyCharm issue tracker.
  • For the full list of issues solved in this build please read the release notes.

The PyCharm Team

Categories: FLOSS Project Planets

Massimo Stella talks Kdenlive

Planet KDE - Thu, 2021-11-18 06:42
Massimo Stella of Kdenlive talks about KDE's 25th Anniversary, how KDE’s video editor came to be, where it is now and what we can look forward to in the future of open source film-making.
Categories: FLOSS Project Planets