FLOSS Project Planets

PyPy has moved its canonical repo and issue tracker from https://foss.heptapod.net/pypy/pypy to https://github.com/pypy/pypy. Obviously, this means development will now be tracked in Git rather than Mercurial.

Motivation

We still feel Mercurial is a better version control system. The named branch model and user interface are superior. But

• foss.heptapod.net is not well indexed in google/bing/duckduckgo search, so people find it harder to search for issues in the project.

• Since Heptapod has tightened its spam control, we get reports that users create issues only to have them flagged as spam.

• Open Source has become synonymous with GitHub, and we are too small to change that.

• Much of the current development comes as a reaction to fixing issues. Tracking interlocking issues is easier if all the code is on the same platform.

• The FAQ presents two arguments against the move. Github notes solves much of point (1): the difficulty of discovering provenance of commits, although not entirely. But the main problem is point (2), it turns out that not moving to GitHub is an impediment to contribution and issue reporting.

• People who wish to continue to use Mercurial can use the same method below to push to GitHub.

• GitHub is more resource rich than foss.heptapod.net. We could add CI jobs to replace some of our aging buildbot infrastructure.

Method

The migration required two parts: migrating the code and then migrating the issues and merge requests.

Code migration 1: code and notes

I used a fork of git-remote-hg to create a local Git repo with all the changesets. Then I wanted to add a Git note to each commit with the branch it came from. So I prepared a file with two columns: the Git commit hash, and the corresponding branch from Mercurial. Mercurial can describe each commit in two ways: either the commit hash or by a number index. I used hg log to convert an index i to the Mercurial hash, and then git-hg-helper from git-remote-hg to convert the Mercurial hash to a Git hash:

$(cd pypy-git; git-hg-helper git-rev $(cd ../pypy-hg; hg log -r $i -T"{node}\n"))

Then I used hg log again to print the Mercurial branch for the index i:

$(cd pypy-hg; hg log -r $i -T'{branch}\n')

Putting these two together, I could loop over all the commits by their numerical index to prepare the file. Then I iterated over each line in the file, and added the Git note. Since the git note add command works on the current HEAD, I needed to checkout each commit in turn and then add the note:

git checkout -q <hash> && git notes --ref refs/notes/branch add -m branch:<branch>

I could then use git push --all to push to GitHub.

Code migration 2: prepare the branches

PyPy has almost 500 open branches. The code migration created all the branch HEADs, but git push --all did not push them. I needed to check them out and push each one. So I created a file with all the branch names

cd pypy-hg; hg branches | cut -f1 -d" " > branches.txt

and then push each one to the GitHub repo

while read branch; do git checkout branches/$branch && git push origin branches/$branch; done < branches.txt

Note that the branches were named branches/XXX by the migration, not branch/XXX. This confuses the merge request migration, more about that later.

Issue and merge request migration

I used the solution from node-gitlab-2-github which worked almost perfectly. It is important to do the conversion on a private repo otherwise every mention of a sucessfully mapped user name notifies the user about the transfer. This can be quite annoying for a repo the size of PyPy with 600 merge requests and over 4000 issues. Issues transfered without a problem: the script properly retained the issue numbers. However the script does not convert the Mercurial hashes to Git hashes, so the bare hashes in comments show up without a link to the commit. Merge requests are more of a problem:

• The Mercurial named branch "disappears" once it is merged, so a merge request to a merged branch does not find the target branch name in Git. The conversion creates an issue instead with the label gitlab merge request.
• For some reason, the branches created by git-remote-hg are called branches/XXX and not branch/XXX as expected by GitLab. This messes up the merge request/PR conversion. For some of the branches (open PRs and main target branches) I manually created additional branches without the es. The net result is that open merge requests became open PRs, merged merge requests became issues, and closed-not-merged merge requests were not migrated.

Layered conversions

PyPy already migrated once from Bitbucket to Heptapod. Many of the issues reflect the multiple transitions: they have lines like "Created originally on Bitbucket by XXX" from the first transition, and an additional line "In Heptapod" from this transition.

Credits

We would like to express our gratitude to the Octobus team who support Heptapod. The transition from Bitbucket was quite an effort, and they have generously hosted our developement since then. We wish them all the best, and still believe that Mercurial should have "won".

Next steps

While the repo at GitHub is live, there are still a few more things we need to do:

• Documentation needs an update for the new repo and the build automation from readthedocs must be adjusted.
• The wiki should be copied from Heptapod.
• buildbot.pypy.org should also look at the new repo. I hope the code is up to the task of interacting with a Git repo.
• speed.pypy.org tracks changes, it too needs to reference the new location
• To keep tracking branches with Git notes on new commits, I activated a github action by Julian to add a Git branch note to each commit. Please see the README there for directions on using Git notes.
• Some of the merge requests were not migrated. If someone wants to, they could migrate those once they figure out the branch naming problems.

Additionally, now is the time for all of you to prove the move is worthwhile:

• Star the repo, let others know how to find it,
• Help fix some of the open issues or file new ones,
• Take advantage of the more familiar workflow to get involved in the project,
• Suggest ways to improve the migration: are there things I missed or could have done better?

How will development change?

Heptapod did not allow personal forks, so we were generous with a commit bit to the main repo. Additionally, we (well, me) have been using a commit-directly-to-main workflow. We will now be adopting a more structured workflow. Please fork the repo and submit a pull request for any changes. We can now add some pre-merge CI to check that the PR at least passes the first stage of translation. The live and active branches will be:

• main: what was "default" in Mercurial, it is the Python2.7 interpreter and the base of the RPython interpreter,
• py3.9: the Python3.9 interpreter, which also includes all RPython changes from main. This is exactly like on Mercurial, and
• py3.10: the Python3.10 interpreter, which also includes all RPython changes from main and all bugfixes from py3.9. This is exactly like on Mercurial.

Working between the repos Finding commits

If you want to figure out how a Mercurial commit relates to a Git commit, you can use git-hg-helper. You run it in the Git repo. It takes the full long hash from one repo and gives you the corresponding hash of the other repo:

$ git-hg-helper git-rev d64027c4c2b903403ceeef2c301f5132454491df 4527e62ad94b0e940a5b0f9f20d29428672f93f7 $ git-hg-helper hg-rev 4527e62ad94b0e940a5b0f9f20d29428672f93f7 d64027c4c2b903403ceeef2c301f5132454491df Finding branches

Branches migrated from Mercurial will have a branches prefix, not branch. While GitLab uses branch for its prefix, the git-remote-hg script uses branches. New work should be in a PR targeting main, py3.9 or py3.10.

Thanks for helping to make PyPy better.

Matti

Three members of the Real Python team are joining us this week: Kate Finegan, Tappan Moore, and Philipp Acsany. We wanted to share a year-end wrap-up with tutorials, step-by-step projects, code conversations, and video courses that showcase what our team created this year.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Here&aposs a quick and easy one.

attrs supports frozen classes. Frozen classes are cool for a multitude of reasons, but they&aposre a tiny bit slower to instantiate compared to non-frozen classes, because they need to perform some additional checks and avoiding these checks in the __init__ uses a tiny bit of time. (We&aposre talking slotted classes here; which are awesome and the default in attrs nowadays.)

But there&aposs a way to avoid this overhead and make frozen classes be the exact same speed as ordinary classes. The only caveat is: you have to use type-checking.

Create a file somewhere in your code base and put this in it:

from functools import partial from typing import TYPE_CHECKING if TYPE_CHECKING: from attrs import frozen else: from attrs import define frozen = partial(define, unsafe_hash=True)

Now import frozen from this module and use it just like you&aposd use attrs.define or attrs.frozen. You&aposre done!

This technique should also work for dataclasses, with a slight adjustment left as an exercize to the reader.

The eagle-eyed reader might notice that we&aposre actually bamboozling the type-checker: your classes won&apost actually be frozen at runtime. The kicker is: they don&apost actually need to be.

As long as you&aposre running one on your codebase, the typechecker is the actual thing that&aposll prevent you from mutating your instances. The unsafe_hash=True will make the classes hashable, and it&aposs only unsafe if you mutate them after construction, which you won&apost. I guess you&aposll have to be careful when using a REPL or a different context where a typechecker might not hold sway, but I think that&aposs not too big of an issue.

If you&aposre still unconvinced, I&aposll leave you with two final thoughts: the memory in your computer is, ultimately, mutable. What makes immutable data structures in other languages immutable is just the amount of hoops you have to jump through to apply a mutation. This also demonstrates a basic technique statically-compiled languages use to be fast: move part of the work out of runtime into a separate, pre-runtime step. Which is exactly what we&aposve done here.

Happy New Year!

Special crossover episode of Python Bytes to wrap up 2023. Topics include: <br/> <br/> <ol> <ul><strong>Michael #1</strong>: <a href="https://hatch.pypa.io/latest/blog/2023/12/11/hatch-v180/">Hatch v1.8</a></ul> <ul><strong>Brian #2:</strong> <a href="https://svcs.hynek.me/en/stable/">svcs : A Flexible Service Locator for Python</a></ul> <ul><strong>Michael #3:</strong> <a href="https://discuss.python.org/t/steering-council-election-results-2024-term/40851">Steering Council 2024 Term Election Results</a></ul> <ul><strong>Brian #4:</strong> <a href="https://typethepipe.com/post/python-protocols-when-to-use">Python protocols. When to use them in your projects to abstract and decoupling</a></ul> <ul>Extras</ul><ul>Joke: <strong>Joke:</strong> <a href="https://mastodon.social/@tveskov/111289358585305218">The dream is dead?</a></ul></ol><br/> <strong>--- Episode sponsors ---</strong><br/> <a href='https://talkpython.fm/posit'>Posit</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>

Review: The Afterward, by E.K. Johnston

Publisher: Dutton Books Copyright: February 2019 Printing: 2020 ISBN: 0-7352-3190-7 Format: Kindle Pages: 339

The Afterward is a standalone young adult high fantasy with a substantial romance component. The title is not misspelled.

Sir Erris and her six companions, matching the number of the new gods, were successful in their quest for the godsgem. They defeated the Old God and destroyed Him forever, freeing King Dorrenta from his ensorcellment, and returned in triumph to Cadrium to live happily ever after. Or so the story goes.

Sir Erris and three of the companions are knights. Another companion is the best mage in the kingdom. Kalanthe Ironheart, who distracted the Old God at a critical moment and allowed Sir Erris to strike, is only an apprentice due to her age, but surely will become a great knight. And then there is Olsa Rhetsdaughter, the lowborn thief, now somewhat mockingly called Thief of the Realm for all the good that does her. The reward was enough for her to buy her freedom from the Thief's Court. It was not enough to pay for food after that, or enough for her to change her profession, and the Thief's Court no longer has any incentive to give her easy (or survivable) assignments.

Kalanthe is in a considerably better position, but she still needs a good marriage. Her reward paid off half of her debt, which broadens her options, but she's still a debt-knight, liable for the full cost of her training once she reaches the age of nineteen. She's mostly made her peace with the decisions she made given her family's modest means, but marriages of that type are usually for heirs, and Kalanthe is not looking forward to bearing a child. Or, for that matter, sleeping with a man.

Olsa and Kalanthe fell in love during the Quest. Given Kalanthe's debt and the way it must be paid, and her iron-willed determination to keep vows, neither of them expected their relationship to survive the end of the Quest. Both of them wish that it had.

The hook is that this novel picks up after the epic fantasy quest is over and everyone went home. This is not an entirely correct synopsis; chapters of The Afterward alternate between "After" and "Before" (and one chapter delightfully titled "More or less the exact moment of"), and by the end of the book we get much of the story of the Quest. It's not told from the perspective of the lead heroes, though; it's told by following Kalanthe and Olsa, who would be firmly relegated to supporting characters in a typical high fantasy. And it's largely told through the lens of their romance.

This is not the best fantasy novel I've read, but I had a fun time with it. I am now curious about the intended audience and marketing, though. It was published by a YA imprint, and both the ages of the main characters and the general theme of late teenagers trying to chart a course in an adult world match that niche. But it's also clearly intended for readers who have read enough epic fantasy quests that they will both be amused by the homage and not care that the story elides a lot of the typical details. Anyone who read David Eddings at an impressionable age will enjoy the way Johnston pokes gentle fun at The Belgariad (this book is dedicated to David and Leigh Eddings), but surely the typical reader of YA fantasy these days isn't also reading Eddings. I'm therefore not quite sure who this book was for, but apparently that group included me.

Johnston thankfully is not on board with the less savory parts of Eddings's writing, as you might have guessed from the sapphic romance. There is no obnoxious gender essentialism here, although there do appear to be gender roles that I never quite figured out. Knights are referred to as sir, but all of the knights in this story are women. Men still seem to run a lot of things (kingdoms, estates, mage colleges), but apart from the mage, everyone on the Quest was female, and there seems to be an expectation that women go out into the world and have adventures while men stay home. I'm not sure if there was an underlying system that escaped me, or if Johnston just mixed things up for the hell of it. (If the latter, I approve.)

This book does suffer a bit from addressing some current-day representation issues without managing to fold them naturally into the story or setting. One of the Quest knights is transgender, something that's revealed in a awkward couple of paragraphs and then never mentioned again. Two of the characters have a painfully earnest conversation about the word "bisexual," complete with a strained attempt at in-universe etymology. Racial diversity (Olsa is black, and Kalanthe is also not white) seemed to be handled a bit better, although I am not the reader to notice if the discussions of hair maintenance were similarly awkward. This is way better than no representation and default-white characters, to be clear, but it felt a bit shoehorned in at times and could have used some more polish.

These are quibbles, though. Olsa was the heart of the book for me, and is exactly the sort of character I like to read about. Kalanthe is pure stubborn paladin, but I liked her more and more as the story continued. She provides a good counterbalance to Olsa's natural chaos. I do wish Olsa had more opportunities to show her own competence (she's not a very good thief, she's just the thief that Sir Erris happened to know), but the climax of the story was satisfying. My main grumble is that I badly wanted to dwell on the happily-ever-after for at least another chapter, ideally two. Johnston was done with the story before I was.

The writing was serviceable but not great and there are some bits that I don't think would stand up to a strong poke, but the characters carried the story for me. Recommended if you'd like some sapphic romance and lightweight class analysis complicating your Eddings-style quest fantasy.

Rating: 7 out of 10

Recently I noticed a new tool called licenserecon written by Peter Blackman, and I helped get licenserecon into Debian. The purpose of licenserecon is to reconcile licenses from debian/copyright against the output from licensecheck, a tool written by Jonas Smedegaard. It assumes DEP5 copyright files. You run the tool in a directory that has a debian/ sub-directory, and its output when it notices mismatches (this is for resolv-wrapper):

# sudo apt install licenserecon jas@kaka:~/dpkg/resolv-wrapper$ lrc Parsing Source Tree .... Running licensecheck .... d/copyright | licensecheck BSD-3-Clauses | BSD-3-clause src/resolv_wrapper.c BSD-3-Clauses | BSD-3-clause tests/dns_srv.c BSD-3-Clauses | BSD-3-clause tests/test_dns_fake.c BSD-3-Clauses | BSD-3-clause tests/test_res_query_search.c BSD-3-Clauses | BSD-3-clause tests/torture.c BSD-3-Clauses | BSD-3-clause tests/torture.h jas@kaka:~/dpkg/resolv-wrapper$

Noticing one-character typos like this may not bring satisfaction except to the most obsessive-compulsive among us, however the tool has the potential of discovering more serious mistakes.

Using it manually once in a while may be useful, however I tend to forget QA steps that are not automated. Could we add this to the Salsa CI/CD pipeline? I recently proposed a merge request to add a wrap-and-sort job to the Salsa CI/CD pipeline (disabled by default) and learned how easy it was to extend it. I think licenserecon is still a bit rough on the edges, and I haven’t been able to successfully use it on any but the simplest packages yet. I wouldn’t want to suggest it is added to the normal Salsa CI/CD pipeline, even if disabled. If you maintain a Debian package on Salsa and wish to add a licenserecon job to your pipeline, I wrote licenserecon.yml for you.

The simplest way to use licenserecon.yml is to replace recipes/debian.yml@salsa-ci-team/pipeline as the Salsa CI/CD configuration file setting with debian/salsa-ci.yml@debian/licenserecon. If you use a debian/salsa-ci.yml file you may put something like this in it instead:

--- include: - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml - https://salsa.debian.org/debian/licenserecon/raw/main/debian/licenserecon.yml

Once you trigger the pipeline, this will result in a new job licenserecon that validates debian/copyright against licensecheck output on every build! I have added this to the libcpucycles package on Salsa and the pipeline contains a new job licenserecon whose output currently ends with:

$ cd ${WORKING_DIR}/${SOURCE_DIR} $ lrc Parsing Source Tree .... Running licensecheck .... No differences found Cleaning up project directory and file based variables

If upstream releases a new version with files not matching our debian/copyright file, we will detect that on the next Salsa build job rather than months later when somebody happens to run the tools manually or there is some license conflict.

Incidentally licenserecon is written in Pascal which brought back old memories with Turbo Pascal back in the MS-DOS days. Thanks Peter for licenserecon, and Jonas for licensecheck making this possible!

 I skim the Linkedin Python group and sometimes comment.

A few days there was a poll asking for the way to check if a key is in a Python dictionary and as I write, more than half of the 900+ respondents have chosen dict.get rather than key in dict, which is the correct answer!

When pushed, it seems they are relying on dict.get returning None if the key is not in the dict, but fail to see tha if the key being tested is in the dict, but has a value of None then their test fails. 

Here's some code:

# %%mydict1 = {'a': 1, 'b': 2}print(mydict1.get('c'))  # -> None
# %%
def key_in_dict1(key_, dict_: dict) -> bool:    "FAULTY IMPLEMENTATION"    return dict_.get(key_) != None
# This worksmydict1 = {'a': 1, 'b': 2}print(key_in_dict1('a', mydict1))  # -> Trueprint(key_in_dict1('x', mydict1))  # -> False# %%
# But adding a key of x with value None gives# the wrong resultmydict2 = {'a': 1, 'b': 2, 'x': None}print(key_in_dict1('x', mydict2))  # -> False
# %%
# The correct way is to use 'in'def key_in_dict2(key_, dict_: dict) -> bool:    "Pythonic IMPLEMENTATION"    return key_ in dict_
# Tests:print(key_in_dict2('a', mydict1))  # -> Trueprint(key_in_dict2('x', mydict1))  # -> False
# %%
# And now for keys with None as a value:print(key_in_dict2('x', mydict2))  # -> True

Ugly, code-golf solution using dict.get()You can alter the default value returned from dict.get if a key is not found. This allows one to use two calls of dict.get with different defaults  to be used to give a correct solution:
def key_in_dict3(key_, dict_: dict) -> bool:    "CODE-GOLF IMPLEMENTATION USING GETs"    default1 = None    default2 = ""    return default1 != default2 \        and ( dict_.get(key_, default1) != default1             or dict_.get(key_, default2) != default2)
# Tests:print(key_in_dict3('a', mydict1))  # -> Trueprint(key_in_dict3('x', mydict1))  # -> Falseprint(key_in_dict3('x', mydict2))  # -> True


Of course, don't use  key_in_dict3, use version 2, i.e. key in dict.
END.

It was 2013, and I was on a break from work between Christmas and New Year of 2013. I had been working at Linaro for well over a year, on the LAVA project. I was living and breathing automated testing infrastructure, mostly for testing low-level components such as kernels and bootloaders, on real hardware.

At this point I was also a Debian contributor for quite some years, and had become an official project members two years prior. Most of my involvement was in the Ruby team, where we were already consistently running upstream test suites during package builds.

During that break, I put these two contexts together, and came to the conclusion that Debian needed a dedicated service that would test the contents of the Debian archive. I was aware of the existance of autopkgtest, and started working on a very simple service that would later become Debian CI.

In January 2014, debci was initially announced on that month's Misc Developer News, and later uploaded to Debian. It's been continuously developed for the last 10 years, evolved from a single shell script running tests in a loop into a distributed system with 47 geographically-distributed machines as of writing this piece, became part of the official Debian release process gating migrations to testing, had 5 Summer of Code and Outrechy interns working on it, and processed beyond 40 million test runs.

In there years, Debian CI has received contributions from a lot of people, but I would like to give special credits to the following:

• Ian Jackson - created autopkgtest.
• Martin Pitt - was the maintainer of autopkgtest when Debian CI launched and helped a lot for some time.
• Paul Gevers - decided that he wanted Debian CI test runs to control testing migration. While at it, became a member of the Debian Release Team and the other half of the permanent Debian CI team together with me.
• Lucas Kanashiro - Google Summer of Code intern, 2014.
• Brandon Fairchild - Google Summer of Code intern, 2014.
• Candy Tsai - Outreachy intern, 2019.
• Pavit Kaur - Google Summer of Code intern, 2021
• Abiola Ajadi - Outreachy intern, December 2021-2022.

If you are planning a job in data engineering, then you should be well prepared for it. We have identified 30 data engineer interview questions that can help in your endeavor. During the interview, you can be asked questions from different related areas. So, we tried to cover these in this tutorial. 30+ Data Engineer […]

The post Top 30 Data Engineer Interview Questions with Answers appeared first on TechBeamers.

People who have visited any of the larger C++ conferences surely know Rainer Grimm, know his talks, workshops and books.

Rainer Grimm

Unfortunately, he has been diagnosed with ALS, a serious progressive nerve condition.

Since ALS research doesn’t get much attention or funding, Rainer started a fund raising campaign for funding ALS research with ALS-Ambulanz of the Charité and I Am ALS organization.

You can support my work on Patreon, or you can get my book Functional Programming in C++ at Manning if you're into that sort of thing. -->

Drupal is an exceptional open-source content management system (CMS) that has captivated the hearts of business organizations worldwide. Its incredible features, scalability, flexibility, and ability to effortlessly handle large and complex websites give it an edge over all its competitors.

Drupal development services comprise a wide range of themes, modules, and features, the CMS is a popular choice for multitudinous applications from personal blogs to enterprise-level websites. It has a standard version that works perfectly for small websites that require limited functionality and an enterprise version for large websites that need more functionality.