Feeds

Real Python: Python's Magic Methods: Leverage Their Power in Your Classes

Planet Python - Wed, 2024-01-03 09:00

As a Python developer who wants to harness the power of object-oriented programming, you’ll love to learn how to customize your classes using special methods, also known as magic methods or dunder methods. A special method is a method whose name starts and ends with a double underscore. These methods have special meanings for Python.

Python automatically calls magic methods as a response to certain operations, such as instantiation, sequence indexing, attribute managing, and much more. Magic methods support core object-oriented features in Python, so learning about them is fundamental for you as a Python programmer.

In this tutorial, you’ll:

  • Learn what Python’s special or magic methods are
  • Understand the magic behind magic methods in Python
  • Customize different behaviors of your custom classes with special methods

To get the most out of this tutorial, you should be familiar with general Python programming. More importantly, you should know the basics of object-oriented programming and classes in Python.

Get Your Code: Click here to download the free sample code that shows you how to use Python’s magic methods in your classes.

Getting to Know Python’s Magic or Special Methods

In Python, special methods are also called magic methods, or dunder methods. This latter terminology, dunder, refers to a particular naming convention that Python uses to name its special methods and attributes. The convention is to use double leading and trailing underscores in the name at hand, so it looks like .__method__().

Note: In this tutorial, you’ll find the terms special methods, dunder methods, and magic methods used interchangeably.

The double underscores flag these methods as core to some Python features. They help avoid name collisions with your own methods and attributes. Some popular and well-known magic methods include the following:

Special Method Description .__init__() Provides an initializer in Python classes .__str__() and .__repr__() Provide string representations for objects .__call__() Makes the instances of a class callable .__len__() Supports the len() function

This is just a tiny sample of all the special methods that Python has. All these methods support specific features that are core to Python and its object-oriented infrastructure.

Note: For the complete list of magic methods, refer to the special method section on the data model page of Python’s official documentation.

The Python documentation organizes the methods into several distinct groups:

Take a look at the documentation for more details on how the methods work and how to use them according to your specific needs.

Here’s how the Python documentation defines the term special methods:

A method that is called implicitly by Python to execute a certain operation on a type, such as addition. Such methods have names starting and ending with double underscores. (Source)

There’s an important detail to highlight in this definition. Python implicitly calls special methods to execute certain operations in your code. For example, when you run the addition 5 + 2 in a REPL session, Python internally runs the following code under the hood:

Python >>> (5).__add__(2) 7 Copied!

The .__add__() special method of integer numbers supports the addition that you typically run as 5 + 2.

Reading between the lines, you’ll realize that even though you can directly call special methods, they’re not intended for direct use. You shouldn’t call them directly in your code. Instead, you should rely on Python to call them automatically in response to a given operation.

Note: Even though special methods are also called magic methods, some people in the Python community may not like this latter terminology. The only magic around these methods is that Python calls them implicitly under the hood. So, the official documentation refers to them as special methods instead.

Magic methods exist for many purposes. All the available magic methods support built-in features and play specific roles in the language. For example, built-in types such as lists, strings, and dictionaries implement most of their core functionality using magic methods. In your custom classes, you can use magic methods to make callable objects, define how objects are compared, tweak how you create objects, and more.

Note that because magic methods have special meaning for Python itself, you should avoid naming custom methods using leading and trailing double underscores. Your custom method won’t trigger any Python action if its name doesn’t match any official special method names, but it’ll certainly confuse other programmers. New dunder names may also be introduced in future versions of Python.

Magic methods are core to Python’s data model and are a fundamental part of object-oriented programming in Python. In the following sections, you’ll learn about some of the most commonly used special methods. They’ll help you write better object-oriented code in your day-to-day programming adventure.

Controlling the Object Creation Process

When creating custom classes in Python, probably the first and most common method that you implement is .__init__(). This method works as an initializer because it allows you to provide initial values to any instance attributes that you define in your classes.

Read the full article at https://realpython.com/python-magic-methods/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

PyCharm: How To Learn Django: A Comprehensive Guide for Beginners

Planet Python - Wed, 2024-01-03 05:56
Learning Django can be an exciting journey for anyone looking to develop web applications, but it can be intimidating at first. In this article, we’ll provide you with a comprehensive guide on how to learn Django effectively. We’ll explore the prerequisites, the time it takes to become proficient, and various resources to help you master […]
Categories: FLOSS Project Planets

CTI Digital: Drupal Core Major Upgrades

Planet Drupal - Wed, 2024-01-03 05:41

Over the past 12 months, our teams have completed numerous Drupal upgrades. We would like to share our experiences and knowledge with anyone who has yet to undergo this process to help make it smoother for you.

Categories: FLOSS Project Planets

Brett Cannon: An experimental pip subcommand for the Python Launcher for Unix

Planet Python - Wed, 2024-01-03 00:49

There are a couple of things I always want to be true when I install Python packages for a project:

  1. I have a virtual environment
  2. Pip is up-to-date

For virtual environments, you would like them to be created as fast as possible and (usually) with the newest version of Python. For keeping pip up-to-date, it would be nice to not have to do that for every single virtual environment you have.

To help make all of this true for myself, I created an experimental Python Launcher for Unix "subcommand": py-pip. The CLI app does the following:

  1. Makes sure there is a globally cached copy of pip, and updates it if necessary
  2. Uses the Python Launcher for Unix to create a virtual environment where it finds a pyproject.toml file
  3. Runs pip using the virtual environment&aposs interpreter

This is all done via a py-pip.pyz file (which you can rename to just py-pip if you want). The py-pip.pyz file available from a release of py-pip can be made executable (e.g. chmod a+x py-pip.pyz). The shebang of the file is already set to #!/usr/bin/env py so it&aposs ready to use the newest version of Python you have installed. Stick that on your PATH and you can then use that instead of py -m pip to run pip itself.

To keep pip up-to-date, the easiest way to do that is to have only a single copy of pip to worry about. Thanks to the pip team releasing a self-contained pip.pyz along with pip always working with supported, it means if we just cache a copy of pip.pyz and keep that up-to-date then we can have that one copy to worry about.

Having a single copy of pip also means we don&apost need to install pip for each virtual environment. That lets us use microvenv and skip the overhead of installing pip in each virtual environment.

Now, this is an experiment. Much like the Python Launcher for Unix, py-pip is somewhat optimized for my own workflow. I am also keeping an eye on PEP 723 and PEP 735 as a way to only install packages that have been written down somewhere instead of ever installing a package à la carte as I think that&aposs a better practice to follow and might actually trump all of this. But since I have seen others have frustration from both forgetting the virtual environment and having to keep pip up-to-date, I decided to open source the code.

Categories: FLOSS Project Planets

FSF Blogs: LibrePlanet 2024: May 4 and 5, Wentworth Institute of Technology, Boston, MA

GNU Planet! - Tue, 2024-01-02 20:00
The dates and location of LibrePlanet 2024 have been announced!
Categories: FLOSS Project Planets

LibrePlanet 2024: May 4 and 5, Wentworth Institute of Technology, Boston, MA

FSF Blogs - Tue, 2024-01-02 20:00
The dates and location of LibrePlanet 2024 have been announced!
Categories: FLOSS Project Planets

John Goerzen: Consider Security First

Planet Debian - Tue, 2024-01-02 19:38

I write this in the context of my decision to ditch Raspberry Pi OS and move everything I possibly can, including my Raspberry Pi devices, to Debian. I will write about that later.

But for now, I wanted to comment on something I think is often overlooked and misunderstood by people considering distributions or operating systems: the huge importance of getting security updates in an automated and easy way.

Background

Let’s assume that these statements are true, which I think are well-supported by available evidence:

  1. Every computer system (OS plus applications) that can do useful modern work has security vulnerabilities, some of which are unknown at any given point in time;

  2. During the lifetime of that computer system, some of these vulnerabilities will be discovered. For a (hopefully large) subset of those vulnerabilities, timely patches will become available.

Now then, it follows that applying those timely patches is a critical part of having a system that it as secure as possible. Of course, you have to do other things as well – good passwords, secure practices, etc – but, fundamentally, if your system lacks patches for known vulnerabilities, you’ve already lost at the security ballgame.

How to stay patched

There is something of a continuum of how you might patch your system. It runs roughly like this, from best to worst:

  1. All components are kept up-to-date automatically, with no intervention from the user/operator

  2. The operator is automatically alerted to necessary patches, and they can be easily installed with minimal intervention

  3. The operator is automatically alerted to necessary patches, but they require significant effort to apply

  4. The operator has no way to detect vulnerabilities or necessary patches

It should be obvious that the first situation is ideal. Every other situation relies on the timeliness of human action to keep up-to-date with security patches. This is a fallible situation; humans are busy, take trips, dismiss alerts, miss alerts, etc. That said, it is rare to find any system living truly all the way in that scenario, as you’ll see.

What is “your system”?

A critical point here is: what is “your system”? It includes:

  • Your kernel
  • Your base operating system
  • Your applications
  • All the libraries needed to run all of the above

Some OSs, such as Debian, make little or no distinction between the base OS and the applications. Others, such as many BSDs, have a distinction there. And in some cases, people will compile or install applications outside of any OS mechanism. (It must be stressed that by doing so, you are taking the responsibility of patching them on your own shoulders.)

How do common systems stack up?
  • Debian, with its support for unattended-upgrades, needrestart, debian-security-support, and such, is largely category 1. It can automatically apply security patches, in most cases can restart the necessary services for the patch to take effect, and will alert you when some processes or the system must be manually restarted for a patch to take effect (for instance, a kernel update). Those cases requiring manual intervention are category 2. The debian-security-support package will even warn you of gaps in the system. You can also use debsecan to scan for known vulnerabilities on a given installation.

  • FreeBSD has no way to automatically install security patches for things in the packages collection. As with many rolling-release systems, you can’t automate the installation of these security patches with FreeBSD because it is not safe to blindly update packages. It’s not safe to blindly update packages because they may bring along more than just security patches: they may represent major upgrades that introduce incompatibilities, etc. Unlike Debian’s practice of backporting fixes and thus producing narrowly-tailored patches, forcing upgrades to newer versions precludes a “minimal intervention” install. Therefore, rolling release systems are category 3.

  • Things such as Snap, Flatpak, AppImage, Docker containers, Electron apps, and third-party binaries often contain embedded libraries and such for which you have no easy visibility into their status. For instance, if there was a bug in libpng, would you know how many of your containers had a vulnerability? These systems are category 4 – you don’t even know if you’re vulnerable. It’s for this reason that my Debian-based Docker containers apply security patches before starting processes, and also run unattended-upgrades and friends.

The pernicious library problem

As mentioned in my last category above, hidden vulnerabilities can be a big problem. I’ve been writing about this for years. Back in 2017, I wrote an article focused on Docker containers, but which applies to the other systems like Snap and so forth. I cited a study back then that “Over 80% of the :latest versions of official images contained at least one high severity vulnerability.” The situation is no better now. In December 2023, it was reported that, two years after the critical Log4Shell vulnerability, 25% of apps were still vulnerable to it. Also, only 21% of developers ever update third-party libraries after introducing them into their projects.

Clearly, you can’t rely on these images with embedded libraries to be secure. And since they are black box, they are difficult to audit.

Debian’s policy of always splitting libraries out from packages is hugely beneficial; it allows finegrained analysis of not just vulnerabilities, but also the dependency graph. If there’s a vulnerability in libpng, you have one place to patch it and you also know exactly what components of your system use it.

If you use snaps, or AppImages, you can’t know if they contain a deeply embedded vulnerability, nor could you patch it yourself if you even knew. You are at the mercy of upstream detecting and remedying the problem – a dicey situation at best.

Who makes the patches?

Fundamentally, humans produce security patches. Often, but not always, patches originate with the authors of a program and then are integrated into distribution packages. It should be noted that every security team has finite resources; there will always be some CVEs that aren’t patched in a given system for various reasons; perhaps they are not exploitable, or are too low-impact, or have better mitigations than patches.

Debian has an excellent security team; they manage the process of integrating patches into Debian, produce Debian Security Advisories, maintain the Debian Security Tracker (which maintains cross-references with the CVE database), etc.

Some distributions don’t have this infrastructure. For instance, I was unable to find this kind of tracker for Devuan or Raspberry Pi OS. In contrast, Ubuntu and Arch Linux both seem to have active security teams with trackers and advisories.

Implications for Raspberry Pi OS and others

As I mentioned above, I’m transitioning my Pi devices off Raspberry Pi OS (Raspbian). Security is one reason. Although Raspbian is a fork of Debian, and you can install packages like unattended-upgrades on it, they don’t work right because they use the Debian infrastructure, and Raspbian hasn’t modified them to use their own infrastructure. I don’t see any Raspberry Pi OS security advisories, trackers, etc. In short, they lack the infrastructure to support those Debian tools anyhow.

Not only that, but Raspbian lags behind Debian in both new releases and new security patches, sometimes by days or weeks.

A future post will include instructions for migrating Raspberry Pis to Debian.

Categories: FLOSS Project Planets

Jacob Adams: Fixing My Shell

Planet Debian - Tue, 2024-01-02 19:00

For an embarassingly long time, my shell has unnecessarily tried to initialize a console font in every kind of interactive terminal.

This leaves the following error message in my terminal:

Couldn't get a file descriptor referring to the console.

It even shows up twice when running tmux!

Clearly I’ve done something horrible to my configuration, and now I’ve got to clean it up.

How does Shell Initialization Work?

The precise files a shell reads at start-up is somewhat complex, and defined by this excellent chart 1:

For the purposes of what I’m trying to fix, there are two paths that matter.

  • Interactive login shell startup
  • Interactive non-login shell startup

As you can see from the above, trying to distinguish these two paths in bash is an absolute mess. zsh, in contrast, is much cleaner and allows for a clear distinction between these two cases, with login shell configuration files as a superset of configuration files used by non-login shells.

How did we get here?

I keep my configuration files in a config repository.

Some time ago I got quite frustrated at this whole shell initialization thing, and just linked everything together in one profile file:

.mkshrc -> config/profile .profile -> config/profile

This appears to have created this mess.

Move to ZSH

I’ve wanted to move to zsh for a while, and took this opportunity to do so. So my new configuration files are .zprofile and .zshrc instead of .mkshrc and .profile (though I’m going to retain those symlinks to allow my old configurations to continue working).

mksh is a nice simple shell, but using zsh here allows for more consistency between my home and $WORK environments, and will allow a lot more powerful extensions.

Updating my Prompt

ZSH prompts use a totally different configuration via variable expansion. However, it also uses the PROMPT variable, so I set that to the needed values for zsh.

There’s an excellent ZSH prompt generator at https://zsh-prompt-generator.site that I used to get these variables, though I’m sure they’re in the zsh documentation somewhere as well.

I wanted a simple prompt with user (%n), host (%m), and path (%d). I also wanted a % at the end to distinguish this from other shells.

PROMPT="%n@%m%d%% " Fixing mksh prompts

This worked but surprisingly mksh also looks at PROMPT, leaving my mksh prompt as the literal prompt string without expansion.

Fixing this requires setting up a proper shrc and linking it to .mkshrc and .zshrc.

I chose to move my existing aliases script to this file, as it also broke in non-login shells when moved to profile.

Within this new shrc file we can check what shell we’re running via $0:

if [ "$0" = "/bin/zsh" ] || [ "$0" = "zsh" ] || [ "$0" = "-zsh" ]

I chose to add plain zsh here in case I run it manually for whatever reason. I also added -zsh to support tmux as that’s what it presents as $0. This also means you’ll need to be careful to quote $0 or you’ll get fun shell errors.

There’s probably a better way to do this, but I couldn’t find something that was compatible with POSIX shell, which is what most of this has to be written in to be compatible with mksh and zsh2.

We can then setup different prompts for each:

if [ "$0" = "/bin/zsh" ] || [ "$0" = "zsh" ] || [ "$0" = "-zsh" ] then PROMPT="%n@%m%d%% " else # Borrowed from # http://www.unixmantra.com/2013/05/setting-custom-prompt-in-ksh.html PS1='$(id -un)@$(hostname -s)$PWD$ ' fi Setting Console Font in a Better Way

I’ve been setting console font via setfont in my .profile for a while. I’m not sure where I picked this up, but it’s not the right way. I even tried to only run this in a console with -t but that only checks that output is a terminal, not specifically a console.

if [ -t 1 ] then setfont /usr/share/consolefonts/Lat15-Terminus20x10.psf.gz fi

This also only runs once the console is logged into, instead of initializing it on boot. The correct way to set this up, on Debian-based systems, is reconfiguring console-setup like so:

dpkg-reconfigure console-setup

From there you get a menu of encoding, character set, font, and then font size to configure for your consoles.

VIM mode

To enable VIM mode for ZSH, you simply need to set:

bindkeys -v

This allows you to edit your shell commands with basic VIM keybinds.

Getting back Ctrl + Left Arrow and Ctrl + Right Arrow

Moving around one word at a time with Ctrl and the arrow keys is broken by vim mode unfortunately, so we’ll need to re-enable it:

bindkey "^[[1;5C" forward-word bindkey "^[[1;5D" backward-word Better History Search

But of course we’re now using zsh so we can do better than just the same configuration as we had before.

There’s an excellent substring history search plugin that we can just source without a plugin manager3

source $HOME/config/zsh-history-substring-search.zsh # Keys are weird, should be ^[[ but it's not bindkey '^[[A' history-substring-search-up bindkey '^[OA' history-substring-search-up bindkey '^[[B' history-substring-search-down bindkey '^[OB' history-substring-search-down

For some reason my system uses ^[OA and ^[OB as up and down keys. It seems ^[[A and ^[[B are the defaults, so I’ve left them in, but I’m confused as to what differences would lead to this. If you know, please let me know and I’ll add a footnote to this article explaining it.

Back to history search. For this to work, we also need to setup history logging:

export SAVEHIST=1000000 export HISTFILE=$HOME/.zsh_history export HISTFILESIZE=1000000 export HISTSIZE=1000000 Why did it show up twice for tmux?

Because tmux creates a login shell. Adding:

echo PROFILE

to profile and:

echo SHRC

to shrc confirms this with:

PROFILE SHRC SHRC

For now, profile sources shrc so that running twice is expected.

But after this exploration and diagram, it’s clear we don’t need that for zsh. Removing this will break remote bash shells (see above diagram), but I can live without those on my development laptop.

Removing that line results in the expected output for a new terminal:

SHRC

And the full output for a new tmux session or console:

PROFILE SHRC

So finally we’re back to a normal state!

This post is a bit unfocused but I hope it helps someone else repair or enhance their shell environment.

If you liked this4, or know of any other ways to manage this I could use, let me know at fixmyshell@tookmund.com.

  1. This chart comes from the excellent Shell Startup Scripts article by Peter Ward. I’ve generated the SVG from the graphviz source linked in the article. 

  2. Technically it has be compatible with Korn shell, but a quick google seems to suggest that that’s actually a subset of POSIX shell. 

  3. I use oh-my-zsh at $WORK but for now I’m going to simplify my personal configuration. If I end up using a lot of plugins I’ll reconsider this. 

  4. Or if you’ve found any typos or other issues that I should fix. 

Categories: FLOSS Project Planets

Gaming only on Linux, one year in

Planet KDE - Tue, 2024-01-02 18:00

I have now been playing games only on Linux for a year, and it has been great.

With the GPU shortage, I had been waiting for prices to come back to reasonable levels before buying a new GPU. So far, I had always bought NVIDIA GPUs as I was using Windows to run games and the NVIDIA drivers had a better “reputation” than the AMD/Radeon ones.

With Valve’s Proton seriously taking off thanks to the Steam Deck, I wanted to get rid of the last computer in the house that was running Microsoft Windows, that I had kept only for gaming.

But the NVIDIA drivers story on Linux had never been great, especially on distributions that move kernel versions quickly to follow upstream releases like Fedora. I had tried using the NVIDIA binary drivers on Fedora Kinoite but quickly ran into some of the issues that we have listed in the docs.

At the time, the Universal Blue project did not exist yet (Jorge Castro started it a bit later in the year), otherwise I would have probably used that instead. If you need NVIDIA support today on Fedora Atomic Desktops (Silverblue, Kinoite, etc.), I heavily recommend using the Universal Blue images.

Hopefully this will be better in the future for NVIDIA users with the work on NVK

So, at the beginning of last year (January 2023), I finally decided to buy an AMD Radeon RX 6700 XT GPU card.

What a delight. Nothing to setup, fully supported out of the box, works perfectly on Wayland. Valve’s Proton does wonders. I can now play on my Linux box all the games that I used to play on Windows and they run perfectly. Just from last year, I played Age of Wonders 4 and Baldur’s Gate 3 without any major issue, and did it pretty close to the launch dates. Older titles usually work fairly well too.

Sure, some games require some little tweaks, but it is nothing compared to the horrors of managing a Windows machine. And some games require tweaks on Windows as well (looking at you Cyberpunk 2077). The best experience is definitely with games bought on Steam which usually work out of the box. For those where it is not the case, protondb is usually a good source to find the tweaks needed to make the games work. I try to keep the list of tweaks I use for the games that I play updated on my profile there.

I am running all of this on Fedora Kinoite with the Steam Flatpak. If you want a more console-like or Steam Deck-like experience on your existing computers, I recommend checking out the great work from the Bazzite team.

Besides Steam, I use Bottles, Cartridge and Heroic Games Launcher as needed (all as Flatpaks). I have not looked at Origins or Uplay/Ubisoft Connect games yet.

According to protondb, the only games from my entire Steam library that are not supported are mostly multiplayer games that require some specific anti-cheat that is only compatible with Windows.

I would like to say a big THANK YOU to all the open source graphics and desktop developers out there and to (in alphabetical order) AMD, Collabora, Igalia, Red Hat, Valve, and other companies for employing people or funding the work that makes gaming on Linux a reality.

Happy new year and happy gaming on Linux!

Categories: FLOSS Project Planets

FSF Events: Free Software Directory meeting on IRC: Friday, January 05, starting at 12:00 EST (17:00 UTC)

GNU Planet! - Tue, 2024-01-02 16:42
Join the FSF and friends on Friday, January 05, from 12:00 to 15:00 EST (17:00 to 20:00 UTC) to help improve the Free Software Directory.
Categories: FLOSS Project Planets

DrupalEasy: Using GitHub Copilot in Visual Studio Code to create a PhpUnit test

Planet Drupal - Tue, 2024-01-02 15:44

Like many folks, I've been fascinated by the incredible evolution of AI tools in 2023. At the same time, I've been working to figure out exactly where AI fits into my various roles as a Drupal developer, trainer, and business owner.

In this blog post, I detail a recent exploration I made into using AI (GitHub Copilot, to be precise) to generate a PhpUnit test in a Drupal contrib module that I maintain.

tl;dr I was impressed.

Prerequisites

For this exercise, I used Visual Studio Code with the GitHub Copilot extension installed. I am using the Copilot Business plan ($19/month, but there is a free trial available).

The task

The Markdown Easy contrib module includes a very simple service class with no dependencies that implements a single method. Normally, Drupal service classes would require a kernel test (due to dependencies,) but in this case a simple unit test will do the job.

While using Drush's generate command has generally been my preferred method for scaffolding a test class, I found that using Copilot provides a much more advanced starting point. But, like anything else generated via AI, knowledge of the right way to perform a task is not optional. Code generated by AI might be correct, but blind confidence in what the AI provides will surely get you into trouble sooner, rather than later.

The getFilterWeights() method that we tested is a relatively simple method that returns an array of filter weights for three filters related to the configuration of the Markdown Easy text filter. The method takes a single parameter: an array of filters assigned to a text format. This method ensures that the configured order of filters related to Markdown Easy provides a secure experience.

Therefore, it makes sense that the unit test should pass in several sets of filters to the getFilterWeights() method and ensure that the returned array is correct - a fairly simple test that utilizes a data provider. To be honest, if I wasn't experimenting with using Copilot to generate tests, I probably wouldn't have this test, as it is almost trivial.

Regardless, adding test coverage to custom or contrib modules is a fantastic way of building modules where sustainability is a top priority. This is one of the reasons why writing PhpUnit tests is a prominent aspect of the Professional Module Development course that I teach. 

Using the Copilot contextual menu - attempt 1

In this example, I placed my cursor at the end of the method name and then navigated the contextual menu to Copilot | Generate tests

By default, this will attempt to create the unit test class in markdown_easy/src/MarkdownUtilityTests.php, which is not the best-practice for location for tests in Drupal modules. Luckily, we can modify the location via the Create as option.

This results in a Save as dialog box in which we can specify the proper location for the test:

The test file is created in the proper place, but (at least in my test) it oddly didn't include an opening <?php tag, nor a namespace. Luckily, Copilot didn't make me work too hard on the namespace, as it correctly suggested the proper namespace via its autocomplete feature:

Once I manually added these (as well as declare(strict_types = 1);), the resulting test was:

<?php declare(strict_types = 1); namespace Drupal\Tests\markdown_easy\Unit; use Drupal\markdown_easy\MarkdownUtility; use Drupal\Tests\UnitTestCase; /** * Tests the MarkdownUtility class. * * @group markdown_easy */ class MarkdownUtilityTest extends UnitTestCase { /** * Tests the getFilterWeights method. */ public function testGetFilterWeights() { $markdownUtility = new MarkdownUtility(); // Test case 1: Empty filters array. $filters = []; $expectedWeights = []; $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); // Test case 2: Non-empty filters array. $filters = [ 'filter1' => 10, 'filter2' => 5, 'filter3' => 15, ]; $expectedWeights = [ 'filter1' => 10, 'filter2' => 5, 'filter3' => 15, ]; $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); // Test case 3: Filters array with duplicate keys. $filters = [ 'filter1' => 10, 'filter2' => 5, 'filter1' => 15, ]; $expectedWeights = [ 'filter1' => 15, 'filter2' => 5, ]; $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); } }

As-is, this test does not pass, as Copilot missed the fact that the method will only return the weights of specific filters, so using filter1, filter2, etc.. doesn't help much...

Using the Copilot contextual menu - attempt 2

In this attempt, I highlighted the entire getFilterWeights() method and then selected Generate tests from the contextual menu. The result (without any changes on my part) was much better:

<?php namespace Drupal\Tests\markdown_easy\Unit; use Drupal\Tests\UnitTestCase; use Drupal\markdown_easy\MarkdownUtility; /** * Unit Test coverage for MarkdownUtility. * * @coversDefaultClass \Drupal\markdown_easy\MarkdownUtility * * @group markdown_easy */ class MarkdownUtilityTest extends UnitTestCase { /** * Tests the getFilterWeights method. * * @covers ::getFilterWeights */ public function testGetFilterWeights(): void { $markdownUtility = new MarkdownUtility(); // Create a sample array of filters. $filters = [ 'filter_html' => [ 'status' => true, 'weight' => 10, ], 'filter_autop' => [ 'status' => true, 'weight' => 20, ], 'markdown_easy' => [ 'status' => true, 'weight' => 30, ], ]; // Call the getFilterWeights method. $result = $markdownUtility->getFilterWeights($filters); // Assert that the weights are set correctly. $this->assertEquals(10, $result['filter_html']); $this->assertEquals(20, $result['filter_autop']); $this->assertEquals(30, $result['markdown_easy']); } }

Not only did Copilot get the opening PHP tag and the namespace correct, but the quality of the test was much better, and passed without any changes. However; this test was generated with only a single set of test data, so I decided to try again…

Using the contextual menu - attempt 3

In this attempt, I again highlighted the entire getFilterWeights() method and then proceeded to select Generate tests again from the contextual menu. The result was much better (but again oddly missing the opening PHP tag):

namespace Drupal\Tests\markdown_easy\Unit; use Drupal\Tests\UnitTestCase; use Drupal\markdown_easy\MarkdownUtility; /** * Tests the MarkdownUtility class. * * @group markdown_easy */ class MarkdownUtilityTest extends UnitTestCase { /** * Tests the getFilterWeights method. */ public function testGetFilterWeights() { $markdownUtility = new MarkdownUtility(); // Test case 1: All filters are disabled. $filters = [ 'filter_html' => ['status' => FALSE], 'filter_autop' => ['status' => FALSE], 'markdown_easy' => ['status' => FALSE], ]; $expectedWeights = [ 'filter_html' => NULL, 'filter_autop' => NULL, 'markdown_easy' => NULL, ]; $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); // Test case 2: Only filter_html is enabled. $filters = [ 'filter_html' => ['status' => TRUE, 'weight' => 10], 'filter_autop' => ['status' => FALSE], 'markdown_easy' => ['status' => FALSE], ]; $expectedWeights = [ 'filter_html' => 10, 'filter_autop' => NULL, 'markdown_easy' => NULL, ]; $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); // Test case 3: All filters are enabled with different weights. $filters = [ 'filter_html' => ['status' => TRUE, 'weight' => 10], 'filter_autop' => ['status' => TRUE, 'weight' => 20], 'markdown_easy' => ['status' => TRUE, 'weight' => 30], ]; $expectedWeights = [ 'filter_html' => 10, 'filter_autop' => 20, 'markdown_easy' => 30, ]; $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); } }

In this attempt, the test was created with three sets of data, and passed without any changes (other than adding the opening PHP tag). But, can we use Copilot to make it better? Ideally, the three sets of input data should be provided as part of a PhpUnit data provider method.

Refactoring the test method to use a PhpUnit data provider

To achieve this, I placed my cursor inside the test method, and selected Copilot | Start inline chat, and entered the prompt: "Can we refactor this method to use a data provider?" The result displayed the proposed result in a Visual Studio Code diff (similar to what is used when resolving Git code conflicts,) with the option for me to either Accept, Discard, or Regenerate (represented by a circle-arrow icon) the proposed solution. I accepted, and the resulting code took a little bit of massaging to make correct. While Copilot did properly create the data provider method and updated the test method, it left a bunch of the original test method code at the bottom of the class, outside of any method. Once I (easily) removed this artifact, the test passed as expected.

The resulting code was:

<?php declare(strict_types = 1); namespace Drupal\Tests\markdown_easy\Unit; use Drupal\markdown_easy\MarkdownUtility; use Drupal\Tests\UnitTestCase; /** * Tests the MarkdownUtility class. * * @group markdown_easy */ class MarkdownUtilityTest extends UnitTestCase { /** * Data provider for testing the getFilterWeights method. * * @return array * An array of test cases. */ public function filterWeightsDataProvider(): array { return [ [ [ 'filter_html' => ['status' => FALSE], 'filter_autop' => ['status' => FALSE], 'markdown_easy' => ['status' => FALSE], ], [ 'filter_html' => NULL, 'filter_autop' => NULL, 'markdown_easy' => NULL, ], ], [ [ 'filter_html' => ['status' => TRUE, 'weight' => 10], 'filter_autop' => ['status' => FALSE], 'markdown_easy' => ['status' => FALSE], ], [ 'filter_html' => 10, 'filter_autop' => NULL, 'markdown_easy' => NULL, ], ], [ [ 'filter_html' => ['status' => TRUE, 'weight' => 10], 'filter_autop' => ['status' => TRUE, 'weight' => 20], 'markdown_easy' => ['status' => TRUE, 'weight' => 30], ], [ 'filter_html' => 10, 'filter_autop' => 20, 'markdown_easy' => 30, ], ], ]; } /** * Tests the getFilterWeights method. * * @dataProvider filterWeightsDataProvider */ public function testGetFilterWeights(array $filters, array $expectedWeights) { $markdownUtility = new MarkdownUtility(); $this->assertEquals($expectedWeights, $markdownUtility->getFilterWeights($filters)); } }

So, that's not too bad for a few minutes of work! But, as a stickler for clean code, there was still some work ahead for me to get an acceptable PhpStan report. 

Code quality changes

Overall, the quality of the code that was provided by Copilot was really good. But, this comes with the caveat that I utilize the PHP Sniffer & Beautifier Visual Studio Code extension (configured to use Drupal coding standards,) so it could be that code generated by Copilot is automatically formatted as it is generated (I really have no idea.). The bottom line is that I had zero phpcs issues in the code generated by Copilot.

For PhpStan, I normally try to achieve level 6 compliance - this can be especially tricky when it comes to "no value type specified in iterable type" issues. Without getting into the details of solving issues of this type, I decided to let Copilot have a go at updating the docblock for the filterWeightsDataProvider() method - and much to my surprise, it was able to provide a reasonable fix:

The process to update the docblock for the testGetFilterWeights() method wasn't as simple, as it was missing the parameter information, so I added that manually and then used Copilot in a similar manner to solve the PhpStan issue.

There was an additional, minor, PhpStan issue that I solved manually as well. With that, I had achieved zero PhpStan level 6 issues, a clean phpcs report, and a passing test! 🎉

This new test has been committed to the module.

Lesson learned
  1. Context matters (a lot) when generating tests (any code, really) using Copilot. In my (limited) experience, the files open in Visual Studio Code, where the cursor is, and what is highlighted makes a significant difference in the results provided. 
  2. Do not be afraid to use the Regenerate button (the circle-arrow icon) when generating any code, including tests. I have found that if I don't like the initial result, regenerating often results in a much better option the second or third time around.

  3. The Start Inline Chat option in the contextual menu is rapidly becoming my new best coding friend. Do not be afraid to experiment with it and use it more than you think you should. I find it very useful for making code more concise, suggesting documentation descriptions, and giving me a headstart in scaffolding entire methods. 
  4. This should go without saying, but don't trust anything that is generated by Copilot. This is a tool that you should look at to save time, but not solve problems for you.

Header image generated using ChatGPT4.

Thanks to Andy Giles and Cindy Garcia for reviewing this blog post. Cindy is a graduate of both Drupal Career Online and Professional Module Development. Andy is the owner of Blue Oak Interactive

Categories: FLOSS Project Planets

DrupalEasy: Introducing the Markdown Easy Drupal module

Planet Drupal - Tue, 2024-01-02 15:44

Markdown is a text processing library that takes readable text and converts it into HTML. Started over 20 years ago, it is now a widely-used library that makes it easy for people to write text documents that can be easily and predictably converted into HTML.

Quick example - the Markdown syntax "this is **important**" converts to "this is important" after passing through a Markdown converter.

There are many "flavors" of Markdown today - most include the basic syntax and many include their version of "extended Markdown". Examples include Github-flavored Markdown and CommonMark.

Predictably, there have been Markdown-related Drupal contrib modules for a number of years, with the standard being the predictably named Markdown module. Started in 2008, it is currently used by about 5,000 Drupal sites (down from a high of more than 12,000 sites a few years ago).

Unfortunately, as of the publishing of this blog post, the current iteration of the Markdown module does not yet have a full release compatible with Drupal 10, although there are efforts ongoing to remedy this.

Why a new module?

DrupalEasy.com uses the Markdown syntax for some of its content. During our effort to upgrade DrupalEasy.com, the Markdown module became our last blocker to achieving that goal.

We took the time to review the current state of the Markdown module and its effort to be upgraded for Drupal 10, and in the end, we decided to pursue a simpler (in terms of module configuration,) less-time-consuming (in terms of contribution time,) and sustainable (in terms of ease of future upgrades) solution.

The existing Markdown module is configurable - really configurable. Some might say "too configurable".

One of the main configuration choices one has to make when using the Markdown module is choosing which Markdown library to utilize. The various open-source Markdown-processor libraries implement their own flavor of Markdown (like the Github and CommonMark libraries mentioned above). In fact, the Markdown module allows you to utilize multiple Markdown processor libraries. Once a library is selected, there are a myriad of additional configuration options available for each library and for each text format where the Markdown filter is utilized. Configuring the Markdown module in a secure manner is (in our experience) a bit of a chore.

I have no doubt that there are users of the Markdown module that require that level of control and fine-grained configuration. In those cases, the Markdown module is probably their best choice.

But, what about users who don't need that level of control? Users who just want to process Markdown formatted text into basic HTML with a minimum of fuss using a tried-and-true Markdown library. It is for these users that the Markdown Easy module was created.

What makes Markdown Easy different?

In short:

  1. It is opinionated about the Markdown library - it uses CommonMark. Full stop.
  2. It is opinionated about its configuration - it is configured to be as secure as possible.
  3. It is easy to install and configure (see 1 and 2 above).

It was decided early on that the CommonMark library would be a dependency of Markdown Easy. It has more than 150 million installs on Packagist.org and is considered by many as the "gold standard" of Markdown libraries. In addition, it comes with the "Github-flavored Markdown" option as well. Selecting between these two "flavors" of Markdown is the only configuration option you have in Markdown Easy.

The module is preconfigured to be as secure as possible. Configured incorrectly, a Markdown library will output virtually any HTML tag - some that can be abused by bad actors. For example, the Markdown Easy module (by default) does not allow unsafe links or HTML tags to be processed (both of these options can be overridden with a Markdown Easy hook implementation). Additionally, validation of any text format utilizing the Markdown Easy filter requires that both the "Limit HTML" and "Convert line breaks" Drupal core filters be enabled and set to run in a prescribed order to ensure secure usage and consistent results (again, this can be overridden with a hook).

What features are planned for future versions of Markdown Easy?

Not many.

The goal is to have an easy-to-maintain (between major versions of Drupal core,) easy-to-configure (for site-builders,) and easy-to-customize (via Drupal hooks) module.

The only feature that we are currently interested in adding is the option of a GitLab-flavored Markdown library.

Is this module ready for prime-time?

We think so. In fact, the article that you're reading was written in Markdown and processed into HTML by the Markdown Easy module (with the exception of the logo - standard HTML was used for styling purposes).

Full documentation, including examples for overriding default validation and configuration, is available on drupal.org.

Markdown Easy's code base is primarily validation and automated test code. The filter plugin is 129 lines of code while validation and automated tests include over 700 lines of code.

We are using Drupal.org's Gitlab CI to run the tests in both Drupal 9 and Drupal 10 on every merge request and the module is covered by Drupal's security advisory policy.

We invite you to check it out and let us know what you think!

Categories: FLOSS Project Planets

FSF Blogs: Boost the FSF's advocacy for free software in education: Help us reach our stretch goal

GNU Planet! - Tue, 2024-01-02 14:50
We're thankful for the support we've received so far. Program manager Miriam Bastian shares some of the FSF's recent efforts in education, the theme of our current fundraiser, and announces a new stretch goal.
Categories: FLOSS Project Planets

Boost the FSF's advocacy for free software in education: Help us reach our stretch goal

FSF Blogs - Tue, 2024-01-02 14:50
We're thankful for the support we've received so far. Program manager Miriam Bastian shares some of the FSF's recent efforts in education, the theme of our current fundraiser, and announces a new stretch goal.
Categories: FLOSS Project Planets

PyCoder’s Weekly: Issue #610 (Jan. 2, 2024)

Planet Python - Tue, 2024-01-02 14:30

#610 – JANUARY 2, 2024
View in Browser »

Build a Scalable Flask Web Project From Scratch

In this tutorial, you’ll explore the process of creating a boilerplate for a Flask web project. It’s a great starting point for any scalable Flask web app that you wish to develop in the future, from basic web pages to complex web applications.
REAL PYTHON

JIT Coming to Python 3.13

Slides related to the upcoming JIT commit for Python 3.13. Note, GitHub paginates it if you don’t download it, so click the “More Pages” button to keep reading.
GITHUB.COM/BRANDTBUCHER

The Biggest Discoveries in Computer Science in 2023

Although 2023 was full of AI news in computer science, it wasn’t the only news. This article summarizes the breakthroughs in 2023.
BILL ANDREWS

PyCon US Received Record Number of Submissions

MARIATTA

Python Jobs Senior Python Architect and Tech Lead (America)

Six Feet Up

Python Tutorial Editor (Anywhere)

Real Python

More Python Jobs >>>

Articles & Tutorials Python, C, Assembly – 2,500x Faster Cosine Similarity

Cosine similarity is a check to see if two vectors point in the same direction, regardless of magnitude. This test is frequently used in some machine learning algorithms. This article details the various steps in speeding up the code, starting with vanilla Python and going all the way down to hand tuned assembly language.
ASH VARDANIAN

PyCoder’s Weekly 2023 Wrap Up

It’s been a fascinating year for the Python language and community. PyCoder’s Weekly included over 1,500 links to articles, blog posts, tutorials, and projects in 2023. Christopher Trudeau is back on the show this week to help wrap up everything by sharing some highlights and Python trends from across the year.
REAL PYTHON podcast

Python’s Soft Keywords

Python includes soft keywords: tokens that are important to the parser but can also be used as variable names. This article shows you what a soft keyword is and how to find them in Python 3.12 (both the easy and hard way).
RODRIGO GIRÃO SERRÃO

The State of Developer Ecosystem in 2023

This is a cross-language developer survey of tools used in the industry. It includes questions about AI adoption, cloud tools, and more. 54% of respondents use Python as their most frequent language.
JETBRAINS

Build an Instant Messaging App With Django

This articles shows you how to take advantage of some the newer async mechanisms in Django to build a messaging app. Things that used to require a third party library are now part of the framework.
TOM DEKAN • Shared by Tom Dekan

Majority and Frequent Elements: Boyer Moore and Misra Gries

The Boyer-Moore majority vote algorithm looks for an element that appears more than n/2 times in a sequence using O(n) time. This article shows you how it works using Python code.
GITHUB.COM/NAUGHTYCONSTRICTOR • Shared by Mohammed Younes ELFRAIHI

Why Should You Become an Engineering Manager?

Many developers dread the idea of becoming a manager, but there are some things you can only learn by doing. This article outlines why management might be the right thing for you.
CHARITY MAJORS

Banish State-Mutating Methods From Data Classes

Redowan has strong opinions on reserving dataclasses for data-class purposes only: their methods should have no data modification side-effects. This article outlines why.
REDOWAN DELOWAR

Django 5.0 Is Out!

Django 5 was recently released and this in-depth article covers what changed, how to upgrade from an earlier version, and how the Django version numbering system works.
ERIC MATTHES

How Many CPU Cores Can You Actually Use in Parallel?

“Figuring out how much parallelism your program can use is surprisingly tricky.” This article shows you why it is complicated and what you can determine.
ITAMAR TURNER-TRAURING

TIL: Forcing pip to Use virtualenv

A quick tip on how to set an environment variable so that pip refuses to install a package unless in an active virtual environment.
DANIEL ROY GREENFIELD

Configuration in Python Applications

This post talks about how to store configuration for your script and how and when to load the information into your program.
ROBERT RODE

Raise the Right Exceptions

Knowing when to raise the right exception is important, but often you don’t have to: Python might do it for you.
JAMES BENNETT

Projects & Code pyapp: Build Self-Bootstrapped Python Applications

GITHUB.COM/OFEK

django-tui: Inspect and Run Django Commands in a TUI

GITHUB.COM/ANZE3DB

cicada: FOSS, Cross-Platform GitHub Actions

GITHUB.COM/CICADA-SOFTWARE

marker: Convert PDF to Markdown

GITHUB.COM/VIKPARUCHURI

Amphion: Toolkit for Audio, Music, and Speech Generation

GITHUB.COM/OPEN-MMLAB

Events Weekly Real Python Office Hours Q&A (Virtual)

January 3, 2024
REALPYTHON.COM

Canberra Python Meetup

January 4, 2024
MEETUP.COM

Sydney Python User Group (SyPy)

January 4, 2024
SYPY.ORG

PiterPy Meetup

January 9, 2024
PITERPY.COM

Leipzig Python User Group Meeting

January 9, 2024
MEETUP.COM

Building Python Communities Around Python for Kids

January 10 to January 11, 2024
NOKIDBEHIND.ORG

Happy Pythoning!
This was PyCoder’s Weekly Issue #610.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Categories: FLOSS Project Planets

Matthew Garrett: Dealing with weird ELF libraries

Planet Debian - Tue, 2024-01-02 14:04
Libraries are collections of code that are intended to be usable by multiple consumers (if you're interested in the etymology, watch this video). In the old days we had what we now refer to as "static" libraries, collections of code that existed on disk but which would be copied into newly compiled binaries. We've moved beyond that, thankfully, and now make use of what we call "dynamic" or "shared" libraries - instead of the code being copied into the binary, a reference to the library function is incorporated, and at runtime the code is mapped from the on-disk copy of the shared object[1]. This allows libraries to be upgraded without needing to modify the binaries using them, and if multiple applications are using the same library at once it only requires that one copy of the code be kept in RAM.

But for this to work, two things are necessary: when we build a binary, there has to be a way to reference the relevant library functions in the binary; and when we run a binary, the library code needs to be mapped into the process.

(I'm going to somewhat simplify the explanations from here on - things like symbol versioning make this a bit more complicated but aren't strictly relevant to what I was working on here)

For the first of these, the goal is to replace a call to a function (eg, printf()) with a reference to the actual implementation. This is the job of the linker rather than the compiler (eg, if you use the -c argument to tell gcc to simply compile to an object rather than linking an executable, it's not going to care about whether or not every function called in your code actually exists or not - that'll be figured out when you link all the objects together), and the linker needs to know which symbols (which aren't just functions - libraries can export variables or structures and so on) are available in which libraries. You give the linker a list of libraries, it extracts the symbols available, and resolves the references in your code with references to the library.

But how is that information extracted? Each ELF object has a fixed-size header that contains references to various things, including a reference to a list of "section headers". Each section has a name and a type, but the ones we're interested in are .dynstr and .dynsym. .dynstr contains a list of strings, representing the name of each exported symbol. .dynsym is where things get more interesting - it's a list of structs that contain information about each symbol. This includes a bunch of fairly complicated stuff that you need to care about if you're actually writing a linker, but the relevant entries for this discussion are an index into .dynstr (which means the .dynsym entry isn't sufficient to know the name of a symbol, you need to extract that from .dynstr), along with the location of that symbol within the library. The linker can parse this information and obtain a list of symbol names and addresses, and can now replace the call to printf() with a reference to libc instead.

(Note that it's not possible to simply encode this as "Call this address in this library" - if the library is rebuilt or is a different version, the function could move to a different location)

Experimentally, .dynstr and .dynsym appear to be sufficient for linking a dynamic library at build time - there are other sections related to dynamic linking, but you can link against a library that's missing them. Runtime is where things get more complicated.

When you run a binary that makes use of dynamic libraries, the code from those libraries needs to be mapped into the resulting process. This is the job of the runtime dynamic linker, or RTLD[2]. The RTLD needs to open every library the process requires, map the relevant code into the process's address space, and then rewrite the references in the binary into calls to the library code. This requires more information than is present in .dynstr and .dynsym - at the very least, it needs to know the list of required libraries.

There's a separate section called .dynamic that contains another list of structures, and it's the data here that's used for this purpose. For example, .dynamic contains a bunch of entries of type DT_NEEDED - this is the list of libraries that an executable requires. There's also a bunch of other stuff that's required to actually make all of this work, but the only thing I'm going to touch on is DT_HASH. Doing all this re-linking at runtime involves resolving the locations of a large number of symbols, and if the only way you can do that is by reading a list from .dynsym and then looking up every name in .dynstr that's going to take some time. The DT_HASH entry points to a hash table - the RTLD hashes the symbol name it's trying to resolve, looks it up in that hash table, and gets the symbol entry directly (it still needs to resolve that against .dynstr to make sure it hasn't hit a hash collision - if it has it needs to look up the next hash entry, but this is still generally faster than walking the entire .dynsym list to find the relevant symbol). There's also DT_GNU_HASH which fulfills the same purpose as DT_HASH but uses a more complicated algorithm that performs even better. .dynamic also contains entries pointing at .dynstr and .dynsym, which seems redundant but will become relevant shortly.

So, .dynsym and .dynstr are required at build time, and both are required along with .dynamic at runtime. This seems simple enough, but obviously there's a twist and I'm sorry it's taken so long to get to this point.

I bought a Synology NAS for home backup purposes (my previous solution was a single external USB drive plugged into a small server, which had uncomfortable single point of failure properties). Obviously I decided to poke around at it, and I found something odd - all the libraries Synology ships were entirely lacking any ELF section headers. This meant no .dynstr, .dynsym or .dynamic sections, so how was any of this working? nm asserted that the libraries exported no symbols, and readelf agreed. If I wrote a small app that called a function in one of the libraries and built it, gcc complained that the function was undefined. But executables on the device were clearly resolving the symbols at runtime, and if I loaded them into ghidra the exported functions were visible. If I dlopen()ed them, dlsym() couldn't resolve the symbols - but if I hardcoded the offset into my code, I could call them directly.

Things finally made sense when I discovered that if I passed the --use-dynamic argument to readelf, I did get a list of exported symbols. It turns out that ELF is weirder than I realised. As well as the aforementioned section headers, ELF objects also include a set of program headers. One of the program header types is PT_DYNAMIC. This typically points to the same data that's present in the .dynamic section. Remember when I mentioned that .dynamic contained references to .dynsym and .dynstr? This means that simply pointing at .dynamic is sufficient, there's no need to have separate entries for them.

The same information can be reached from two different locations. The information in the section headers is used at build time, and the information in the program headers at run time[3]. I do not have an explanation for this. But if the information is present in two places, it seems obvious that it should be able to reconstruct the missing section headers in my weird libraries? So that's what this does. It extracts information from the DYNAMIC entry in the program headers and creates equivalent section headers.

There's one thing that makes this more difficult than it might seem. The section header for .dynsym has to contain the number of symbols present in the section. And that information doesn't directly exist in DYNAMIC - to figure out how many symbols exist, you're expected to walk the hash tables and keep track of the largest number you've seen. Since every symbol has to be referenced in the hash table, once you've hit every entry the largest number is the number of exported symbols. This seemed annoying to implement, so instead I cheated, added code to simply pass in the number of symbols on the command line, and then just parsed the output of readelf against the original binaries to extract that information and pass it to my tool.

Somehow, this worked. I now have a bunch of library files that I can link into my own binaries to make it easier to figure out how various things on the Synology work. Now, could someone explain (a) why this information is present in two locations, and (b) why the build-time linker and run-time linker disagree on the canonical source of truth?

[1] "Shared object" is the source of the .so filename extension used in various Unix-style operating systems
[2] You'll note that "RTLD" is not an acryonym for "runtime dynamic linker", because reasons
[3] For environments using the GNU RTLD, at least - I have no idea whether this is the case in all ELF environments

comments
Categories: FLOSS Project Planets

Luca Saiu: Languages and complexity, Part I: why I love Anki

GNU Planet! - Tue, 2024-01-02 13:47
Lately I have not been as active in GNU (https://www.gnu.org) as I would have liked—which I plan to change. Apart from work I was busy with happy family life next to E.; and, I guess, with contemplating the dismal state of the West as it descends further and further into tyranny amid the general indifference. Maybe in part seeking solace from the news I focused with renewed intensity on my hobby, studying the Russian language for no reason much more practical than my love for Nineteen-Century novels. I have heard more than one Russian teacher vocally disapproving of literature as ... [Read more]
Categories: FLOSS Project Planets

Hynek Schlawack: How to Ditch Codecov for Python Projects

Planet Python - Tue, 2024-01-02 11:39

Codecov’s unreliability breaking CI on my open source projects has been a constant source of frustration for me for years. I have found a way to enforce coverage over a whole GitHub Actions build matrix that doesn’t rely on third-party services.

Categories: FLOSS Project Planets

Chapter Three: 15 Tips for Writing Better Web Copy

Planet Drupal - Tue, 2024-01-02 11:12
If you’re reading this blog post, something had to bring you here as opposed to the other roughly 600 million blogs out there. Perhaps you’re a regular visitor to this site. Maybe you Googled “writing tips for better web copy” and this showed up. Or you might be one of our social media followers and liked the look of this title. Search engines are a lot smarter than they were ten years ago. Gone are the days when you could “cheat” the system by front-loading your text with a bunch of keywords. Today’s SEO reads for quality of content as well as keywords. But once you’ve got a web user’s attention, you want to keep it — and hopefully encourage them to visit other pages of your site and take whatever actions you want them to take. Again, quality web content is key.
Categories: FLOSS Project Planets

Thomas Koch: Good things come ... state folder

Planet Debian - Tue, 2024-01-02 11:05
Posted on January 2, 2024 Tags: debian, free software, life

Just a little while ago (10 years) I proposed the addition of a state folder to the XDG basedir specification and expanded the article XDGBaseDirectorySpecification in the Debian wiki. Recently I learned, that version 0.8 (from May 2021) of the spec finally includes a state folder.

Granted, I wasn’t the first to have this idea (2009), nor the one who actually made it happen.

Now, please go ahead and use it! Thank you.

Categories: FLOSS Project Planets

Pages