Feeds

I became interested in Network Time Protocol (NTP) last year after learning how fundamental this protocol is to the functioning of the global Internet. NTP helps synchronize clocks on devices over the Internet, which is essential for secure browsing, timestamping, keeping everyone in sync or just checking what time it is. Computers usually have a hardware real-time clock (RTC) but that deviates over time, so an occasional sync over NTP is required to keep the time accurate. Many network and IoT devices don’t have hardware RTC so have even more reliance on NTP.

Accurate time keeping starts with reference clocks like atomic clocks, GPS etc. Multiple government standard agencies host these reference clocks, which are regarded as Stratum 0. Stratum 1 servers are known as primary servers, and directly connect to Stratum 0 clocks for time. Stratum 1 servers then distribute time to Stratum 2 and further down the hierarchy. Computers typically connects to one or more Stratum 1/2/3… servers to get their time.

Someone has to host these public Stratum 1,2,3… NTP servers. That’s what NTP pool, a global effort by volunteers does. They provide NTP servers for the public to use. As of today, there are 4700+ servers in the pool which are free to use for anyone.

Now let’s come to the reason for writing this post. Indian Computer Emergency Response Team (CERT-In) in April 2022 released a set of cybersecurity directions which set the alarm bells ringing. Internet Society (and almost everyone else) wrote about it.

And then there was this specific section about NTP:

All service providers, intermediaries, data centres, body corporate and Government organisations shall connect to the Network Time Protocol (NTP) Server of National Informatics Centre (NIC) or National Physical Laboratory (NPL) or with NTP servers traceable to these NTP servers, for synchronisation of all their ICT systems clocks. Entities having ICT infrastructure spanning multiple geographies may also use accurate and standard time source other than NPL and NIC, however it is to be ensured that their time source shall not deviate from NPL and NIC.

CSIR-National Physical Laboratory (NPL) is the official timekeeper for India and hosts the only public Stratum 1 clock in India, according to NTP pool website. So I was naturally curious to know what kind of infrastructure they’re running for NTP. India has a Right to Information (RTI) Act which, like the Freedom of Information Act (FOIA) in the United States, gives citizens rights to request information from governmental entities, to which they have to respond in under 30 days. So last year, I filed two sets of RTI (one after the first reply came) inquiring about NPL’s public NTP server setup.

The first RTI had some generic questions:

First RTI. Click to enlarge

This gave a vague idea about the setup, so I sat down and came with some specific questions in the next RTI.

Second RTI. Click to enlarge

Feel free to make your conclusions from it now. Bear in mind these were filled last year so things might have changed. Do let me know if you have more information about it.

Since its introduction, JSON has rapidly emerged as the predominant standard for the exchange of information. Whether you want to transfer data with an API or store information in a document database, it’s likely you’ll encounter JSON. Fortunately, Python provides robust tools to facilitate this process and help you manage JSON data efficiently.

In this tutorial, you’ll learn how to:

• Understand the JSON syntax
• Convert Python data to JSON
• Deserialize JSON to Python
• Write and read JSON files
• Validate JSON syntax
• Prettify JSON in the terminal
• Minify JSON with Python

While JSON is the most common format for data distribution, it’s not the only option for such tasks. Both XML and YAML serve similar purposes. If you’re interested in how the formats differ, then you can check out the tutorial on how to serialize your data with Python.

Free Bonus: Click here to download the free sample code that shows you how to work with JSON data in Python.

Take the Quiz: Test your knowledge with our interactive “Working With JSON Data in Python” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Working With JSON Data in Python

In this quiz, you'll test your understanding of working with JSON in Python. JSON has become the de facto standard for information exchange, and Python provides easy-to-use tools to handle JSON data.

Introducing JSON

The acronym JSON stands for JavaScript Object Notation. As the name suggests, JSON originated from JavaScript. However, JSON has transcended its origins to become language-agnostic and is now recognized as the standard for data interchange.

The popularity of JSON can be attributed to native support by the JavaScript language, resulting in excellent parsing performance in web browsers. On top of that, JSON’s straightforward syntax allows both humans and computers to read and write JSON data effortlessly.

To get a first impression of JSON, have a look at this example code:

JSON hello_world.json { "greeting": "Hello, world!" } Copied!

You’ll learn more about the JSON syntax later in this tutorial. For now, recognize that the JSON format is text-based. In other words, you can create JSON files using the code editor of your choice. Once you set the file extension to .json, most code editors display your JSON data with syntax highlighting out of the box:

The screenshot above shows how VS Code displays JSON data using the Bearded color theme. You’ll have a closer look at the syntax of the JSON format next!

Examining JSON Syntax

In the previous section, you got a first impression of how JSON data looks. And as a Python developer, the JSON structure probably reminds you of common Python data structures, like a dictionary that contains a string as a key and a value. If you understand the syntax of a dictionary in Python, you already know the general syntax of a JSON object.

Note: Later in this tutorial, you’ll learn that you’re free to use lists and other data types at the top level of a JSON document.

The similarity between Python dictionaries and JSON objects is no surprise. One idea behind establishing JSON as the go-to data interchange format was to make working with JSON as convenient as possible, independently of which programming language you use:

[A collection of key-value pairs and arrays] are universal data structures. Virtually all modern programming languages support them in one form or another. It makes sense that a data format that is interchangeable with programming languages is also based on these structures. (Source)

To explore the JSON syntax further, create a new file named hello_frieda.json and add a more complex JSON structure as the content of the file:

JSON hello_frieda.json 1{ 2 "name": "Frieda", 3 "isDog": true, 4 "hobbies": ["eating", "sleeping", "barking"], 5 "age": 8, 6 "address": { 7 "work": null, 8 "home": ["Berlin", "Germany"] 9 }, 10 "friends": [ 11 { 12 "name": "Philipp", 13 "hobbies": ["eating", "sleeping", "reading"] 14 }, 15 { 16 "name": "Mitch", 17 "hobbies": ["running", "snacking"] 18 } 19 ] 20} Copied!

In the code above, you see data about a dog named Frieda, which is formatted as JSON. The top-level value is a JSON object. Just like Python dictionaries, you wrap JSON objects inside curly braces ({}).

In line 1, you start the JSON object with an opening curly brace ({), and then you close the object at the end of line 20 with a closing curly brace (}).

Note: Although whitespace doesn’t matter in JSON, it’s customary for JSON documents to be formatted with two or four spaces to indicate indentation. If the file size of the JSON document is important, then you may consider minifying the JSON file by removing the whitespace. You’ll learn more about minifying JSON data later in the tutorial.

Inside the JSON object, you can define zero, one, or more key-value pairs. If you add multiple key-value pairs, then you must separate them with a comma (,).

Read the full article at https://realpython.com/python-json/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Long gone are the days of using HTML tables for page layouts. That time almost seems shrouded in myth. One of the (many) reasons tables fell out of use was the advent of responsive design to meet the needs of different contexts and screen sizes. Tables are rigid. Their presence naturally hampers the responsiveness of a website to conform to a smaller screen size.

But tables are still useful. In some cases, they are necessary.

In this quiz, you’ll test your understanding of Python’s Magic Methods.

By working through this quiz, you’ll revisit the concept of magic methods in Python, how they work, and how you can use them to customize the behavior of your classes.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Experience Builder (XB) is set to redefine how we build and launch Drupal sites. As a cornerstone of the Starshot initiative, the XB module bridges the gap between Drupal installation and site launch, simplifying the process for all users. Kazima Abbas, sub-editor at The Drop Times, interviews Lauri Eskola, the Product Lead for the XB initiative. Lauri shares the inspiration behind XB, its integration with Starshot, and how it enhances Drupal's user experience and accessibility.

tl;dr

Whenever you need to download files through the terminal and don't feel like using wget:

wcurl example.com/filename.txt

Manpage:
https://manpages.debian.org/unstable/curl/wcurl.1.en.html

Availability (comes installed with the curl package):

• Debian unstable - Since 2024-07-02
• Debian testing - Coming up between the second and third week of July 2024.
• Debian 12/bookworm backports - As soon as the package gets to Debian testing, I'll upload it to bookworm.
• Debian 12/bookworm - Depends on whether Debian's release team will approve it, it could be available in the next point release.
• Debian derivatives - Rolling releases will get it by the time it's on Debian testing (e.g.: Kali Linux). Stable derivatives only in their next major release.

If you don't want to wait for the package update to arrive, you can always copy the script and place it in your /usr/bin, the code is here:
https://github.com/Debian/wcurl/blob/main/wcurl
https://salsa.debian.org/debian/wcurl/-/blob/main/wcurl?ref_type=heads

Smoother CLI experience

Starting with curl version 8.8.0-2, the Debian's curl package now ships a wcurl executable.

wcurl is the solution for those who just need to download files without having to remember curl's parameters for things like automatically naming the files.

Some people, myself included, would fall back to using wget whenever there was a need to download a file. Sometimes even installing wget just for that usecase. After all, it's easier to remember "apt install wget" rather than "curl -L -O -C - ...".

wcurl consists of a simple shell script that provides sane defaults for the curl invocation, for when the use case is to just download files.

By default, wcurl will:

• Encode whitespaces in URLs;
• Download multiple URLs in parallel;
• Follow redirects;
• Automatically choose a filename as output;
• Perform retries;
• Resume from broken/interrupted downloads.
• Set the downloaded file timestamp to the value provided by the server, if available;

Example, to download a single file:

wcurl example.com/filename.txt

If you ever need to set a custom flag, you can make use of the -o/--opts wcurl option, anything set there will be passed to the curl invocation. Just beware that if you need to set any custom flags, it's likely you will be better served by calling curl directly. The -o/--opts options are there to allow for some flexibility in unforeseen circumstances.

The need for wcurl

I've always felt a bit ashamed of not remembering curl's parameters for downloading a file and automatically naming it, having resorted to wget most of the times this was needed (even installing wget when it wasn't there, just for this). I've spoken to a few other experienced people I know and confirmed what could be obvious to others: a lot of people struggle with this.

Recently, the curl project released the results of 2024's curl survey, which also showed this is as a much needed feature, just look at some of the answers:

Q: Which curl command line option do you think needs improvement and how?

-O, I really want wget like functionality where I don't have to specify the name

Downloading a file (like wget) could be improved - with automatic naming of the file

downloading files - wget is much cleaner

I wish the default behaviour when GETting a binary was to drop it on disk. That's the only reason 'wget foo.tgz" is still ingrained in my muscle memory .

Maybe have a way to download without specifying something in -o (the only reason i used wget still)

--remote-time should be default

--remote-name-all could really use a short flag

Q: If you miss support for something, tell us what!

"Write the data to the file named in the URL (or in redirects if I'm feeling daring), and timestamp the file to the last-modified-date". This is the main reason I'm still using wget.

I can finally feel less bad about falling back to wget due to not remembering the parameters I want.

Idealization vs. reality

I don't believe curl will ever change its default behavior in such a way that would accommodate this need, as that would have a side-effect of breaking things which expect the current behavior (the blast radius is literally the solar system).

This means a new executable needs to be shipped side-by-side with curl, an opportunity to start fresh and work with a more focused use case (to download files).

Ideally, this new executable would be maintained by the curl project, make use of libcurl under-the-hood, and be available everywhere. Nobody wants to worry if their systems have the tool or not, it should always be there.

Given I'm just a Debian Developer, with not as much free time as I wish, I've decided to write a simple shell script wrapper calling the curl CLI under-the-hood.

wcurl will come installed with the curl package from now on, and I will check with the release team about shipping it on the current Debian stable as well. Shipping wcurl in other distros will be up to them (Debian-derivatives should pick it up automatically, though).

We've tried to make it easy for anyone to ship this by using the curl license, keeping the script POSIX-compliant, and shipping a manpage.

Maybe if there's enough interest across distributions, someone might sign up for implementing this in upstream curl and increase its reach. I would be happy with the curl project reusing the wcurl name when that happens. It's unlikely that wcurl would be shipped by curl upstream as it is, assuming they would prefer a solution that uses libcurl direclty (more similar to curl the CLI, to maintain).

In the worst case, wcurl becomes a Debian-specific tool that only a few people are aware of, in the best case, it becomes the new go-to CLI tool for simply downloading files. I would be happy if at least someone other than me finds it useful.

Naming is hard

When I started working on it, I was calling the new executable "curld" (stands for "curl download"), but then when discussing this in one of our weekly calls in the Debian Brasília community, it was mentioned that this could be confused for a daemon.

We then settled for the name "wcurl", which doesn't really stand for anything, but it's very easy to remember.

You know... "it's that wget alternative for when you want to use curl instead" :)

Feedback

I'm hosting the code on Github and Debian's GitLab instance, feel free to open an issue to provide feedback.
https://salsa.debian.org/debian/wcurl
https://github.com/Debian/wcurl

We also have a Matrix room for the Debian curl maintainers:
https://matrix.to/#/#debian-curl-maintainers:matrix.org
We have historically spoken Portuguese in the room but we'll switch to English in case anyone joins.

Acknowledgments

The idea for wcurl came a few days before the curl-up conference 2024. I've been thinking a lot about developer productivity in the terminal lately, different tools and better defaults. Before curl-up, I was also thinking about packaging improvements for the curl package. I don't remember what exactly happened, but I likely had to download something and felt a bit ashamed of maintaining curl and not remembering the parameters to download files the way I wanted.

I first discussed this idea in the conference, where I asked the participants about it and there were no concerns raised, and some people said I should give it a go. Participating in curl-up was a really great experience and I'm thankful for the interactions I've had there.

On the Debian side, I've got reviews of the code and manpage by Sergio Durigan Junior <sergiodj>, Guilherme Puida Moreira <puida> and Carlos Henrique Lima Melara <charles>. Sergio ended up rewriting the tool to be POSIX-compliant (my version was written in bash), so he takes all the credit for the portability.

Calamares is an indepdendent Linux distro-installer, and we just released Calamares 3.3.7. There’s a couple of known issues that need tracking down, but it is a slow process – one entirely dependent on how much time volunteers are able and willing to put into careful bug reporting (and reproduction) and then dealing with code to fix them. Anyway, here’s some semi-coherent notes about Calamares and distro’s and issues and things.

Installing Kubuntu

I recently had to install an “emergency PC” for my mom, and picked Kubuntu – fairly arbitrarily, I must say. The install-Kubuntu icon does not look like Calamares, and first comes up with some other selection dialog in a different style, so I was pleasantly surprised to see it using Calamares after that.

It picked up Dutch from geo-location, which was fine. Less fine is that the Dutch translations aren’t complete. I’m pretty sure I could fix that.

Installation was straightforward, although I always pick “erase whole disk” and click through nearly all of the defaults. No complicated installs for me, and it worked without a hitch.

Mom didn’t need the emergency PC after all, so now I do have a spare low-end desktop machine with Kubuntu. I might turn it into a try-real-installs box for a while.

Building Calamares on KDE Neon

KDE Neon was a CI target for Calamares for a long time, but I switched it off in March 2024. Dependencies were a mess at the time, and the automatic CI builds were failing every night. Other distro’s didn’t have that struggle, so I put more CI weight on Fedora.

I did keep one of my desktop machines installed with KDE Neon, as a gaming machine. At least I can see what Plasma 6 is supposed to be like, as a reference for when we land it in FreeBSD. Untangling the package mess to return it to a development machine was just too much of a hassle.

At some point I removed Qt5 – and everything that depended on it – and then reinstalled some bits and did a pkcon update and whatever and the machine finally ended up in a workable state for development again, but don’t ask me exactly what I did and don’t talk to me about the upgrade experience.

Building Calamares on FreeBSD

It’s possible, just probably not very useful. Clang spits out mountains of warnings, which I occasionally try to address.

Not all of the warnings are all that useful – when dealing with command-line arguments, for instance. The relationship between argc and argv (conventional names for parameters) in main() is clear, but there’s nothing in the type of either to express that, so you get warnings like this one:

src/libcalamares/geoip/test_geoip.cpp:37:45: warning: unsafe buffer access [-Wunsafe-buffer-usage] 37 | QString selector = argc == 3 ? QString( argv[ 2 ] ) : QString(); Building Calamares on EndeavourOS

The live-ISO for EndeavourOS is one of my favorites. I don’t know why they ship an ISO with git in the live-image, but it means that I trivially have a system with a working Calamares configuration, which I can update with the latest version:

git clone http://github.com/calamares/calamares cd calamares sudo ./ci/deps-endeavouros.sh sudo pacman -Scc export CMAKE_ARGS="-DWITH_QT6=ON -DCMAKE_BUILD_TYPE=Debug" export BUILDDIR=build ./ci/build.sh

I have a VM with the live-ISO, but also two virtual disks attached. By doing the checkout and package download on one of the disks, and doing test-installs to the other, this is by far the easiest and most pleasant develop-and-test setup I have right now.

And the wallpaper is pretty nice, too.

We just released skrub 0.2.0. This release markedly simplifies learning on complex dataframes.

model = tabular_learner(‘classifier’) Simple, yet solid default baseline

The highlight of the release is the tabular_learner function, which facilitates creating pipelines that readily perform machine learning on dataframes, adding preprocessing to a scikit-learn compatible learner. The function packs defaults and heuristics to transform all forms of dataframes to a representation that is well suited to a learner, and it can adapt these transformation: tabular_learner(HistGradientBoostingClassifier()) encodes categories differently than tabular_learner(LogisticRegression()).

The heuristics are tuned based on much benchmarking and experience shows that they give good tradeoffs. The default tabular_learner(‘classifier’) is often a strong baseline.

The benefit are visible in a really simple example:

>>> # First retrieve data >>> from skrub.datasets import fetch_employee_salaries >>> dataset = fetch_employee_salaries() >>> df = dataset.X >>> y = dataset.y >>> # The dataframe is a quite rich and complex dataframe, with various columns >>> df

We can then easily build a learner that applies readily to this dataframe, without any transformation:

>>> from skrub import tabular_learner >>> learner = tabular_learner('regressor') >>> # The resulting learner can apply all the machine-learning conveniences (eg cross-validation) directly on the dataframe >>> from sklearn.model_selection import cross_val_score >>> cross_val_score(learner, df, y) array([0.89370447, 0.89279068, 0.92282557, 0.92319094, 0.92162666]) transformer = TableVectorizer() Making encoding complex dataframes easy

Behind the hood, the work is done by the skrub.TableVectorizer(), a scikit-learn compatible transformer that facilitates combining multiple transformations on the different columns of a dataframe. The TableVectorizer is not new in the 0.2.0 release, but we have completely revamped its internals to cover really well edge cases. Indeed, one challenge is to make sure that nothing different or strange happens at test time. Actually, enforcing consistency between train-time and test-time transformation is the real value of skrub compared to using pandas or polars to do transformation.

Increasing support of polars Short-term goal of optimized support for pandas and polars

We have implemented a new mechanism for supporting both pandas and polars. It has not been applied on all the codebase, hence the support is still imperfect. However, we are seeing increasing support for polars in skrub, and our goal in the short term is to provide rock-solid polar support.

Try skrub out! It’s still young, but in mind opinion, it provides a lot of value to tabular learning.

#636 – JULY 2, 2024
View in Browser »

Build a GUI Calculator With PyQt and Python

In this video course, you’ll learn how to create graphical user interface (GUI) applications with Python and PyQt. Once you’ve covered the basics, you’ll build a fully functional desktop calculator that can respond to user events with concrete actions.
REAL PYTHON course

Satellites Spotting Ships

Umbra Space has released a data set consisting of satellite based radar images of shipping. This article from Mark shows you how to grab the data, visualize, and annotate it.
MARK LITWINTSCHIK

Discover the Power of Observability With Pydantic Logfire

Logfire, by the makers of Pydantic, is an observability platform that will help you understand your app’s behavior with less code and time. Built on OpenTelemetry, it features user-friendly dashboards, SQL querying, and Python-specific integrations. Get started today →
PYDANTIC sponsor

Modern Good Practices for Python Development

This is a very detailed list of best practices for developing in Python. It includes tools, language features, application design, which libraries to use an more.
STUART ELLIS

PSF Board Candidates for 2024

PYTHON SOFTWARE FOUNDATION

Python 3.13.0 Beta 3 Released

CPYTHON DEV BLOG

Django 5.1 Beta 1 Released

DJANGO SOFTWARE FOUNDATION

PyBay 2024 Call for Proposals

PYBAY

Articles & Tutorials Build a Guitar Synthesizer: Play Musical Tablature in Python

In this tutorial, you’ll build a guitar synthesizer using the Karplus-Strong algorithm in Python. You’ll model vibrating strings, simulate strumming techniques, read musical notation and tablature, and apply audio effects. By the end, you’ll have created a digital guitar that can play any song. This tutorial was also discussed on Real Python Podcast Episode #210.
REAL PYTHON

Python’s Security Model After the xz-utils Backdoor

The backdoor introduced to the xz-utils compression project through social engineering was one of the topics at the Python Language Summit. Participants discussed what can be done to prevent similar social engineering attacks on the Python source.
PYTHON SOFTWARE FOUNDATION

Authentication Your Whole Team Will Love

“With PropelAuth, I think I’ve spent about a day – total – on auth over the past year.” PropelAuth is easy to integrate and provides all the tools your team needs to manage your users - dashboards, user insights, impersonation, SSO and more →
PROPELAUTH sponsor

Running Prettier Against Django or Jinja Templates

“Prettier” is a JavaScript based linting tool for templates. For folks not familiar with the world of npm, it can be a bit daunting to get it going. Simon fiddled with it so you don’t have to and posted how he got it working on his system.
SIMON WILLISON

Write Less Code, You Must

An often overlooked aspect of software development is architecture at the module & function level. It is important to write code that is simple and easy to move from one place to another.
DAVIDVUJIC.BLOGSPOT.COM • Shared by David Vujic

Quickstart for Playing With LLMs Locally

This is a simple, quick guide to getting started running LLMs on your local computer. It covers the basics of the powerful libraries Ollama and LangChain for controlling these AI models.
JOSHUA COOK • Shared by Joshua Cook

Under the Hood of Python’s set Data Structure

This tutorial covers hash tables, collision handling, performance optimization and how it relates to the implementation of the set data structure in Python.
ABHINAV UPADHYAY

Ways to Have an Atomic Counter in Django

Keeping a counter across objects in Django means having to be careful about race conditions. This article outlines several approaches to the problem.
GONÇALO VALÉRIO

Saying Thanks to Open Source Maintainers

Brett talks about the different ways you can support the many maintainers of open source projects, and often times just saying “thanks” means a lot.
BRETT CANNON

A Guide to Python’s Weak References

Learn all about weak references in Python: reference counting, garbage collection, and practical uses of the weakref module
MARTIN HEINZ • Shared by Martin Heinz

Creating Great README Files for Your Python Projects

In this tutorial, you’ll learn how to create, organize, and format high-quality README files for your Python projects.
REAL PYTHON

Get Terminal Size

This quick TIL post from Rodrigo shows you how to get information about the terminal size from the shutil module.
RODRIGO GIRÃO SERRÃO

A Complete Guide to pytest Fixtures

Learn how to use pytest fixtures for writing maintainable and isolated tests.
STANLEY ULILI

Projects & Code burr: Build Apps That Make Decisions (Chatbots, Agents, etc)

GITHUB.COM/DAGWORKS-INC

Lazy f-strings

GITHUB.COM/POMPONCHIK • Shared by pomponchik

oxo: Security Scanning Orchestrator

GITHUB.COM/OSTORLAB

jax: Composable Transformations of Python+NumPy Programs

GITHUB.COM/GOOGLE

dbt-utils: Utility Functions for DBT Projects

GITHUB.COM/DBT-LABS

Events Weekly Real Python Office Hours Q&A (Virtual)

July 3, 2024
REALPYTHON.COM

Canberra Python Meetup

July 4, 2024
MEETUP.COM

Sydney Python User Group (SyPy)

July 4, 2024
SYPY.ORG

EuroPython 2024

July 8 to July 15, 2024
EUROPYTHON.EU

SciPy US 2024

July 8 to July 14, 2024
SCIPY.ORG

PyCon Nigeria 2024

July 10 to July 14, 2024
PYCON.ORG

Happy Pythoning!
This was PyCoder’s Weekly Issue #636.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

For a while now I'd see an annoying problem when trying to analyze data. I would be trying to import into numpy an innocuous-looking data file like this:

# image x y z temperature image1.png 1 2 5 34 image2.png 3 4 1 35

As usual, I would be using vnlog.slurp() (a thin wrapper around numpy.loadtxt()) to read this in, but that doesn't work: the image filenames aren't parseable as numerical values. Up until now I would work around this by using the suprocess module to fork off a vnl-filter -p !image and then slurp that, but it's a pain and slow and has other issues. I just solved this conclusively using the numpy structured dtypes. I can now do this:

dtype = np.dtype([ ('image', 'U16'), ('x y z', int, (3,)), ('temperature', float), ]) arr = vnlog.slurp("data.vnl", dtype=dtype)

This will read the image filename, the xyz points and the temperature into different sub-arrays, with different types each. Accessing the result looks like this:

print(arr['image']) ---> array(['image1.png', 'image2.png'], dtype='<U16') print(arr['x y z']) ---> array([[1, 2, 5], [3, 4, 1]]) print(arr['temperature']) ---> array([34., 35.])

Notes:

• The given structured dtype defines both how to organize the data, and which data to extract. So it can be used to read in only a subset of the available columns. Here I could have omitted the temperature column, for instance
• Sub-arrays are allowed. In the example I could say either dtype = np.dtype([ ('image', 'U16'), ('x y z', int, (3,)), ('temperature', float), ])

  or

  dtype = np.dtype([ ('image', 'U16'), ('x', int), ('y', int), ('z', int), ('temperature', float), ])

  The latter would read x, y, z into separate, individual arrays. Sometime we want this, sometimes not.

• Nested structured dtypes are not allowed. Fields inside other fields are not supported, since it's not clear how to map that to a flat vnlog legend
• If a structured dtype is given, slurp() returns the array only, since the field names are already available in the dtype

We still do not support records with any null values (-). This could probably be handled with the converters kwarg of numpy.loadtxt(), but that sounds slow. I'll look at that later.

This is available today in vnlog 1.38.

Dear Debian community,

Statement on Daniel Pocock

The Debian project has successfully taken action to secure its trademarks and interests worldwide, as detailed in our press statement. I would like to personally thank everyone in the community who was involved in this process. I would have loved for you all to have spent your volunteer time on more fruitful things.

Debian Boot team might need help

I think I've identified the issue that finally motivated me to contact our teams: for a long time, I have had the impression that Debian is driven by several "one-person teams" (to varying extents of individual influence and susceptibility to burnout). As DPL, I see it as my task to find ways to address this issue and provide support.

I received private responses from Debian Boot team members, which motivated me to kindly invite volunteers to some prominent and highly visible fields of work that you might find personally challenging. I recommend subscribing to the Debian Boot mailing list to see where you might be able to provide assistance.

/usrmerge

Helmut Grohne confirmed that the last remaining packages shipping aliased files inside the package set relevant to debootstrap were uploaded. Thanks a lot for Helmut and all contributors that helped to implement DEP17.

Contacting more teams

I'd like to repeat that I've registered a BoF for DebConf24 in Busan with the following description:

This BoF is an attempt to gather as much as possible teams inside Debian to exchange experiences, discuss workflows inside teams, share their ways to attract newcomers etc.

Each participant team should prepare a short description of their work and what team roles (“openings”) they have for new contributors. Even for delegated teams (membership is less fluid), it would be good to present the team, explain what it takes to be a team member, and what steps people usually go to end up being invited to participate. Some other teams can easily absorb contributions from salsa MRs, and at some point people get commit access. Anyway, the point is that we work on the idea that the pathway to become a team member becomes more clear from an outsider point-of-view.

I'm lagging a bit behind my team contacting schedule and will not manage to contact every team before DebConf. As a (short) summary, I can draw some positive conclusions about my efforts to reach out to teams. I was able to identify some issues that were new to me and which I am now working on. Examples include limitations in Salsa and Salsa CI. I consider both essential parts of our infrastructure and will support both teams in enhancing their services.

Some teams confirmed that they are basically using some common infrastructure (Salsa team space, mailing lists, IRC channels) but that the individual members of the team work on their own problems without sharing any common work. I have also not read about convincing strategies to attract newcomers to the team, as we have established, for instance, in the Debian Med team.

DebConf attendance

The amount of money needed to fly people to South Korea was higher than usual, so the DebConf bursary team had to make some difficult decisions about who could be reimbursed for travel expenses. I extended the budget for diversity and newcomers, which enabled us to invite some additional contributors. We hope that those who were not able to come this year can make it next year to Brest or to MiniDebConf Cambridge or Toulouse

tag2upload

On June 12, Sean Whitton requested comments on the debian-vote list regarding a General Resolution (GR) about tag2upload. The discussion began with technical details but unfortunately, as often happens in long threads, it drifted into abrasive language, prompting the community team to address the behavior of an opponent of the GR supporters. After 560 emails covering technical details, including a detailed security review by Russ Allbery, Sean finally proposed the GR on June 27, 2024 (two weeks after requesting comments).

Firstly, I would like to thank the drivers of this GR and acknowledge the technical work behind it, including the security review. I am positively convinced that Debian can benefit from modernizing its infrastructure, particularly through stronger integration of Git into packaging workflows.

Sam Hartman provided some historical context [1], [2], [3], [4], noting that this discussion originally took place five years ago with no results from several similarly lengthy threads. My favorite summary of the entire thread was given by Gregor Herrmann, which reflects the same gut feeling I have and highlights a structural problem within Debian that hinders technical changes. Addressing this issue is definitely a matter for the Debian Project Leader, and I will try to address it during my term.

At the time of writing these bits, a proposal from ftpmaster, which is being continuously discussed, might lead to a solution. I was also asked to extend the GR discussion periods which I will do in separate mail.

Talk: Debian GNU/Linux for Scientific Research

I was invited to have a talk in the Systems-Facing Track of University of British Columbia (who is sponsoring rack space for several Debian servers). I admit it felt a bit strange to me after working more than 20 years for establishing Debian in scientific environments to be invited to such a talk "because I'm DPL". 😉

Kind regards Andreas.

GNU direvent version 5.4 is available for download.

New in this version:

Simultaneous execution limits

It is possible to limit number of command instances that are allowed to run simultaneously for a particular watcher.  This is done using
the max-instances statement in watcher section.

Restore the "nowait" default

In previous version, watchers waited for the handler to terminate, unless given the nowait option explicitly.  It is now fixed and nowait is the default, as described in the documentation.

Fix bug in generic to system event translation

Fix sentinel code

In some cases setting the sentinel effectively removed the original watcher.  That happened if the full file name of the original watcher
and its directory part produced the same hash code.

GNU dbm version 1.24 is available for download. New in this version:

New gdbm_load option: --update

The --update (-U) option instructs gdbm_load to update an existing database.

Fix semantics of gdbm_load -r

The --replace (-r) is valid only when used together with --update.

Use getline in gdbmtool shell

New function: gdbm_load_from_file_ext

In contrast to gdbm_load and gdbm_load_from_file, which derive the value of the flag parameter for gdbm_open from the value of their replace argument, this function allows the caller to specify it explicitly. 

Bugfixes

• Fix binary dump format for key and/or data of zero size (see bug 656)
• Fix location tracking and recover command in gdbtool (see bug 566)
• Fix possible buffer underflow in gdbmload.
• Ensure any padding bytes in avail_elem structure are filled with 0. This fixes debian bug 1031276.
• Improve the documentation.

This is the first article of a 5-episode blog post series written by Guido Berhörster, member of staff at my company Fre(i)e Software GmbH. Thanks, Guido for being on the Polis project.

Enjoy the read on the work Guido has been doing over the past months,
Mike

A team lead by Raoul Kramer/BetaBreak is currently adapting Polis for evaluation and testing by several Dutch provincial governments and central government ministries. Guido Berhörster (author of this article) who is an employee at Fre(ie) Software GmbH has been involved in this project as the main software developer. This series of blog posts describes how and why Polis was initially modified and adapted, what issues the team ran into and how this ultimately lead them to start a new Open Source project called Particiapp for accelerating the development of alternative Polis frontends compatible to but independent from the upstream project.

Table of Contents of the Blog Post Series

1. Introduction (this article)
2. Initial evaluation and adaptation
3. Issues extending Polis and adjusting our goals
4. Creating (a) new frontend(s) for Polis
5. Current status and roadmap

Polis - The Introduction What is Polis?

Polis is a platform for participation which helps to gather, analyze and understand viewpoints of large groups of participants on complex issues. In practical terms participants take part in “conversations” on a predefined topic by voting on statements or submitting their own statements (referred to as “comments” in Polis) for others to vote on1.

Through statistical analysis including machine learning participants are sorted into groups based on similarities in voting behavior. In addition, group-informed and overall consensus statements are identified and presented to participants in real-time. This allows for participants to react to and refine statements and either individually or through a predefined process to come to an overall consensus.

Furthermore, the order in which statements are presented to participants is influenced by a complex weighting system based on a number of factors such as variance, recency, and frequency of skipping. This so called “comment routing” is intended to facilitate a meaningful contribution of participants without requiring them to vote on each of a potentially huge number of statements 2.

Polis open-ended nature sets it apart from online surveys using pre-defined questions and allows its users to gather a more accurate picture of the public opinion. In contrast to a discussion forum or comment section where participants directly reply to each other, it discourages unproductive behavior such as provocations or personal attacks by not presenting statements in chronological order in combination with voting. Finally, its “comment routing” is intended to provide scalability towards a large number of participants which generate a potentially large number of statements.

The project was developed and is maintained by The Computational Democracy Project, a USA-based non-profit organization which provides a hosted version and offers related services. It is also released as Open Source software under the AGPL 3.0 license.

Polis has been used in a variety of different contexts as part of broader political processes facilitating broader political participation and opinion-forming, and gathering feedback and creative input.

Use of Polis in Taiwan

One prominent use case of Polis is its adoption as part of the vTaiwan participatory governance project. Established by the g0v civic tech community in the wake of the 2014 mass protests by the Sunflower movement, the vTaiwan project enables consultations on proposed legislation among a broad range of stakeholders including government ministries, lawmakers, experts, interest groups, civil society as well as the broader public. Although the resulting recommendations are non-binding, they exert pressure on the government to take action and recommendations have been adopted into legislation.345

vTaiwan uses Polis for large-scale online deliberations as part of a structured participation process. These deliberations take place after identifying and involving stakeholders and experts and providing through information about the topic at hand to the public. Citizens are then given the opportunity to vote on statements or provide alternative proposals which allows for the refinement of ideas and ideally leads to a consensus at the end. The results of these online deliberations are then curated, discussed in publicly broadcast face-to-face meetings which ultimately produce concrete policy recommendations. vTaiwan has in numerous cases given impulses resulting in government action and provided significant input e.g. on legislation regulating Uber or technological experiments by Fintech startups.35

See also

• Polis Case Studies
• BBC Click: Can Taiwan Reboot Democracy?
• Polis: Scaling Deliberation by Mapping High Dimensional Opinion Spaces

1. https://compdemocracy.org/Polis/ ↩

2. https://compdemocracy.org/comment-routing/ ↩

3. https://info.vtaiwan.tw/ ↩ ↩

4. https://www.theguardian.com/world/2020/sep/27/taiwan-civic-hackers-polis-consensus-social-media-platform ↩

5. https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/ ↩ ↩

In programming, the term constant refers to names representing values that don’t change during a program’s execution. Constants are a fundamental concept in programming, and Python developers use them in many cases. However, Python doesn’t have a dedicated syntax for defining constants. In practice, Python constants are just variables that never change.

To prevent programmers from reassigning a name that’s supposed to hold a constant, the Python community has adopted a naming convention: use uppercase letters. For every Pythonista, it’s essential to know what constants are, as well as why and when to use them.

In this video course, you’ll learn how to:

• Properly define constants in Python
• Identify some built-in constants
• Use constants to improve your code’s readability, reusability, and maintainability
• Apply different approaches to organize and manage constants in a project
• Use several techniques to make constants strictly constant in Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

A comprehensive analysis by The DropTimes (TDT) examines CMS usage across 8,134 non-profit and charity organizations in the United States. The findings highlight a dominant preference for Free and Open Source Software (FOSS) solutions, with WordPress being the most widely used CMS. Drupal also shows significant usage, especially among top-ranked websites.

At DrupalCon Portland, Dries announced Starshot during his State of Drupal presentation. Part of Starshot is the idea that we have Drupal CMS and Core. The big difference is that the Drupal CMS offering comes with standard contributed modules used by almost every existing Drupal build. Then, Dries showed a proposed wireframe for the Drupal.org download page. One of the first things I noticed was the "Launch" in the Drupal CMS section. I wasn't sure about the end goal: 

Low-code solutions are a quick and adaptable substitute for the conventional development cycle as digitization picks up speed. 

Low-code developments are quick, flexible, and affordable but perform better at certain tasks than others. Businesses usually use them mainly in these particular areas- Workflow Automation, Customer Relationship Management (CRM), Enterprise Resource Planning (ERP), Data Integration, Mobile and Web Applications, Business Process Management (BPM), Internal Tools and Dashboards, and E-commerce Platforms.

Before this, let’s delve into the concept of low-code development to gain insight into how it benefits businesses.