Feeds

"I will turn every stone, and I will turn them on time," notes a very confident Janne Kalliola. Over a decade ago, Janne's idea to host a dinner for Drupal business leaders at DrupalCon Prague 2013 sparked the Drupal Business Survey, now in its ninth year. In an interview with Alka Elizabeth, Janne discusses the survey's origins, its evolution, and the key players involved. He shares the benefits for the Drupal community, significant changes, and expectations for this year's survey, highlighting its role in gathering vital insights for Drupal businesses and agencies worldwide.

If you want to get better at something, often times the path is pretty clear. If you get better at swimming, you go to the pool and practice your strokes and put in time doing the laps. If you want to get better at mountain biking, hit the trails and work on drills focusing on different aspects of riding. You can do the same for programming. Reuven Lerner is back on the podcast to talk about his book Pandas Workout. We dive into strategies for learning Pandas and Python as well as some of his workout exercises.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/scalablepath'>Scalable Path</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Reuven Lerner on Twitter</b>: <a href="https://twitter.com/reuvenmlerner" target="_blank" rel="noopener">@reuvenmlerner</a><br/> <b>Pandas Workout Book</b>: <a href="https://www.manning.com/books/pandas-workout" target="_blank" rel="noopener">manning.com</a><br/> <b>Bamboo Weekly: Solar eclipse</b>: <a href="https://www.bambooweekly.com/bw-61-solar-eclipse/" target="_blank" rel="noopener">bambooweekly.com</a><br/> <b>Bamboo Weekly: Avocado hand</b>: <a href="https://www.bambooweekly.com/bw-73-avocado-hand/" target="_blank" rel="noopener">bambooweekly.com</a><br/> <b>Scaling data science across Python and R</b>: <a href="https://talkpython.fm/episodes/show/236/scaling-data-science-across-python-and-r" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=x1-t4xRS9XA" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/471/learning-and-teaching-pandas" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

LDAP and Active Directory as Python API Services 2024-07-22, by Dariusz Suchojad

LDAP and Active Directory often play key a role in the management of a company's network resources yet it is not always very convenient to query a directory directly using the LDAP syntax and protocol that few people truly specialize in. This is why in this article we are using Zato to offer a REST API on top of directory services so that API clients can use REST and JSON instead.

Installing Zato

Start off by installing Zato - if you are not sure what to choose, pick the Docker Quickstart option and this will set up a working environment in a few minutes.

Creating connections

Once Zato is running, connections can be easily created in its Dashboard (by default, http://127.0.0.1:8183). Navigate to Connections -> Outgoing -> LDAP ..

.. and then click Create a new connection which will open a form as below:

The same form works for both regular LDAP and Active Directory - in the latter case, make sure that Auth type is set to NTLM.

The most important information is:

• User credentials
• Authentication type
• Server or servers to connect to

Note that if authentication type is not NTLM, user credentials can be provided using the LDAP syntax, e.g. uid=MyUser,ou=users,o=MyOrganization,dc=example,dc=com.

Right after creating a connection be sure to set its password too - the password asigned by default is a randomly generated one.

Pinging

It is always prudent to ping a newly created connection to ensure that all the information entered was correct.

Note that if you have more than one server in a pool then the first available one of them will be pinged - it is the whole pool that is pinged, not a particular part of it.

Active Directory as a REST service

As the first usage example, let's create a service that will translate JSON queries into LDAP lookups - given username or email the service will basic information about the person's account, such as first and last name.

Note that the conn object returned by client.get() below is capable of running any commands that its underlying Python library offers - in this case we are only using searches but any other operation can also be used, e.g. add or modify as well.

# -*- coding: utf-8 -*- # stdlib from json import loads # Bunch from bunch import bunchify # Zato from zato.server.service import Service # Where in the directory we expect to find the user search_base = 'cn=users, dc=example, dc=com' # On input, we are looking users up by either username or email search_filter = '(&(|(uid={user_info})(mail={user_info})))' # On output, we are interested in username, first name, last name and the person's email query_attributes = ['uid', 'givenName', 'sn', 'mail'] class ADService(Service): """ Looks up users in AD by their username or email. """ class SimpleIO: input_required = 'user_info' output_optional = 'message', 'username', 'first_name', 'last_name', 'email' response_elem = None skip_empty_keys = True def handle(self): # Connection name to use conn_name = 'My AD Connection' # Get a handle to the connection pool with self.out.ldap[conn_name].conn.client() as client: # Get a handle to a particular connection with client.get() as conn: # Build a filter to find a user by user_info = self.request.input['user_info'] user_filter = search_filter.format(user_info=user_info) # Returns True if query succeeds and has any information on output if conn.search(search_base, user_filter, attributes=query_attributes): # This is where the actual response can be found response = conn.entries # In this case, we expect at most one user matching input criteria entry = response[0] # Convert it to JSON for easier handling .. entry = entry.entry_to_json() # .. and load it from JSON to a Python dict entry = loads(entry) # Convert to a Bunch instance to get dot access to dictionary keys entry = bunchify(entry['attributes']) # Now, actually produce a JSON response. For simplicity's sake, # assume that users have only one of email or other attributes. self.response.payload.message = 'User found' self.response.payload.username = entry.uid[0] self.response.payload.first_name = entry.givenName[0] self.response.payload.last_name = entry.sn[0] self.response.payload.email = entry.mail[0] else: # No business response = no such user found self.response.payload.message = 'No such user'

After creating a REST channel, we can invoke the service from command line, thus confirming that we can offer the directory as a REST service:

$ curl "localhost:11223/api/get-user?user_info=MyOrganization\\MyUser" ; echo { "message": "User found", "username": "MyOrganization\\MyUser", "first_name": "First", "last_name": "Last", "email": "address@example.com" } $ More resources

➤ Python API integration tutorial
➤ What is a Network Packet Broker? How to automate networks in Python?
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?

More blog posts➤

Not everything made by KDE uses C++. This is probably obvious to some people, but it’s worth mentioning nevertheless. And I don’t mean this as just “well duh, KDE uses QtQuick which is written with C++ and QML”. I also don’t mean this as “well duh, Qt has a lot of bindings to other languages”. I mean explicitly “KDE has tools written primarily in certain languages and specialized formats”. Note that I said “specialized formats”.

YouTube without YouTube Shorts About • Blog • Newsletter • Links YouTube without YouTube Shorts

Published 2024-07-22 by Seth Larson
Reading time: minutes

Back in October 2023 I wrote about how “For You” wasn't for me, and my dislike for infinitely-scrolling algorithmically-suggested content. In that post I mentioned that I removed YouTube from my phone due to the “YouTube Shorts” feature which functions similar to TikTok.

That didn't last long. I enjoy many creators on YouTube, so eventually I ended up reinstalling the app. But my problem persisted: I would find myself opening up YouTube and occasionally scrolling through Shorts for longer than I wanted to.

What I wanted was YouTube without Shorts, but there didn't seem to be any way to disable them in the app. I was complaining about this to a friend, who out of nowhere mentioned that disabling your “Watch History” will disable the YouTube Shorts infinite feed. My mind was blown! 🤯

No more YouTube Shorts! 🥳

I disabled my Watch History and deleted my existing history and sure enough, no Shorts! (also no home page, which I can also live without). Here's how to do it:

• Open YouTube app, select “You” (bottom-right on iOS)
• Settings cog in the top-right
• Manage all history, wait for the Google Account page to load
• Select “Turn Off” for YouTube History
• Select “Manage History” and delete all history
• Go back to the home page and see that Shorts no longer loads!
• Might have to restart the app, if I remember correctly I had to?

Lesson learned: complain about hyper-specific problems to your friends more often.

Thanks for reading! ♡ Did you find this article helpful and want more content like it? Get notified of new posts by subscribing to the RSS feed or the email newsletter.

This work is licensed under CC BY-SA 4.0

Here comes the 3rd article of the 5-episode blog post series on Polis, written by Guido Berhörster, member of staff at my company Fre(i)e Software GmbH.

Enjoy also this read on Guido's work on Polis,
Mike

Table of Contents of the Blog Post Series

1. Introduction
2. Initial evaluation and adaptation
3. Issues extending Polis and adjusting our goals (this article)
4. Creating (a) new frontend(s) for Polis
5. Current status and roadmap

Polis - Issues extending Polis and adjusting our Goals

After the initial implementation of limited branding support, user feedback and the involvement of an UX designer lead to the conclusion that we needed more far-reaching changes to the user interface in order to reduce visual clutter, rearrange and improve UI elements, and provide better integration with the websites in which conversations are embedded.

Challenges when visualizing Data in Polis

Polis visualizes groups using a spatial projection of users based on similarities in voting behavior and places them in two to five groups using a clustering algorithm. During our testing and evaluation users were rarely able to interpret the visualization and often intuitively made incorrect assumptions e.g. by associating the filled area of a group with its significance or size. After consultation with a member of the Multi-Agent Systems (MAS) Group at the University of Groningen we chose to temporarily replace the visualization offered by Polis with simple bar charts representing agreement or disagreement with statements of a group or the majority. We intend to revisit this and explore different forms of visualization at a later point in time.

The different factors playing into the weight attached to statements which determine the pseuodo-random order in which they are presented for voting (“comment routing”) proved difficult to explain to stakeholders and users and the admission of the ad-hoc and heuristic nature of the used algorithm1 by Polis’ authors lead to the decision to temporarily remove this feature. Instead, statements should be placed into three groups, namely

1. metadata questions,
2. seed statements,
3. and participant statements

Statements should then be sorted by group but in a fully randomized order within the group so that metadata questions would be presented before seed statements which would be presented before participant’s statements. This simpler method was deemed sufficient for the scale of our pilot projects, however we intend to revisit this decision and explore different methods of “comment routing” in cooperation with our scientific partners at a later point in time.

An evaluation of the requirements for implementing mandatory authentication and adding support for additional authentication methods to Polis showed that significant changes to both the administration and participation frontend were needed due to a lack of an abstraction layer or extension mechanism and the current authentication providers being hardcoded in many parts of the code base.

A New Frontend is born: Particiapp

Based on the implementation details of the participation frontend, the invasive nature of the changes required, and the overhead of keeping up with active upstream development it became clear that a different, more flexible approach to development was needed. This ultimately lead to the creation of Particiapp, a new Open Source project providing the building blocks and necessary abstraction layers for rapid protoyping and experimentation with different fontends which are compatible with but independent from Polis.

1. Small, Christopher T., Bjorkegren, Michael, Erkkilä, Timo, Shaw, Lynette and Megill, Colin (2021). Polis: Scaling deliberation by mapping high dimensional opinion spaces. Recerca. Revista de Pensament i Anàlisi, 26(2), pp. 1-26. ↩

Make sure you commit anything you want to end up in the KDE Gear 24.08
releases to them

Next Dates  

• July 25, 2024: 24.08 Freeze and Beta (24.07.80) tag & release
• August  8, 2024: 24.08 RC (24.07.90) Tagging and Release
• August 15, 2024: 24.08 Tagging
• August 22, 2024: 24.08 Release

https://community.kde.org/Schedules/KDE_Gear_24.08_Schedule

The recent issue of Windows security software killing computers has reminded me about the issue of management software for Dell systems. I wrote policy for the Dell management programs that extract information from iDRAC and store it in Linux. After the break I’ve pasted in the policy. It probably needs some changes for recent software, it was last tested on a PowerEdge T320 and prior to that was used on a PowerEdge R710 both of which are old hardware and use different management software to the recent hardware. One would hope that the recent software would be much better but usually such hope is in vain. I deliberately haven’t submitted this for inclusion in the reference policy because it’s for proprietary software and also it permits many operations that we would prefer not to permit.

The policy is after the break because it’s larger than you want on a Planet feed. But first I’ll give a few selected lines that are bad in a noteworthy way:

1. sys_admin means the ability to break everything
2. dac_override means break Unix permissions
3. mknod means a daemon creates devices due to a lack of udev configuration
4. sys_rawio means someone didn’t feel like writing a device driver, maintaining a device driver for DKMS is hard and getting a driver accepted upstream requires writing quality code, in any case this is a bad sign.
5. self:lockdown is being phased out, but used to mean bypassing some integrity protections, that would usually be related to sys_rawio or similar.
6. dev_rx_raw_memory is bad, reading raw memory allows access to pretty much everything and execute of raw memory is something I can’t imagine a good use for, the Reference Policy doesn’t use this anywhere!
7. dev_rw_generic_chr_files usually means a lack of udev configuration as udev should do that.
8. storage_raw_write_fixed_disk shouldn’t be needed for this sort of thing, it doesn’t do anything that involves managing partitions.

Now without network access or other obvious ways of remote control this level of access while excessive isn’t necessarily going to allow bad things to happen due to outside attack. But if there are bugs in the software there’s nothing to stop it from giving the worst results.

allow dell_datamgrd_t self:capability { dac_override dac_read_search mknod sys_rawio sys_admin }; allow dell_datamgrd_t self:lockdown integrity; dev_rx_raw_memory(dell_datamgrd_t) dev_rw_generic_chr_files(dell_datamgrd_t) dev_rw_ipmi_dev(dell_datamgrd_t) dev_rw_sysfs(dell_datamgrd_t) storage_raw_read_fixed_disk(dell_datamgrd_t) storage_raw_write_fixed_disk(dell_datamgrd_t) allow dellsrvadmin_t self:lockdown integrity; allow dellsrvadmin_t self:capability { sys_admin sys_rawio }; dev_read_raw_memory(dellsrvadmin_t) dev_rw_sysfs(dellsrvadmin_t) dev_rx_raw_memory(dellsrvadmin_t)

The best thing that Dell could do for their customers is to make this free software and allow the community to fix some of these issues.

Here is dellsrvadmin.te:

policy_module(dellsrvadmin,1.0.0) require {   type dmidecode_exec_t;   type udev_t;   type device_t;   type event_device_t;   type mon_local_test_t; } type dellsrvadmin_t; type dellsrvadmin_exec_t; init_daemon_domain(dellsrvadmin_t, dellsrvadmin_exec_t) type dell_datamgrd_t; type dell_datamgrd_exec_t; init_daemon_domain(dell_datamgrd_t, dell_datamgrd_t) type dellsrvadmin_var_t; files_type(dellsrvadmin_var_t) domain_transition_pattern(udev_t, dellsrvadmin_exec_t, dellsrvadmin_t) modutils_domtrans(dellsrvadmin_t) allow dell_datamgrd_t device_t:dir rw_dir_perms; allow dell_datamgrd_t device_t:chr_file create; allow dell_datamgrd_t event_device_t:chr_file { read write }; allow dell_datamgrd_t self:process signal; allow dell_datamgrd_t self:fifo_file rw_file_perms; allow dell_datamgrd_t self:sem create_sem_perms; allow dell_datamgrd_t self:capability { dac_override dac_read_search mknod sys_rawio sys_admin }; allow dell_datamgrd_t self:lockdown integrity; allow dell_datamgrd_t self:unix_dgram_socket create_socket_perms; allow dell_datamgrd_t self:netlink_route_socket r_netlink_socket_perms; modutils_domtrans(dell_datamgrd_t) can_exec(dell_datamgrd_t, dmidecode_exec_t) allow dell_datamgrd_t dellsrvadmin_var_t:dir rw_dir_perms; allow dell_datamgrd_t dellsrvadmin_var_t:file manage_file_perms; allow dell_datamgrd_t dellsrvadmin_var_t:lnk_file read; allow dell_datamgrd_t dellsrvadmin_var_t:sock_file manage_file_perms; kernel_read_network_state(dell_datamgrd_t) kernel_read_system_state(dell_datamgrd_t) kernel_search_fs_sysctls(dell_datamgrd_t) kernel_read_vm_overcommit_sysctl(dell_datamgrd_t) # for /proc/bus/pci/* kernel_write_proc_files(dell_datamgrd_t) corecmd_exec_bin(dell_datamgrd_t) corecmd_exec_shell(dell_datamgrd_t) corecmd_shell_entry_type(dell_datamgrd_t) dev_rx_raw_memory(dell_datamgrd_t) dev_rw_generic_chr_files(dell_datamgrd_t) dev_rw_ipmi_dev(dell_datamgrd_t) dev_rw_sysfs(dell_datamgrd_t) files_search_tmp(dell_datamgrd_t) files_read_etc_files(dell_datamgrd_t) files_read_etc_symlinks(dell_datamgrd_t) files_read_usr_files(dell_datamgrd_t) logging_search_logs(dell_datamgrd_t) miscfiles_read_localization(dell_datamgrd_t) storage_raw_read_fixed_disk(dell_datamgrd_t) storage_raw_write_fixed_disk(dell_datamgrd_t) can_exec(mon_local_test_t, dellsrvadmin_exec_t) allow mon_local_test_t dellsrvadmin_var_t:dir search; allow mon_local_test_t dellsrvadmin_var_t:file read_file_perms; allow mon_local_test_t dellsrvadmin_var_t:file setattr; allow mon_local_test_t dellsrvadmin_var_t:sock_file write; allow mon_local_test_t dell_datamgrd_t:unix_stream_socket connectto; allow mon_local_test_t self:sem { create read write destroy unix_write }; allow dellsrvadmin_t self:process signal; allow dellsrvadmin_t self:lockdown integrity; allow dellsrvadmin_t self:sem create_sem_perms; allow dellsrvadmin_t self:fifo_file rw_file_perms; allow dellsrvadmin_t self:packet_socket create; allow dellsrvadmin_t self:unix_stream_socket { connectto create_stream_socket_perms }; allow dellsrvadmin_t self:capability { sys_admin sys_rawio }; dev_read_raw_memory(dellsrvadmin_t) dev_rw_sysfs(dellsrvadmin_t) dev_rx_raw_memory(dellsrvadmin_t) allow dellsrvadmin_t dellsrvadmin_var_t:dir rw_dir_perms; allow dellsrvadmin_t dellsrvadmin_var_t:file manage_file_perms; allow dellsrvadmin_t dellsrvadmin_var_t:lnk_file read; allow dellsrvadmin_t dellsrvadmin_var_t:sock_file write; allow dellsrvadmin_t dell_datamgrd_t:unix_stream_socket connectto; kernel_read_network_state(dellsrvadmin_t) kernel_read_system_state(dellsrvadmin_t) kernel_search_fs_sysctls(dellsrvadmin_t) kernel_read_vm_overcommit_sysctl(dellsrvadmin_t) corecmd_exec_bin(dellsrvadmin_t) corecmd_exec_shell(dellsrvadmin_t) corecmd_shell_entry_type(dellsrvadmin_t) files_read_etc_files(dellsrvadmin_t) files_read_etc_symlinks(dellsrvadmin_t) files_read_usr_files(dellsrvadmin_t) logging_search_logs(dellsrvadmin_t) miscfiles_read_localization(dellsrvadmin_t)

Here is dellsrvadmin.fc:

/opt/dell/srvadmin/sbin/.*        --        gen_context(system_u:object_r:dellsrvadmin_exec_t,s0) /opt/dell/srvadmin/sbin/dsm_sa_datamgrd        --        gen_context(system_u:object_r:dell_datamgrd_t,s0) /opt/dell/srvadmin/bin/.*        --        gen_context(system_u:object_r:dellsrvadmin_exec_t,s0) /opt/dell/srvadmin/var(/.*)?                        gen_context(system_u:object_r:dellsrvadmin_var_t,s0) /opt/dell/srvadmin/etc/srvadmin-isvc/ini(/.*)?        gen_context(system_u:object_r:dellsrvadmin_var_t,s0)

Related posts:

1. creating a new SE Linux policy module Creating a simple SE Linux policy module is not difficult....
2. Debian SE Linux policy bug checkmodule -m -o local.mod local.te semodule_package -o local.pp -m local.mod...
3. New SE Linux Policy for Squeeze I have just uploaded refpolicy version 0.2.20100524-1 to Unstable. This...

GNU Parallel 20240722 ('Assange') has been released. It is available for download at: lbry://@GnuParallel:4

Quote of the month:

  parallel is frickin great for launching jobs on multiple
  machines. Ansible and Jenkins and others may be good too but I was
  able to jump right in with parallel.
    -- dwhite21787@reddit
 
New in this release:

• No new features. This is a candidate for a stable release.
• Bug fixes and man page updates.

News about GNU Parallel:

• Scientific Workflows at Scale using GNU Parallel https://web.cvent.com/event/f318e73c-2230-432a-a044-b75625020543/websitePage:afd80266-008e-414b-9f94-2fd9b4dd1924?session=fe79a785-ec60-414c-8d2b-c29208f53d4c&shareLink=true
• Use GNU Parallel to render blender movies distributed by a bunch of nodes https://github.com/tfmoraes/blender_gnu_parallel_render
• Lessons Learned from Scaling to Multi-Terabyte Datasets https://v2thegreat.com/2024/06/19/lessons-learned-from-scaling-to-multi-terabyte-datasets/
• Efisiensi Maksimal: Cara Paralelisasi Perintah di CLI Linux https://medium.com/@nfrozi/efisiensi-maksimal-cara-paralelisasi-perintah-di-cli-linux-f4fda3afe2a0
• Introduction to GNU parallel https://datascience.101workbook.org/06-hpc/06-parallel/01-intro-to-gnu-parallel/#gsc.tab=0

GNU Parallel - For people who live life in the parallel lane.

If you like GNU Parallel record a video testimonial: Say who you are, what you use GNU Parallel for, how it helps you, and what you like most about it. Include a command that uses GNU Parallel if you feel like it.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

For example you can run this to convert all jpeg files into png and gif files and have a progress bar:

  parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif

Or you can generate big, medium, and small thumbnails of all jpeg files in sub dirs:

  find . -name '*.jpg' |
    parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/} :::: - ::: 50 100 200

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with:

    $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
       fetch -o - http://pi.dk/3 ) > install.sh
    $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
    12345678 883c667e 01eed62f 975ad28b 6d50e22a
    $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
    cc21b4c9 43fd03e9 3ae1ae49 e28573c0
    $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
    79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
    fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
    $ bash install.sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your command line will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2018): GNU Parallel 2018, March 2018, https://doi.org/10.5281/zenodo.1146014.

If you like GNU Parallel:

• Give a demo at your local user group/team/colleagues
• Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
• Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
• Request or write a review for your favourite blog or magazine
• Request or build a package for your favourite distribution (if it is not already there)
• Invite me for your next conference

If you use programs that use GNU Parallel for research:

• Please cite GNU Parallel in you publications (use --citation)

If GNU Parallel saves you money:

• (Have your company) donate to FSF https://my.fsf.org/donate/

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

It has been three weeks since the start of the OSPP project, during which my project has made some progress.

Week 1, July 1st to July 7th

In the first week of the project, with the help of my mentor, I first set up a suitable development environment and identified Blinken as the first application to be migrated for the project. In addition to this, I also set up this project's blog, which has now been included in KDE Planet.

Currently, I am using a development environment on a VirtualBox virtual machine running Fedora Workstation 40. With this setup, I can compile and build KDE applications and perform Qt development.

For building the KDE development environment, it is recommended to usekdesrc-build provided by KDE official, and it is also recommended to develop under KDE Neon system, which can be done using docker or virtual machines

A straightforward method for setting up Qt Android development environment is to use[Qt Online Installer](Get and Install Qt | Qt 6.7) and Qt Creator. After installing Qt Creator, navigate to Editing -> Preferences -> Devices -> Android to select the necessary development kit. Qt Creator will automatically download the required SDK and NDK.

Week 2, July 8th to July 14th

Upon my mentor's suggestion, I developed a simple Tic-Tac-Toe QML game as a practice exercise. This game uses QML to create a simple interface and employs a C++ class to handle game logic, which will also be the architecture for the upcoming game migration.

The current features implemented in the game are:

1. Multilingual localization support
2. Unit testing capability
3. Cross-platform compatibility

The Tic-Tac-Toe game has been open-sourced on KDE Invent: hanyang zhang / TicTacToe · GitLab

Supporting localization for QML applications requires the use of Qt's localization tools such as lupdate and lrelease. However, since the project is built with CMake, Qt also provides corresponding CMake methods: qt_add_translations | Qt Linguist Manual

Additionally, I encountered some difficulties while building QML Android applications, as described here: Building and Running QML Android Applications | Blog

Week 3, July 15th to July 21st

During this week, I officially began the migration work for Blinken.

After studying Blinken's source code, I found that the interface of Blinken is drawn by a class named Blinken, which spans over 1000 lines. The drawing logic involves manipulating elements from Blinken.svg and using QPainter for direct drawing.

Unfortunately, QML does not provide built-in support for manipulating SVG images like QtWidgets. Therefore, I split the SVG images into separate files and assembled these elements using QML.

Modifying the UI took longer than I anticipated. However, after a week, I have nearly completed the interface drawing for Blinken. Next steps involve refining some page details and migrating Blinken's logic over.

It's worth mentioning that QML does not provide a non-rectangular MouseArea for use, requiring the creation of a custom class to achieve this functionality. Fortunately, an example for this existed in earlier versions of the documentation: maskedmousearea example. Although this example seems to have been removed in the latest version, it should still be feasible to implement based on reference.

DHT Technical Specification Milestone 5

We are happy to announce the completion of milestone 5 for the DHT specification. The general objective is to provide a detailed and comprehensive guide for implementors of the GNUnet DHT "R 5 N". As part of this milestone, the specification was updated and interoperability testing conducted. We submitted the draft to the Independent Stream Editor (ISE) who is going to decide if it will be adopted and shepherded through the RFC process.

The current protocol is implemented as part of GNUnet and gnunet-go as announced on the mailing list when the previous implementation milestones were finished .

We again invite any interested party to read the document and provide critical review and feedback. This greatly helps us to improve the protocol and help future implementations. Contact us at the gnunet-developers mailing list .

• Plain text version
• HTML version
• Git sources

This work is generously funded by NLnet as part of their NGI Assure fund .

While list comprehesions in Python don't support the else keyword directly, conditional expressions can be embedded within list comprehension.

Table of contents

1. Do list comprehensions support else?
2. But else does work in comprehensions...
3. Python's ternary operator
4. Conditional expressions in list comprehension
5. Making more readable list comprehensions
6. Avoid complex comprehensions
7. Python's list comprehensions and if-else

Do list comprehensions support else?

This list comprehension isn't valid:

>>> counts = [2, -1, 4, 7, -3, 6] >>> sanitized_counts = [n for n in counts if n > 0 else 0] File "<stdin>", line 1 sanitized_counts = [n for n in counts if n > 0 else 0] ^^^^ SyntaxError: invalid syntax

Python's list comprehensions don't have any support for an else keyword.

But else does work in comprehensions...

But wait! I've seen else …

Read the full article: https://www.pythonmorsels.com/comprehension-with-else/

Yes, that’s right. The title just goes perfectly, with these long weeks! From Week 3 to Week 7! So, first a small back story on the title. On 25th June, my semester exam ended. I was returning to my hometown, and on my way, I got into a bike accident. My right hand got bruised onto the road. Luckily, it didn’t break. But, the sheer pain was enough to make me cry.

These past two weeks were big for Wayland accessibility support, as Nicolas Fella did a lot of work to improve support for sticky keys to equal the state they were in on X11. This work is not complete, but so far it’s taken a big bite out of the open issues! The work lands in a mixture of Plasma 6.1.3 and 6.2.0 (link 1, link 2, link 3, link 4, link 5, link 6).

Beyond this, it’s notable that Plasma developers fixed the five most common Plasma crashes, as well as a bunch of less common ones, which you’ll see mentioned below. These were caused by a mix of Qt regressions and our own code defects. The new automatic crash reporting system has been a huge boon here, allowing us to see which crashes are actually affecting people most. So please do continue to report them!

And of course there’s lots more too. Check it out:

New Features

Elisa now offers a feature to add a track, album, etc. directly to the playlist after current song, playing it next (Jack Hill, Elisa 24.08.0. Link):

System Settings’ Drawing Tablet page now has a calibration tool (Joshua Goins, Plasma 6.2.0. Link)

The default width of Icons-and-Text Task Manager items is now user- configurable, and should also exhibit slightly smarter shrinking behavior as space gets filled up (Kisaragi Hiu, Plasma 6.2.0. Link)

Added support for Plasma’s charging threshold feature to OpenBSD (Rafael Sadowski, Plasma 6.2.0. Link)

UI Improvements

Dolphin now features a lovely super premium user experience for installing Filelight if it’s not already installed (Felix Ernst, Dolphin 24.08.0. Link):

https://i.imgur.com/My2yyWy.mp4

Filelight now has a more illuminating and welcoming homepage (me: Nate Graham, Filelight 24.08.0. Link):

Spectacle has now adopted the “inline messages touch the header” paradigm (me: Nate Graham, Spectacle 24.08.0. Link):

System Settings’ Night Light page now accepts creative custom times, no longer internally clamping them to a set of acceptable times (Vlad Zahorodnii, Plasma 6.1.3. Link)

Improved the smoothness of resizing Plasma widgets (Vlad Zahorodnii, Plasma 6.1.4. Link)

Made it impossible to remove administrator privileges from your current user unless there’s at least one other admin user on the system, to ensure that someone is an admin and can reverse the decision if needed! (Thomas Duckworth, Plasma 6.2.0. Link)

The “launch this app” shortcut for apps on System Settings’ Shortcuts page is now named “Launch”, making its purpose clear (me: Nate Graham, Plasma 6.2.0. Link)

Added some relevant clipboard-related keywords to System Settings’ General Behavior page, so you can find it in a search for things like “paste” and “selection” and stuff like that (Christoph Wolk, Plasma 6.2.0. Link)

Plasma’s Digital Clock widget now uses a typographic space to separate the time from the date when in single-row mode, so that it looks better especially when using a monospace font (Kisaragi Hiu, Plasma 6.2.0. Link)

Removed the filter for different job types from Plasma’s Printers widget, because it never worked (seriously) and apparently no one ever noticed because we didn’t even have any bug reports about it! (Mike Noe, Plasma 6.2.0. Link)

Bug Fixes

Filelight no longer fails to initiate a second scan after leaving the first one and going back to the Overview page (Harald Sitter, Filelight 24.08.0. Link)

Fixed one of the most common ways that Plasma could crash randomly, out of the blue (Akseli Lahtinen, Plasma 6.1.3. Link)

Fixed one of the most common ways that Plasma could crash on Wayland when a screen turns off (Marco Martin, Plasma 6.1.3. Link)

Fixed another one of the apparently many ways that Plasma can crash while handling various types of clipboard data, which appeared to be very common (David Edmundson, Plasma 6.1.3. Link)

Fixed a somewhat common way that Powerdevil could crash when waking the system from sleep with certain types of monitors (Jakob Petsovits, Plasma 6.1.3. Link)

Fixed an issue that caused Spectacle to crash after finishing a screen recording on systems using PipeWire 1.2.0 (Arjen Hiemstra, Plasma 6.1.3. Link)

Fixed an issue that caused Plasma on Wayland to crash when dragging a Task Manager task with no .desktop file associated with it (such as a Steam game) onto the desktop (Akseli Lahtinen, Plasma 6.1.3. Link)

Fixed a recent regression that caused System Tray icons for GTK2 apps to stop responding after the first time they’re clicked. Added an autotest to make sure this doesn’t regress again, too (David Edmundson and Fushan Wen, Plasma 6.1.3. Link)

It’s now possible to set your user avatar image to a file those full path contains “special characters” like spaces, ampersands, etc. (Daniil-Viktor Ratkin, Plasma 6.1.3. Link)

It’s now more reliable to change the date or time using System Settings on a distro not using Systemd (Fabio Bas, Plasma 6.1.3. Link)

Plasma’s RDP server now works properly after again a prior failed connection using a non-H.264-capable client app (Arjen Hiemstra, Plasma 6.1.3. Link)

The Fcitx input method’s “show input method info when switching input focus” setting is now compatible with Plasma’s zoom-out style edit mode, and no longer causes it to exit immediately when the input method popup is shown (Weng Xuetian, Plasma 6.1.3. Link)

Fixed a different somewhat common out-of-the-blue Plasma crash (Méven Car, Plasma 6.1.4. Link)

Fixed a case where KWin could crash on X11 when compositing gets toggled on or off (Vlad Zahorodnii, Plasma 6.1.4. Link)

Fixed a regression in Discover that caused various alerts and information items in certain apps’ description pages to not appear as expected (Aleix Pol Gonzalez, Plasma 6.1.4. Link)

Fixed a regression in Plasma 6.0 that caused the “remove this item” hover icons in KRunner’s history view to be invisible (Ivan Tkachenko, Plasma 6.1.4. Link)

OpenVPN VPNs requiring a challenge-response work again with NetworkManager 1.64 or later (Benjamin Robin, Plasma 6.2.0. Link)

“Text Only” System Monitor sensors on horizontal Plasma panels are once again correctly-sized with centered text (Arjen Hiemstra, Plasma 6.2.0. Link)

“Line Chart” System Monitor sensors now show their legends as expected when placed on a wide vertical Plasma panel (Arjen Hiemstra, Plasma 6.2.0. Link)

When using Plasma’s “Raise maximum volume” setting, it now applies to the per-app volume sliders in the System Tray popup as well as the device volume sliders (Roberto Chamorro, Plasma 6.2.0. Link)

Using KWin’s wide color gamut or ICC profile features no longer increases the transparency of already-transparent windows (Xaver Hugl, Plasma 6.2.0. Link)

Fixed the source of bizarre view corruption in Dolphin introduced in the recently-released Frameworks 6.4 (Vlad Zahorodnii, Frameworks 6.4.1. Link)

Fixed a case where Plasma could crash when trying to re-assign a shortcut for a widget to one already used by something else (Arjen Hiemstra, Frameworks 6.5. Link)

On the “Get New [thing]” dialogs, downloading one file from an entry that includes many now works from the details page (Akseli Lahtinen, Frameworks 6.5. Link)

Fixed another fairly common way that Plasma could crash on Wayland when a screen turns off (David Edmundson, Qt 6.7.3. Link)

Fixed yet another common way that Plasma could crash on Wayland, this time when showing notifications (David Edmundson, Qt 6.7.3. Link)

Fixed a case where Plasma could crash when you trigger the Meta+V keyboard shortcut to open the clipboard history menu over and over again in rapid succession (Vlad Zahorodnii, Qt 6.7.3. Link)

Fixed a case where trying to save a file in a Flatpak or Snap app using the standard Save dialog could fail and cause the saving app to quit instead! (Nicolas Fella, Qt 6.7.3. Link)

Other bug information of note:

• 2 Very high priority Plasma bugs (down from 3 two weeks ago). Current list of bugs
• 31 15-minute Plasma bugs (same as two weeks ago). Current list of bugs
• 185 KDE bugs of all kinds fixed over the last two weeks. Full list of bugs

Performance & Technical

Receiving a Plasma notification no longer blocks KWin’s “direct scan-out” feature, e.g. while playing a game, so it should no longer briefly reduce performance (Xaver Hugl, Plasma 6.1.3. Link)

Improved KWin’s detection for whether triple buffering on Wayland will improve things so that it won’t occasionally turn on and off repeatedly, impairing performance (Xaver Hugl, Plasma 6.1.3. Link)

Plasma’s RDP server is now capable of listening for IPv6 connections (Arjen Hiemstra, Plasma 6.1.3. Link)

KWin now disables 10 bits-per-color (BPC) support for monitors plugged into a dock, as these often limit the signal to 8 BPC but don’t tell KWin, causing issues when KWin tries to enable 10 BPC mode because it thinks it should be possible (Xaver Hugl, Plasma 6.1.4. Link)

KWin now operates with “realtime” capabilities on systemd using musl instead of glibc (Vlad Zahorodnii, Plasma 6.1.4. Link)

Plasma’s RDP server now also works as expected on systems using the OpenH264 video codec (Fabian Vogt, Plasma 6.2.0. Link)

Relevant only for cutting-edge distro-builders: It’s now possible to compile KWin with support for only the Wayland session, so support for X11 apps would be provided exclusively through XWayland (Neal Gompa, Plasma 6.2.0. Link)

Improved performance for everything in KDE that uses KFileItem::isHidden (Volker Krause, Frameworks 6.5. Link)

Created a new WindowStateSaver QML object you can add to apps’ windows to make them remember their size, maximization state (and position, on X11) (Joshua Goins, Frameworks 6.5. Link)

Apps storing their transient state data separately from their persistent configuration data now do so by putting the state config file in the standard XDG state folder of ~/.local/state/ (Harald Sitter, Frameworks 6.5. Link)

Automation & Systematization

Added an autotest to test clearing the clipboard history (Fushan Wen, link)

…And Everything Else

This blog only covers the tip of the iceberg! If you’re hungry for more, check out https://planet.kde.org, where you can find more news from other KDE contributors.

How You Can Help

If you use have multiple systems or an adventurous personality, you can really help us out by installing beta versions of Plasma using your distro’s available repos and reporting bugs. Arch, Fedora, and openSUSE Tumbleweed are examples of great distros for this purpose. So please please do try out Plasma beta versions. It truly does help us! Heck, if you’re very adventurous, live on the nightly repos. I’ve been doing this full-time for 5 years with my sole computer and it’s surprisingly stable.

Does that sound too scary? Consider donating today instead! That helps too.

Otherwise, visit https://community.kde.org/Get_Involved to discover other ways to be part of a project that really matters. Each contributor makes a huge difference in KDE; you are not a number or a cog in a machine! You don’t have to already be a programmer, either. I wasn’t when I got started. Try it, you’ll like it! We don’t bite!

The European Union must keep funding free software

Initially publishead by petites singularités. English translation provided by OW2.

If you want to sign this letter, please publish this text on your website and add yourself or your organization to the table you will find on the original site.

Open Letter to the European Commission

Since 2020, Next Generation Internet (NGI) programmes, part of European Commission's Horizon programme, fund free software in Europe using a cascade funding mechanism (see for example NLnet's calls).

Quite a few of KDE's projects have benefited from NGI's funding, including NeoChat, Kaidan, KDE Connect, KMail, and many others. KDE e.V. is a European non-profit with limited resources that relies on donations, sponsors and funding like that offered by NGI, to push the development of our projects forward.

However, this year, according to the Horizon Europe working draft detailing funding programmes for 2025, we notice that Next Generation Internet is not mentioned any more as part of Cluster 4.

NGI programmes have shown their strength and importance to supporting the European software infrastructure, as a generic funding instrument to fund digital commons and ensure their long-term sustainability. We find this transformation incomprehensible, moreover when NGI has proven efficient and economical to support free software as a whole, from the smallest to the most established initiatives. This ecosystem diversity backs the strength of European technological innovation, and maintaining the NGI initiative to provide structural support to software projects at the heart of worldwide innovation is key to enforce the sovereignty of a European infrastructure. Contrary to common perception, technical innovations often originate from European rather than North American programming communities, and are mostly initiated by small-scaled organizations.

Previous Cluster 4 allocated 27 million euros to:

• "Human centric Internet aligned with values and principles commonly shared in Europe" ;
• "A flourishing internet, based on common building blocks created within NGI, that enables better control of our digital life" ;
• "A structured ecosystem of talented contributors driving the creation of new internet commons and the evolution of existing internet commons".

In the name of these challenges, more than 500 projects received NGI funding in the first 5 years, backed by 18 organisations managing these European funding consortia.

NGI contributes to a vast ecosystem, as most of its budget is allocated to fund third parties by the means of open calls, to structure commons that cover the whole Internet scope - from hardware to application, operating systems, digital identities or data traffic supervision. This third-party funding is not renewed in the current program, leaving many projects short on resources for research and innovation in Europe.

Moreover, NGI allows exchanges and collaborations across all the Euro zone countries as well as "widening countries"1, currently both a success and an ongoing progress, likewise the Erasmus programme before us. NGI also contributes to opening and supporting longer relationships than strict project funding does. It encourages implementing projects funded as pilots, backing collaboration, identification and reuse of common elements across projects, interoperability in identification systems and beyond, and setting up development models that mix diverse scales and types of European funding schemes.

While the USA, China or Russia deploy huge public and private resources to develop software and infrastructure that massively capture private consumer data, the EU can't afford this renunciation. Free and open source software, as supported by NGI since 2020, is by design the opposite of potential vectors for foreign interference. It lets us keep our data local and favors a community-wide economy and know-how, while allowing an international collaboration.

This is all the more essential in the current geopolitical context: the challenge of technological sovereignty is central, and free software allows to address it while acting for peace and sovereignty in the digital world as a whole.

1 As defined by Horizon Europe, widening Member States are Bulgaria, Croatia, Cyprus, the Czech Republic, Estonia, Greece, Hungary, Latvia, Lituania, Malta, Poland, Portugal, Romania, Slovakia and Slovenia. Widening associated countries (under condition of an association agreement) include Albania, Armenia, Bosnia, Feroe Islands, Georgia, Kosovo, Moldavia, Montenegro, Morocco, North Macedonia, Serbia, Tunisia, Turkey and Ukraine. Widening overseas regions are : Guadeloupe, French Guyana, Martinique, Reunion Island, Mayotte, Saint-Martin, The Azores, Madeira, the Canary Islands.

Leonardo and I are happy to announce the release of another maintenance release 0.1.3 of our dtts package which has been on CRAN for a good two years now.

dtts builds upon our nanotime package as well as the beloved data.table to bring high-performance and high-resolution indexing at the nanosecond level to data frames. dtts aims to offers the time-series indexing versatility of xts (and zoo) to the immense power of data.table while supporting highest nanosecond resolution.

This release contains two nice and focussed contributed pull requests. Tomas Kalibera, who as part of R Core looks after everything concerning R on Windows, and then some, needed an adjustment for pending / upcoming R on Windows changes for builds with LLVM which is what Arm-on-Windows uses. We happily obliged: neither Leonardo nor I see much of Windows these decades. (Easy thing to say on a day like today with its crowdstrike hammer falling!) Similarly, Michael Chirico supplied a PR updating one of our tests to an upcoming change at data.table which we are of course happy to support.

The short list of changes follows.

Changes in version 0.1.3 (2024-07-18)

• Windows builds use localtime_s with LLVM (Tomas Kalibera in #16)

• Tests code has been adjusted for an upstream change in data.table tests for all.equal (Michael Chirico in #18 addressing #17)

Courtesy of my CRANberries, there is also a report with diffstat for this release. Questions, comments, issue tickets can be brought to the GitHub repo. If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

In my last post I've described how to use Z3 to find simple local peephole optimization patterns for the integer operations in PyPy's JIT. An example is int_and(x, 0) -> 0. In this post I want to scale up the problem of identifying possible optimizations to much bigger instruction sequences, also using Z3. For that, I am starting with the JIT traces of real benchmarks, after they have been optimized by the optimizer of PyPy's JIT. Then we can ask Z3 to find inefficient integer operations in those traces.

Starting from the optimized traces of real programs has some big advantages over the "classical" superoptimization approach of generating and then trying all possible sequences of instructions. It avoids the combinatorical explosion that happens with the latter approach. Also, starting from the traces of benchmarks or (even better) actual programs makes sure that we actually care about the missing optimizations that are found in this way. And because the traces are analyzed after they have been optimized by PyPy's optimizer, we only get reports for missing optimizations, that the JIT isn't able to do (yet).

The techniques and experiments I describe in this post are again the result of a bunch of discussions with John Regehr at a conference a few weeks ago, as well as reading his blog posts and papers. Thanks John! Also thanks to Max Bernstein for super helpful feedback on the drafts of this blog post (and for poking me to write things in general).

High-Level Approach

The approach that I took works as follows:

• Run benchmarks or other interesting programs and then dump the IR of the JIT traces into a file. The traces have at that point been already optimized by the PyPy JIT's optimizer.
• For every trace, ignore all the operations on non-integer variables.
• Translate every integer operation into a Z3 formula.
• For every operation, use Z3 to find out whether the operation is redundant (how that is done is described below).
• If the operation is redundant, the trace is less efficient than it could have been, because the optimizer could also have removed the operation. Report the inefficiency.
• Minimize the inefficient programs by removing as many operations as possible to make the problem easier to understand.

In the post I will describe the details and show some pseudocode of the approach. I'll also make the proper code public eventually (but it needs a healthy dose of cleanups first).

Dumping PyPy Traces

PyPy will write its JIT traces into the file out if the environment variable PYPYLOG is set as follows:

PYPYLOG=jit-log-opt:out pypy <program.py>

This environment variable works for PyPy, but also for other virtual machines built with RPython.

(This is really a side point for the rest of the blog post, but since the question came up I wanted to clarify it: Operations on integers in the Python program that the JIT is running don't all correspond 1-to-1 with the int_... operations in the traces. The int_... trace operations always operate on machine words. The Python int type supports arbitrarily large integers. PyPy will optimistically try to lower the operations on Python integers into machine word operations, but adds the necessary guards into the trace to make sure that overflow outside of the range of machine words is caught. In case one of these guards fails the interpreter switches to a big integer heap-allocated representation.)

Encoding Traces as Z3 formulas

The last blog post already contained the code to encode the results of individual trace operations into Z3 formulas, so we don't need to repeat that here. To encode traces of operations we introduce a Z3 variable for every operation in the trace and then call the z3_expression function for every single one of the operations in the trace.

For example, for the following trace:

[i1] i2 = uint_rshift(i1, 32) i3 = int_and(i2, 65535) i4 = uint_rshift(i1, 48) i5 = int_lshift(i4, 16) i6 = int_or(i5, i3) jump(i6, i2) # equal

We would get the Z3 formula:

z3.And(i2 == LShR(i1, 32), i3 == i2 & 65535, i4 == LShR(i1, 48), i5 == i4 << 16)

Usually we won't ask for the formula of the whole trace at once. Instead we go through the trace operation by operation and try to find inefficiencies in the current one we are looking at. Roughly like this (pseudo-)code:

def newvar(name): return z3.BitVec(name, INTEGER_WIDTH) def find_inefficiencies(trace): solver = z3.Solver() var_to_z3var = {} for input_argument in trace.inputargs: var_to_z3var[input_argument] = newz3var(input_argument) for op in trace: var_to_z3var[op] = z3resultvar = newz3var(op.resultvarname) arg0 = op.args[0] z3arg0 = var_to_z3var[arg0] if len(op.args) == 2: arg1 = op.args[1] z3arg1 = var_to_z3var[arg1] else: z3arg1 = None res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # checking for inefficiencies, see the next sections ... if ...: return "inefficient", op # not inefficient, assert op into the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Identifying constant booleans with Z3

To get started finding inefficiencies in a trace, we can first focus on boolean variables. For every operation in the trace that returns a bool we can ask Z3 to prove that this variable must be always True or always False. Most of the time, neither of these proofs will succeed. But if Z3 manages to prove one of them, we know have found an ineffiency: instead of computing the boolean result (eg by executing a comparison) the JIT's optimizer could have replaced the operation with the corresponding boolean constant.

Here's an example of an inefficiency found that way: if x < y and y < z are both true, PyPy's JIT could conclude that x < z must also be true. However, currently the JIT cannot make that conclusion because it only reasons about the concrete ranges (lower and upper bounds) for every integer variable, but it has no way to remember anything about relationships between different variables. This kind of reasoning would quite often be useful to remove list/string bounds checks. Here's a talk about how LLVM does this (but it might be too heavyweight for a JIT setting).

Here are some more examples found that way:

• x - 1 == x is always False
• x - (x == -1) == -1 is always False. The pattern x - (x == -1) happens a lot in PyPy's hash computations: To be compatible with the CPython hashes we need to make sure that no object's hash is -1 (CPython uses -1 as an error value on the C level).

Here's pseudo-code for how to implement checking boolean operations for inefficiencies:

def find_inefficiencies(trace): ... for op in trace: ... res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # check for boolean constant result if op.has_boolean_result(): if prove(solver, res == 0): return "inefficient", op, 0 if prove(solver, res == 1): return "inefficient", op, 1 # checking for other inefficiencies, see the next sections ... # not inefficient, add op to the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Identifying redundant operations

A more interesting class of redundancy is to try to find two operations in a trace that compute the same result. We can do that by asking Z3 to prove for each pair of different operations in the trace to prove that the result is always the same. If a previous operation returns the same result, the JIT could have re-used that result instead of re-computing it, saving time. Doing this search for equivalent operations with Z3 is quadratic in the number of operations, but since traces have a maximum length it is not too bad in practice.

This is the real workhorse of my script so far, it's what finds most of the inefficiencies. Here's a few examples:

• The very first and super useful example the script found is int_eq(b, 1) == b if b is known to be a boolean (ie and integer 0 or 1). I have already implemented this optimization in the JIT.
• Similarly, int_and(b, 1) == b for booleans.
• (x << 4) & -0xf == x << 4
• ((x >> 63) << 1) << 2) >> 3 == x >> 63. In general the JIT is quite bad at optimizing repeated shifts (the infrastructure for doing better with that is already in place, so this will be a relatively easy fix).
• (x & 0xffffffff) | ((x >> 32) << 32) == x. Having the JIT optimize this would maybe require first recognizing that (x >> 32) << 32 can be expressed as a mask: (x & 0xffffffff00000000), and then using (x & c1) | (x & c1) == x & (c1 | c2)
• A commonly occurring pattern is variations of this one: ((x & 1345) ^ 2048) - 2048 == x & 1345 (with different constants, of course). xor is add without carry, and x & 1345 does not have the bit 2048 set. Therefore the ^ 2048 is equivalent to + 2048, which the - 2048 cancels. More generally, if a & b == 0, then a + b == a | b == a ^ b. I don't understand at all why this appears so often in the traces, but I see variations of it a lot. LLVM can optimize this, but GCC can't, thanks to Andrew Pinski for filing the bug!

And here's some implementation pseudo-code again:

def find_inefficiencies(trace): ... for op in trace: ... res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # check for boolean constant result ... # searching for redundant operations for previous_op in trace: if previous_op is op: break # done, reached the current op previous_op_z3var = var_to_z3var[previous_op] if prove(solver, previous_op_z3var == res): return "inefficient", op, previous_op ... # more code here later ... # not inefficient, add op to the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Synthesizing more complicated constants with exists-forall

To find out whether some integer operations always return a constant result, we can't simply use the same trick as for those operations that return boolean results, because enumerating 2⁶⁴ possible constants and checking them all would take too long. Like in the last post, we can use z3.ForAll to find out whether Z3 can synthesize a constant for the result of an operation for us. If such a constant exists, the JIT could have removed the operation, and replaced it with the constant that Z3 provides.

Here a few examples of inefficiencies found this way:

• (x ^ 1) ^ x == 1 (or, more generally: (x ^ y) ^ x == y)
• if x | y == 0, it follows that x == 0 and y == 0
• if x != MAXINT, then x + 1 > x

Implementing this is actually slightly annoying. The solver.add calls for non-inefficient ops add assertions to the solver, which are now confusing the z3.ForAll query. We could remove all assertion from the solver, then do the ForAll query, then add the assertions back. What I ended doing instead was instantiating a second solver object that I'm using for the ForAll queries, that remains empty the whole time.

def find_inefficiencies(trace): solver = z3.Solver() empty_solver = z3.Solver() var_to_z3var = {} ... for op in trace: ... res, valid_if = z3_expression(op.name, z3arg0, z3arg1) # check for boolean constant result ... # searching for redundant operations ... # checking for constant results constvar = z3.BitVec('find_const', INTEGER_WIDTH) condition = z3.ForAll( var_to_z3var.values(), z3.Implies( *solver.assertions(), expr == constvar ) ) if empty_solver.check(condition) == z3.sat: model = empty_solver.model() const = model[constvar].as_signed_long() return "inefficient", op, const # not inefficient, add op to the solver and continue with the next op solver.add(z3resultvar == res) return None # no inefficiency found Minimization

Analyzing an inefficiency by hand in the context of a larger trace is quite tedious. Therefore I've implemented a (super inefficient) script to try to make the examples smaller. Here's how that works:

• First throw out all the operations that occur after the inefficient operation in the trace.
• Then we remove all "dead" operations, ie operations that don't have their results used (all the operations that we can analyze with Z3 are without side effects).
• Now we try to remove every guard in the trace one by one and check afterwards, whether the resulting trace still has an inefficiency.
• We also try to replace every single operation with a new argument to the trace, to see whether the inefficiency is still present.

The minimization process is sort of inefficient and I should probably be using shrinkray or C-Reduce instead. However, it seems to work well in practice and the runtime isn't too bad.

Results

So far I am using the JIT traces of three programs: 1) Booting Linux on the Pydrofoil RISC-V emulator, 2) booting Linux on the Pydrofoil ARM emulator, and 3) running the PyPy bootstrap process on top of PyPy.

I picked these programs because most Python programs don't contain interesting amounts of integer operations, and the traces of the emulators contain a lot of them. I also used the bootstrap process because I still wanted to try a big Python program and personally care about the runtime of this program a lot.

The script identifies 94 inefficiencies in the traces, a lot of them come from repeating patterns. My next steps will be to manually inspect them all, categorize them, and implement easy optimizations identified that way. I also want a way to sort the examples by execution count in the benchmarks, to get a feeling for which of them are most important.

I didn't investigate the full set of Python benchmarks that PyPy uses yet, because I don't expect them to contain interesting amounts of integer operations, but maybe I am wrong about that? Will have to try eventually.

Conclusion

This was again much easier to do than I would have expected! Given that I had the translation of trace ops to Z3 already in place, it was a matter of about a day's of programming to use this infrastructure to find the first problems and minimizing them.

Reusing the results of existing operations or replacing operations by constants can be seen as "zero-instruction superoptimization". I'll probably be rather busy for a while to add the missing optimizations identified by my simple script. But later extensions to actually synthesize one or several operations in the attempt to optimize the traces more and find more opportunities should be possible.

Finding inefficiencies in traces with Z3 is significantly less annoying and also less error-prone than just manually inspecting traces and trying to spot optimization opportunities.

Random Notes and Sources

Again, John's blog posts:

• Let’s Work on an LLVM Superoptimizer
• Early Superoptimizer Results
• A Few Synthesizing Superoptimizer Results
• Synthesizing Constants

and papers:

• A Synthesizing Superoptimizer
• Hydra: Generalizing Peephole Optimizations with Program Synthesis

I remembered recently that I had seen the approach of optimizing the traces of a tracing JIT with Z3 a long time ago, as part of the (now long dead, I think) SPUR project. There's a workshop paper from 2010 about this. SPUR was trying to use Z3 built into the actual JIT (as opposed to using Z3 only to find places where the regular optimizers could be improved). In addition to bitvectors, SPUR also used the Z3 support for arrays to model the C# heap and remove redundant stores. This is still another future extension for all the Z3 work I've been doing in the context of the PyPy JIT.

Here's what I've been working on for my Drupal contributions this week. Thanks to Code Enigma for sponsoring the time to work on these.

Let’s go for my web review for the week 2024-29.

The graying open source community needs fresh blood • The Register

Tags: tech, foss, community

This is indeed a problem. Somehow it became much harder to attract younger developers.

https://www.theregister.com/2024/07/15/opinion_open_source_attract_devs/

“Privacy-Preserving” Attribution: Mozilla Disappoints Us Yet Again

Tags: tech, browser, mozilla, privacy

You’d expect Mozilla to know better. This is disappointing, they’re no living up to their responsibility.

https://blog.privacyguides.org/2024/07/14/mozilla-disappoints-us-yet-again-2/

Commission sends preliminary findings to X for breach of DSA

Tags: tech, twitter, social-media, law

The European Commission starts showing it’s muscles. Twitter is an obvious one to pursue since it became the X cesspool.

https://ec.europa.eu/commission/presscorner/detail/en/IP_24_3761

Goldman Sachs: AI Is Overhyped, Wildly Expensive, and Unreliable

Tags: tech, ai, machine-learning, gpt, economics, ecology, criticism

I’m rarely on the side of a Goldman Sachs… Still this paper seems to be spot on. The equation between the costs (financial and ecological) and the value we get out of generative AI isn’t balanced at all. Also, since it is stuck on trying to improve mostly on model scale and amount of data it is doomed to plateau in its current form.

https://www.404media.co/goldman-sachs-ai-is-overhyped-wildly-expensive-and-unreliable/

Facebook Is the ‘Zombie Internet’

Tags: tech, ai, social-media, facebook

Or examples of the collapse of a shared reality. This has nothing to do with “social” media anymore. Very nice investigation in any case.

https://www.404media.co/email/24eb6cea-6fa6-4b98-a2d2-8c4ba33d6c04/

AT&T says criminals stole phone records of ‘nearly all’ customers in new data breach

Tags: tech, security, leak

Wow! This is a really bad data breach. Apparently related to the recent data theft on the Snowflake end.

https://techcrunch.com/2024/07/12/att-phone-records-stolen-data-breach/

git-pr: A new git collaboration service

Tags: tech, git, codereview, tools

Interesting approach to building a new code review system. I somehow doubt it’ll get traction unfortunately but it has nice ideas baked in.

https://pr.pico.sh/

gpu.cpp: portable GPU compute for C++ with WebGPU – Answer.AI

Tags: tech, c++, gpu, computation

Looks like an interesting library to build portable GPU compute workloads. Cleverly tries to leverage WebGPU.

https://www.answer.ai/posts/2024-07-11–gpu-cpp.html

C++ Design Patterns For Low-Latency Applications

Tags: tech, c++, performance, optimization, pattern

A paper listing patterns to reduce latency as much as possible. There are lesser known tricks in there.

https://hackaday.com/2024/07/13/c-design-patterns-for-low-latency-applications/

22 Common Filesystem Tasks in C++20

Tags: tech, c++

Nice little reference of what can be done with std::filesystem.

https://www.cppstories.com/2024/common-filesystem-cpp20/

C++ Must Become Safer

Tags: tech, c++, safety, memory

Definitely this. C++ isn’t going away anytime soon. Rewrites won’t be worth it in important cases, so improving the safety of the language matters.

https://www.alilleybrinker.com/blog/cpp-must-become-safer/

Django Migration Operations aka how to rename Models

Tags: tech, django, databases

Django doesn’t always generate the migration you’d expect. Read them before going to production. Also it’s fine to adjust them.

https://micro.webology.dev/2024/07/15/django-migration-operations.html

Gotchas with SQLite in Production

Tags: tech, backend, sqlite, databases

Where are the limitations of using SQLite in production for web applications? Here is a good list.

https://blog.pecar.me/sqlite-prod

SQLite Transactions

Tags: tech, databases, sqlite

Some improvements coming in SQLite transactions. Here are some early tests.

https://reorchestrate.com/posts/sqlite-transactions/

Tests you love to read, write and change

Tags: tech, tests

Three good advices on writing automated tests. This is necessary but not sufficient though.

https://jaywhy13.hashnode.dev/tests-you-love-to-read-write-and-change

Lessons learned in 35 years of making software – Jim Grey

Tags: tech, career

Quite a few good lessons in there. Again it’s more about social skills than technical skills.

https://dev.jimgrey.net/2024/07/03/lessons-learned-in-35-years-of-making-software/

Story Points are Pointless, Measure Queues

Tags: tech, project-management, product-management, estimates, agile

A bit long and a couple of mistakes when pointing out the flaws of story points. Still, it’s definitely a worthwhile read. Quite a lot of the criticism of story points is warranted and the proposed approach based on queue theory is welcome. This is stuff you can find in Kanban like approaches and mature XP.

https://www.brightball.com/articles/story-points-are-pointless-measure-queues

Managing Underperformers | Jack Danger

Tags: management

Nice advices to deal with underperforming teams or individuals. Making the distinction between refusal to align or failure to execute is particularly useful.

https://jackdanger.com/managing-underperformers/

All I Need to Know About Engineering Leadership I Learned From Leave No Trace - Jacob Kaplan-Moss

Tags: tech, leadership, engineering, ecology, funny

Funny experiment at drawing parallels between engineering leadership and how you should behave when hiking in nature. This works surprisingly well.

https://jacobian.org/2024/jul/12/lnt-for-engineering-leadership/

Progress can be slow

Tags: work, life, improving, coaching, habits

This one is more self-help than I’m usually comfortable with… somehow something rung true to me with it. It’s indeed a good reminder that changing habits takes a while. It’s an exercise in patience and there are good reasons for it.

https://jeanhsu.substack.com/p/progress-can-be-slow?isFreemail=true&post_id=146457673

German Navy still uses 8-inch floppy disks, working on emulating a replacement | Ars Technica

Tags: tech, hardware, storage, history

We keep finding floppies in use at surprising places. There’s clearly lot of inertia for technologies getting replaced.

https://arstechnica.com/gadgets/2024/07/german-navy-still-uses-8-inch-floppy-disks-working-on-emulating-a-replacement/

Bye for now!

The following contributors got their Debian Developer accounts in the last two months:

• Dennis van Dok (dvandok)
• Peter Wienemann (wiene)
• Quentin Lejard (valde)
• Sven Geuer (sge)
• Taavi Väänänen (taavi)
• Hilmar Preusse (hille42)
• Matthias Geiger (werdahias)
• Yogeswaran Umasankar (yogu)

The following contributors were added as Debian Maintainers in the last two months:

• Bernhard Miklautz
• Felix Moessbauer
• Maytham Alsudany
• Aquila Macedo
• David Lamparter
• Tim Theisen
• Stefano Brivio
• Shengqi Chen

Congratulations!