Feeds

We can use uv to make linting and testing on GitHub Actions around 1.5 times as fast.

Linting

When using pre-commit for linting:

name: Lint on: [push, pull_request, workflow_dispatch] env: FORCE_COLOR: 1 permissions: contents: read jobs: lint: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: persist-credentials: false - uses: actions/setup-python@v5 with: python-version: "3.x" cache: pip - uses: pre-commit/action@v3.0.1

We can replace pre-commit/action with tox-dev/action-pre-commit-uv:

- uses: actions/setup-python@v5 with: python-version: "3.x" - cache: pip - - uses: pre-commit/action@v3.0.1 + - uses: tox-dev/action-pre-commit-uv@v1 name: Lint on: [push, pull_request, workflow_dispatch] env: FORCE_COLOR: 1 permissions: contents: read jobs: lint: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 with: persist-credentials: false - uses: actions/setup-python@v5 with: python-version: "3.x" - uses: tox-dev/action-pre-commit-uv@v1

This means uv will create virtual environments and install packages for pre-commit, which is faster for the initial seed operation when there's no cache.

Lint comparison

For example: python/blurb#32

Before After Times faster No cache 60s 37s 1.62 With cache 11s 11s 1.00 Testing

When testing with tox:

name: Test on: [push, pull_request, workflow_dispatch] permissions: contents: read env: FORCE_COLOR: 1 jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-version: ["3.9", "3.10", "3.11", "3.12", "3.13", "3.14"] steps: - uses: actions/checkout@v4 with: persist-credentials: false - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} allow-prereleases: true cache: pip - name: Install dependencies run: | python --version python -m pip install -U pip python -m pip install -U tox - name: Tox tests run: | tox -e py

We can replace tox with tox-uv:

- name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} allow-prereleases: true - cache: pip - - name: Install dependencies - run: | - python --version - python -m pip install -U pip - python -m pip install -U tox + - name: Install uv + uses: hynek/setup-cached-uv@v2 - name: Tox tests run: | - tox -e py + uvx --with tox-uv tox -e py name: Test on: [push, pull_request, workflow_dispatch] permissions: contents: read env: FORCE_COLOR: 1 jobs: test: runs-on: ubuntu-latest strategy: fail-fast: false matrix: python-version: ["3.9", "3.10", "3.11", "3.12", "3.13"] steps: - uses: actions/checkout@v4 with: persist-credentials: false - name: Set up Python ${{ matrix.python-version }} uses: actions/setup-python@v5 with: python-version: ${{ matrix.python-version }} allow-prereleases: true - name: Install uv uses: hynek/setup-cached-uv@v2 - name: Tox tests run: | uvx --with tox-uv tox -e py

tox-uv is tox plugin to replace virtualenv and pip with uv in your tox environments. We only need to install uv, and use uvx to both install tox-uv and run tox, for faster installs of tox, the virtual environment, and the dependencies within it.

Test comparison

For example: python/blurb#32

Before After Times faster No cache 2m 0s 1m 26s 1.40 With cache 1m 58s 1m 22s 1.44 Bonus tip

Run the new tool zizmor to find security issues in GitHub Actions.

Header photo: "Road cycling at the 1952 Helsinki Olympics" by Olympia-Kuva Oy & Helsinki City Museum, Public Domain.

Surprise! This blog post series has now been moved to blogs.kde.org so it’s now open for others to participate and contribute! This week’s post can be found at https://blogs.kde.org/2024/11/02/this-week-in-plasma-spoooooky-ooooooooom-notifications

That’s probably where it should have been all along, as this work is much bigger than me. I’ll remain the editor-in-chief for now, but do welcome contributions to help lighten the load.

Unfortunately, due to GDPR restrictions, I’m unable to migrate existing email subscribers to the new email digest over there. So if you’d like to re-subscribe to “This week in Plasma.” head to https://newsletter.kde.org/subscription/form and re-subscribe.

I’ll still be blogging here about KDE topics of interest to me and hopefully you as well, just not the weekly Plasma news. So I do hope you’ll stick around.

Two months ago I bought a Thinkpad X1 Yoga Gen3 [1]. I’m still very happy with it, the screen is a great improvement over the FullHD screen on my previous Thinkpad. I have yet to discover what’s the best resolution to have on a laptop if price isn’t an issue, but it’s at least 1440p for a 14″ display, that’s 210DPI. The latest Thinkpad X1 Yoga is the 7th gen and has up to 3840*2400 resolution on the internal display for 323DPI. Apple apparently uses the term “Retina Display” to mean something in the range of 250DPI to 300DPI, so my current laptop is below “Retina” while the most expensive new Thinkpads are above it.

I did some tests on external displays and found that this Thinkpad along with a Dell Latitude of the same form factor and about the same age can only handle one 4K display on a Thunderbolt dock and one on HDMI. On Reddit u/Carlioso1234 pointed out this specs page which says it supports a maximum of 3 displays including the built in TFT [2]. The Thunderbolt/USB-C connection has a maximum resolution of 5120*2880 and the HDMI port has a maximum of 4K. The latest Yoga can support four displays total which means 2*5K over Thunderbolt and one 4K over HDMI. It would be nice if someone made a 8000*2880 ultrawide display that looked like 2*5K displays when connected via Thunderbolt. It would also be nice if someone made a 32″ 5K display, currently they all seem to be 27″ and I’ve found that even for 4K resolution 32″ is better than 27″.

With the typical configuration of Linux and the BIOS the Yoga Gen3 will have it’s touch screen stop working after suspend. I have confirmed this for stylus use but as the finger-touch functionality is broken I couldn’t confirm that. On r/thinkpad u/p9k told me how to fix this problem [3]. I had to set the BIOS to Win 10 Sleep aka Hybrid sleep and then put the following in /etc/systemd/system/thinkpad-wakeup-config.service :

# https://www.reddit.com/r/thinkpad/comments/1blpy20/comment/kw7se2l/?context=3 [Unit] Description=Workarounds for sleep wakeup source for Thinkpad X1 Yoga 3 After=sysinit.target After=systemd-modules-load.service [Service] Type=oneshot ExecStart=/bin/sh -c "echo 'enabled' > /sys/devices/platform/i8042/serio0/power/wakeup" ExecStart=/bin/sh -c "echo 'enabled' > /sys/devices/platform/i8042/serio1/power/wakeup" ExecStart=/bin/sh -c "echo 'LID' > /proc/acpi/wakeup" [Install] WantedBy=multi-user.target

Now it works fine, for stylus at least. I still get kernel error messages like the following which don’t seem to cause problems:

wacom 0003:056A:5146.0005: wacom_idleprox_timeout: tool appears to be hung in-prox. forcing it out.

When it wasn’t working I got the above but also kernel error messages like:

wacom 0003:056A:5146.0005: wacom_wac_queue_insert: kfifo has filled, starting to drop events

This change affected the way suspend etc operate. Now when I connect the laptop to power it will leave suspend mode. I’ve configured KDE to suspend when the lid is closed and there’s no monitor connected.

• [1] https://etbe.coker.com.au/2024/01/29/thinkpad-x1-yoga-gen3/
• [2] https://tinyurl.com/2cvq6qts
• [3] https://tinyurl.com/26ho6vxk

Related posts:

1. Thinkpad X1 Yoga Gen3 I just bought myself a Thinkpad X1 Yoga Gen3 for...
2. More About Kogan 5120*2160 Monitor On the 18th of May I blogged about my new...
3. Thinkpad X1 Carbon Gen 6 In February I reviewed a Thinkpad X1 Carbon Gen 1...

I previously wrote about the possibility of transferring work between devices as an alternative to “convergence” (using a phone or tablet as a desktop) [1]. This idea has been implemented in some commercial products already.

MrWhosTheBoss made a good YouTube video reviewing recent Huawei products [2]. At 2:50 in that video he shows how you can link a phone and tablet, control one from the other, drag and drop of running apps and files between phone and tablet, mirror the screen between devices, etc. He describes playing a video on one device and having it appear on the other, I hope that it actually launches a new instance of the player app as the Google Chromecast failed in the market due to remote display being laggy. At 7:30 in that video he starts talking about the features that are available when you have multiple Huawei devices, starting with the ability to move a Bluetooth pairing for earphones to a different device.

At 16:25 he shows what Huawei is doing to get apps going including allowing apk files to be downloaded and creating what they call “Quick Apps” which are instances of a web browser configured to just use one web site and make it look like a discrete app, we need something like this for FOSS phone distributions – does anyone know of a browser that’s good for it?

Another thing that we need is to have an easy way of transferring open web pages between systems. Chrome allows sending pages between systems but it’s proprietary, limited to Chrome only, and also takes an unreasonable amount of time. KDEConnect allows sharing clipboard contents which can be used to send URLs that can then be pasted into a browser, but the process of copy URL, send via KDEConnect, and paste into other device is unreasonably slow. The design of Chrome with a “Send to your devices” menu option from the tab bar is OK. But ideally we need a “Send to device” for all tabs of a window as well, we need it to run from free software and support using your own server not someone else’s server (AKA “the cloud”). Some of the KDEConnect functionality but using a server rather than direct connection over the same Wifi network (or LAN if bridged to Wifi) would be good.

What else do we need?

• [1] https://etbe.coker.com.au/2024/04/26/convergence-vs-transference/
• [2] https://www.youtube.com/watch?v=IhYDtLzmRQI

Related posts:

1. Convergence vs Transference I previously wrote a blog post titled Considering Convergence [1]...
2. Moving from a Laptop to a Cloud Lifestyle My Laptop History In 1998 I bought my first laptop,...
3. Cheap NAS Devices Suck There are some really good Network Attached Storage (NAS) devices...

I recently had someone describe a Mac Mini as a “workstation”, which I strongly disagree with. The Wikipedia page for Workstation [1] says that it’s a type of computer designed for scientific or technical use, for a single user, and would commonly run a multi-user OS.

The Mac Mini runs a multi-user OS and is designed for a single user. The issue is whether it is for “scientific or technical use”. A Mac Mini is a nice little graphical system which could be used for CAD and other engineering work. But I believe that the low capabilities of the system and lack of expansion options make it less of a workstation.

The latest versions of the Mac Mini (to be officially launched next week) have up to 64G of RAM and up to 8T of storage. That is quite decent compute power for a small device. For comparison the HP ML 110 Gen9 workstation I’m currently using was released in 2021 and has 256G of RAM and has 4 * 3.5″ SAS bays so I could easily put a few 4TB NVMe devices and some hard drives larger than 10TB. The HP Z640 workstation I have was released in 2014 and has 128G of RAM and 4*2.5″ SATA drive bays and 2*3.5″ SATA drive bays. Previously I had a Dell PowerEdge T320 which was released in 2012 and had 96G of RAM and 8*3.5″ SAS bays.

In CPU and GPU power the recent Mac Minis will compare well to my latest workstations. But they compare poorly to workstations from as much as 12 years ago for RAM and storage. Which is more important depends on the task, if you have to do calculations on 80G of data with lots of scans through the entire data set then a system with 64G of RAM will perform very poorly and a system with 96G and a CPU less than half as fast will perform better. A Dell PowerEdge T320 from 2012 fully loaded with 192G of RAM will outperform a modern Mac Mini on many tasks due to this and the T420 supported up to 384G.

Another issue is generic expansion options. I expect a workstation to have a number of PCIe slots free for GPUs and other devices. The T320 I used to use had a PCIe power cable for a power hungry GPU and I think all the T320 and T420 models with high power PSUs supported that.

I think that a usable definition of a “workstation” is a system having a feature set that is typical of servers (ECC RAM, lots of storage for RAID, maybe hot-swap storage devices, maybe redundant PSUs, and lots of expansion options) while also being suitable for running on a desktop or under a desk. The Mac Mini is nice for running on a desk but that’s the only workstation criteria it fits. I think that ECC RAM should be a mandatory criteria and any system without it isn’t a workstation. That excludes most Apple hardware. The Mac Mini is more of a thin-client than a workstation.

My main workstation with ECC RAM could run 3 VMs that each have more RAM than the largest Mac Mini that will be sold next week.

If 32G of non-ECC RAM is considered enough for a “workstation” then you could get an Android phone that counts as a workstation – and it will probably cost less than a Mac Mini.

• [1] https://en.wikipedia.org/wiki/Workstation

Related posts:

1. BTRFS Rebuild Time In February I replaced a Dell T320 server with a...
2. SSD for a Workstation SSDs have been dropping in price recently so I just...
3. T320 iDRAC Failure and new HP Z640 The Dell T320 Almost 2 years ago I made a...

Welcome to the new home of "This Week in Plasma"! No longer is it a private personal thing on my (Nate Graham's) blog, but now it's a weekly series hosted here on KDE's infrastructure, open to anyone's participation and contribution! I'll remain the editor-in-chief for now, and welcome contributions via direct push to the relevant merge request on invent.kde.org. And after a post is published, if you find a typo or broken link, feel free to just fix it.

Anyway, this week we added a useful service to detect out-of-memory (OOM) conditions, did some UI polishing, and also a lot of bug-fixing! Check it out:

Notable New Features

When the system has run out of memory (OOM) and the kernel terminated an app, there's now a little service to detect this and show you a system notification explaining what happened, plus suggestions for what you can do about it in the future. (Harald Sitter, 6.3.0. Link)

Notable UI Improvements

The Emoji Selector app now does sub-string matching from the middle of words too, so you can find emojis more easily. (Eren Karakas, 6.2.3. Link)

The grouping indicator in the Task Manager widget (which looks like a little plus sign) is no longer always green; now it follows the current accent color! (Tem PQD, 6.3.0. Link)

Appropriate symbolic icons from will now be automatically substituted for apps' system tray icons — when they exist in the icon theme. If you'd like to extend this to more apps, figure out what icon name the app asks for, then create a symbolic version of it, append -symbolic to the name, and submit it to the Breeze Icons repo! (Marco Martin, 6.3.0. Link)

Notable Bug Fixes

Fixed a way that misbehaving XWayland-using apps could make KWin freeze. (Vlad Zahorodnii, 6.2.3. Link)

Fixed a bug causing notifications about modifier key changes (if you've turned them on) to not actually appear. (Nicolas Fella, 6.2.3. Link)

Fixed a bug in the Bluetooth pairing wizard that caused it to be annoying to enter digits. (Daniil-Viktor Ratkin, 6.2.3. Link)

Reviews on Discover's app pages now load properly when the app is accessed from the Home page, as opposed to after browsing or searching. (Aleix Pol Gonzalez, 6.2.3. Link)

Fixed the Task Manager widget's setting to reverse the direction that new tasks appear in so that it works as originally designed, and not only halfway there. (Michael Rivnak, 6.2.3. Link)

Fixed a bug that caused customized user avatars for other currently-logged-in users to not be displayed in the User Switcher widget. (Blazer Silving, 6.2.3. Link)

Fixed a bug causing the "Show Logout Screen" item in the desktop context menu to not show enough items if you had customized the "default logout option" in the past, back when we offered that as a user-facing setting. (Nate Graham, 6.2.3 Link)

With global animations set to "instant", window thumbnails in KWin's Desktop Grid effect are no longer too small. (Niccolò Venerandi, 6.2.3. Link)

When you've got the Application Dashboard widget set up with an icon naming style that causes the text to be long and get elided, hovering over the icon now shows a tooltip with the correct full text in it. (Tomislav Pap, 6.2.3. Link)

Fixed a case where KWin could crash when moving certain drawing tablet pens to the surface of the pad. (Vlad Zahorodnii, 6.3.0. Link)

Fixed an issue that caused XWayland-using apps to resize in a janky and jumpy manner. (Vlad Zahorodnii, 6.3.0. Link, see detailed blog post)

How You Can Help

KDE has become important in the world, and your time and contributions have helped us get there. As we grow, we need your support to keep KDE sustainable.

You can help KDE by becoming an active community member and getting involved somehow. Each contributor makes a huge difference in KDE — you are not a number or a cog in a machine!

You don’t have to be a programmer, either. Many other opportunities exist:

• Filter and confirm bug reports, maybe even identify their root cause
• Contribute designs for wallpapers, icons, and app interfaces
• Design and maintain websites
• Translate user interface text items into your own language
• Promote KDE in your local community
• …And a ton more things!

You can also help us by donating to our yearly fundraiser! Any monetary contribution — however small — will help us cover operational costs, salaries, travel expenses for contributors, and in general just keep KDE bringing Free Software to the world.

To get a new Plasma feature or a bugfix mentioned here, feel free to push a commit to the relevant merge request on invent.kde.org.

In Plasma 6.2, KWin switched from doing linear blending with HDR to blending in a gamma 2.2 space. Let’s take a look at what that means, and why it was done.

What is blending?

When KWin composites, it paints window by window, going by the order of how the windows are stacked - the bottom-most window first, the topmost window last. When a window is opaque, you just overwrite the pixels in the framebuffer with the ones from the window. When a window is semi-transparent though, we need to additionally do blending.

To do blending, the GPU calculates the value that the framebuffer should have with some equation that gets the pixel from the window and the existing value in the framebuffer, and outputs some appropriate value. Usually that equation is1

framebuffer = framebuffer.rgb * (1 - window.alpha) + window.rgb * window.alpha

where window.alpha is a per-pixel value that describes how opaque the pixel is.

Blending with color management

In Plasma 5, compositing happened in the display color space. As displays could be assumed to be roughly the same, with brightness levels encoded in sRGB, that resulted in blending looking very similar everywhere.

With HDR in Plasma 6 however, that assumption was no longer true, so we had to do something else. As it was considered the most “correct” thing and allows us to ensure the exact same blending result even with displays that have wildly different colorspaces, linear blending was chosen for Plasma 6.0. The result of linear blending would look different from sRGB, but it would at least be consistent everywhere.

With linear blending, instead of just taking the rgb values as is, you first convert them into light-linear values, for example with sRGB you’d just do

rgb_linear = pow(rgb_sRGB, 2.2)

then apply the blending equation, and at the end convert it back to whatever encoding the screen needs.

Because of limitations in KWin’s renderer and the ancient OpenGL versions KWin supports, the only way to do this was to render first into a so-called shadow buffer2 with linear values, and after compositing a second shader pass would run, taking that shadow buffer as the input and outputting a buffer with non-linear values that’s suitable for sending to the screen.

The caveats of linear blending

It’ll be pretty obvious to most that doing a fullscreen copy each frame is not ideal for performance or battery life, but there was even a second problem: If you use only 8 or 10 bits per color (bpc) to store linear brightness values, you get visible banding in the dark areas of the image, as human vision is very non-linear. So on top of doing fullscreen copies, KWin also had to use a floating point buffer with 16 bpc, which uses twice the memory bandwidth vs. 8 bpc and makes performance and power usage even worse.

With that performance hit, we couldn’t enable this on all hardware, but had to restrict linear blending to those displays where HDR or an ICC profile is enabled. This threw the whole consistency benefit out of the window, because now a semi-transparent surface would look a lot more transparent just because you enabled HDR… causing the very problem we wanted to avoid!

SDR HDR

We had to do something when blending in HDR though, so a different approach had to be taken.

Custom transfer functions

When storing colors in buffers, usually we use non-linear encodings with transfer functions like sRGB or PQ. All the transfer functions have some sort of luminance levels attached to them, for example an sRGB value of 1.0 is defined to result in a luminance of 80 nits in its reference viewing environment3.

There’s no reason to be restricted to those definitions though - we can make transfer functions that mean whatever we want or need. In KWin’s case, we switched the shadow buffer from using a linear encoding to a gamma 2.2 encoding, in which 1.0 means the maximum luminance of your monitor. This means we do blending in a very similar way as on SDR screens4, and translucent surfaces on the screen look pretty much the same again, no matter what display settings you have.

As the gamma 2.2 encoding allocates more numbers to darker parts of the image, we can also avoid banding with fewer bits per color. Whenever HDR is enabled and the driver supports buffers with 10 bpc, KWin now prefers to use that instead of 16 bpc, alleviating some of the performance issues. There was still more that could be done though…

KMS offloading

On most graphics cards, the scanout hardware that takes care of sending the image to the display has some fixed function blocks to change the colors in various ways - the most common being a simple look up table (LUT) per color channel.

Through the DRM/KMS API, the compositor can set this LUT, so we can use it to change the encoding from whatever we did blending in to the one the display needs. With HDR screens, this means we

• convert from our shadow buffer encoding with gamma 2.2 and the maximum luminance of the screen to linear
• possibly apply rgb factors for night light
• convert from linear to the PQ encoding the screen needs

With that LUT in place, we don’t have to run a shader pass to convert from the shadow buffer encoding to the screen anymore, so the whole fullscreen copy falls away.

Except for a few additional instructions in the shaders used for compositing, enabling HDR in Plasma 6.2 thus has no performance impact anymore on the vast majority of hardware!

1. in practice, window.rgb is already “pre-multiplied” with window.alpha, but that doesn’t really matter here ↩

2. it’s called that because it’s never actually shown on the screen ↩

3. the viewing environment basically defines a standardized room, in which the content can be viewed as intended without doing any brightness adjustments ↩

4. it’s not exactly the same though. If you need some exactly specific blending behavior in your application, it’s best if you do it yourself ↩

Welcome to the Free Software Supporter, the Free Software Foundation's (FSF) monthly news digest and action update -- being read by you and 231,355 other activists.

Welcome to the Free Software Supporter, the Free Software Foundation's (FSF) monthly news digest and action update -- being read by you and 231,355 other activists.

Seven new GNU releases in the last month (as of October 31, 2024):

Seven new GNU releases in the last month (as of October 31, 2024):

Let’s go for my web review for the week 2024-44.

What You Can Learn from Just Seven Pages by Hannah Arendt

Tags: tech, philosophy, history, politics

A very precious philosopher from the 20th century. Her texts are still very precious and resonate today. In this piece it’s focusing about tech relevant excerpts, she had plenty to say about today’s politics as well.

https://www.honest-broker.com/p/what-you-can-learn-from-just-seven

The Open Source AI Definition

Tags: tech, foss, ai, machine-learning

Nice initiative from the OSI. It is timely, such a definition was surely needed. The data information part seems fairly weak though… for sure you could make a system which doesn’t respect the four freedoms that way.

https://opensource.org/ai/open-source-ai-definition

Does Open Source AI really exist?

Tags: tech, foss, ai, machine-learning

Like me, you find the Open Source AI Definition weak on the training data information side? You’d be right and there’s a reason for it… it’s probably hiding quite some open washing for the larger models. This is a good explanation of the motives and consequences.

https://tante.cc/2024/10/16/does-open-source-ai-really-exist/

Platform Strategy and Its Discontents

Tags: tech, web, mobile, react, framework, criticism

This is definitely true. As long as web frontends are dominated by large frameworks, the web will always have subpar experience on mobile. And the solution isn’t going to come from the mobile providers too happy to gatekeep their app store.

https://infrequently.org/2024/10/platforms-are-competitions/#fn-failure-on-repeat-2

Matrix 2.0 Is Here!

Tags: tech, matrix, protocols

This is definitely getting there in terms of performance and usability. The mobile clients seem mature enough, just need the desktop clients to catch up before this becomes really something I’d feel confident enough to recommend and push for.

https://matrix.org/blog/2024/10/29/matrix-2.0-is-here/

What’s New in POSIX 2024

Tags: tech, unix, posix

It’s nice to see the standard still moves. Some of the additions are definitely welcome.

https://blog.toast.cafe/posix2024-xcu

Australia/Lord_Howe is the weirdest timezone

Tags: tech, time, culture, internationalization

Time management and timezones are definitely complicated. In a way it’s culture colliding with computers and localisation… it can’t be simple.

https://ssoready.com/blog/engineering/truths-programmers-timezones/

Improving SSH’s security with SSHFP DNS records

Tags: tech, tools, ssh, dns, security

Nice technique for automating the verification of SSH host keys. It’d be nice to see wider adoption.

https://blog.apnic.net/2022/12/02/improving-sshs-security-with-sshfp-dns-records/

Introducing zizmor: now you can have beautiful clean workflows

Tags: tech, github, ci, security, tools

Definitely an interesting tool. GitHub Actions workflow aren’t easy to setup while ensuring they’re secure, having a tool analyzing them for issues can only help.

https://blog.yossarian.net/2024/10/27/Now-you-can-have-beautiful-clean-workflows

Toward safe transmutation in Rust

Tags: tech, rust, type-systems, memory

Interesting progress on safe type casting in Rust. This should bring nice zero copy parsing of binary data in some cases.

https://lwn.net/SubscriberLink/994334/5e1f97f08916b494/

Lessons learned from a successful Rust rewrite

Tags: tech, c++, rust, legacy

This is a good view of what you’re getting into with the “rewrite it in Rust” knee-jerk reaction.

https://gaultier.github.io/blog/lessons_learned_from_a_successful_rust_rewrite.html

Vector Databases Are the Wrong Abstraction

Tags: tech, postgresql, databases, ai, machine-learning, language

I definitely like the approach of having vectorisation in the RDBMS directly. This is one less moving part, less complexity at the application level to synchronize everything together. In this case it’s a Postgres extension.

https://www.timescale.com/blog/vector-databases-are-the-wrong-abstraction/

Database Remote-Copy Tool For SQLite

Tags: tech, sqlite, databases, tools

Interesting, there’s now an official tool to replicate sqlite databases. It’s still early days, we’ll see which features it’ll get.

https://sqlite.org/rsync.html

How to profile a performance issue using Spring Boot profiling tools

Tags: tech, java, spring, profiling

A quick tour of the available tools to profile Spring Boot applications.

https://foojay.io/today/how-to-profile-a-performance-issue-using-spring-boot-profiling-tools/

Working with stacked branches in Git is easier with –update-refs

Tags: tech, git, tools

Yet another Git option I missed. This is definitely useful, I’ll try it out.

https://andrewlock.net/working-with-stacked-branches-in-git-is-easier-with-update-refs/

pygfx

Tags: tech, python, 3d, webgpu, data-visualization

Looks like a very interesting Python library to build interactive 3d visualizations.

https://docs.pygfx.org/stable/index.html

Rudimentary 3D on the 2D HTML Canvas

Tags: tech, 2d, 3d, graphics, mathematics

Ever wondered how to simulate 3D from 2D based primitives? Here is a nice experiment explaining how to approach it.

https://www.charlespetzold.com/blog/2024/09/Rudimentary-3D-on-the-2D-HTML-Canvas.html

Classic 3D videogame shadow techniques

Tags: tests, 3d, graphics

A nice list of the techniques used to render shadows in games.

https://30fps.net/pages/videogame-shadows/

On Crafting Painterly Shaders

Tags: tech, 3d, graphics, shader

Very nice deep dive into a post-processing shader to create a painted scene effect.

https://blog.maximeheckel.com/posts/on-crafting-painterly-shaders/?ck_subscriber_id=2669647738

The Basics

Tags: tech, craftsmanship, learning, career

Indeed, those are fundamental traits to make sure you learn and make progress on your journey.

https://registerspill.thorstenball.com/p/the-basics

Background Work - by Kent Beck

Tags: tech, learning, career, craftsmanship

Another excellent piece from Kent Beck, he’s right that the real differentiator in our profession is about digging deep on topics, seeing them through even if that’s on the side. Curiosity is a key trait.

https://tidyfirst.substack.com/p/background-work

the death of the architect

Tags: tech, agile, architecture, history

Good explanation on how the agile movement scaled down about design over time in its literature. It’s probably its biggest failure. The good thing is that the pendulum is starting to swing in the other direction a bit (that’s probably why Beck is now working on a book series on software design).

https://explaining.software/archive/the-death-of-the-architect/

How to make Product give a shit about your architecture proposal

Tags: tech, quality, product-management, project-management

This is accurate in my opinion. Engineering and product teams need to properly negotiate, otherwise quality will suffer.

https://gieseanw.wordpress.com/2024/10/09/how-to-make-product-give-a-shit-about-your-architecture-proposal/

How to not be a prioritization machine

Tags: tech, project-management, product-management, decision-making

Interesting guidelines idea to help teams manage the priorities themselves. It’s written in the context of a product manager but I think it is lightweight and generic enough to apply in other contexts.

https://productmanagers.substack.com/p/how-to-not-be-a-prioritization-machine

We Fell For The Oldest Lie On The Internet

Tags: science, complexity

It’s sometimes extremely difficult to get to the original source of a scientific claim. Our corpus of science is so large and complex now that finding where a claim comes from can be a daunting task.

https://www.youtube.com/watch?v=bgo7rm5Maqg

Changes in primary visual and auditory cortex of blind and sighted adults following 10 weeks of click-based echolocation training | Cerebral Cortex | Oxford Academic

Tags: science, neuroscience

This is an amazing example of the brain plasticity. It’s also great to have a patch for increased quality of life with a training of only a few weeks.

https://academic.oup.com/cercor/article/34/6/bhae239/7696241?login=false

Bye for now!

We’re excited to announce the November 2024 release of the Python, Pylance and Jupyter extensions for Visual Studio Code!

This release includes the following announcements:

• Generate docstrings with Pylance
• New fold and unfold all docstrings commands
• Import suggestions can now include aliases from user files
• Experimental AI Code Action for implementing abstract classes
• Native REPL variables view

If you’re interested, you can check the full list of improvements in our changelogs for the Python, Jupyter and Pylance extensions.

Generate docstrings with Pylance

You can now more conveniently generate documentation for your Python code with Pylance‘s docstrings template generation feature! You can generate a docstring template for classes or methods by typing """ or ''', pressing kbstyle(Ctrl+Space), or selecting the lightbulb to invoke the Generate Docstring Code Action. The generated docstring includes fields for the function’s description, parameter descriptions, parameter types, return value description, and return type.

This feature is currently behind an experimental setting, but we look forward to making it the default experience soon. You can try it out today by enabling the python.analysis.supportDocstringTemplate setting.

New fold and unfold all docstrings commands

Docstrings are great for providing context and explanations for your code, but sometimes you might want to fold them to focus on the code itself. You can now more easily do so by folding docstrings with the new Pylance: Fold All Docstrings command, which can also be bound to a keybinding of your choice. To unfold them, use the Pylance: Unfold All Docstrings command.

Import suggestions with aliases from user files

One of Pylance’s most powerful features is its ability to provide auto-import suggestions. By default, Pylance offers the import suggestion from where the symbol is defined, but you might want it to import from a file where the symbol is imported (i.e. aliased). With the new python.analysis.includeAliasesFromUserFiles setting, you can now control whether Pylance includes alias symbols from user files in its auto-import suggestions and in the add import Quick Fix.

Note: Enabling this setting can negatively impact performance, especially in large codebases, as Pylance may need to index more symbols and monitor more files for changes, which can increase resource usage.

Experimental AI Code Action: Implement Abstract Classes

You can now get the best of both worlds with AI and static analysis with the new experimental Code Action to implement abstract classes! This feature requires both the Pylance and the GitHub Copilot extensions. To try it out, you can select the Implement all inherited abstract classes with Copilot Code Action when defining a class that inherits from an abstract one.

You can disable this feature by setting "python.analysis.aiCodeActions": {"implementAbstractClasses": false} in your User settings.

Native REPL Variables View

The Native Python REPL now provides up-to-date variables for the built-in Variables view. This lets you dig into the state of the interpreter as you execute code from files or through the REPL input box.

Upcoming deprecation of Python 3.8 support

Python 3.8 reached end-of-life (EOL) on 2024-10-07. As such, official support for Python 3.8 in the Python extension will stop in three months, in the February 2025 release of the Python extension. There are no plans to actively remove support for Python 3.8, and so we expect the extension will continue to work unofficially with Python 3.8 for the foreseeable future.

Other Changes and Enhancements

We have also added small enhancements and fixed issues requested by users that should improve your experience working with Python and Jupyter Notebooks in Visual Studio Code. Some notable changes include:

• There’s a new documentation page on the various ways to run Python code in VS Code.
• Pixi functionality has been restored only when Pixi is available (vscode-python#24310).
• You can now change the type checking mode to strict or standard from the Language Status menu (pylance-release#6080)

We would also like to extend special thanks to this month’s contributors:

• @mnoah1 Add customizable interpreter discovery timeout in vscode-python#24227
• @brokoli777 Refactor code to remove unused JSDoc types in vscode-python#24300
• @T-256 Make python_server.py compatible to Python 3.7 in vscode-python#24252

  Note: This doesn’t guarantee full compatibility nor support for Python 3.7 in other parts of the Python extension. The minimum Python version we support is still Python 3.8 until February 2025, when the minimum officially supported version will be Python 3.9.

Try out these new improvements by downloading the Python extension and the Jupyter extension from the Marketplace, or install them directly from the extensions view in Visual Studio Code (Ctrl + Shift + X or ⌘ + ⇧ + X). You can learn more about Python support in Visual Studio Code in the documentation. If you run into any problems or have suggestions, please file an issue on the Python VS Code GitHub page.

The post Python in Visual Studio Code – November 2024 Release appeared first on Python.

Let’s explore why it’s the CMS of choice for councils across the UK!

Exciting news: the PyCon US 2025 conference website, Call for Proposals, and sponsorship program are open!

To learn more about the location, deadlines, and other details, check out the links below:

• PyCon US 2025 Launch Blog Post
• PyCon US 2025 Website
• Submit Your PyCon US 2025 Proposal on Pretalx
• Sponsorship Application
• PyCon US 2025 Changes Blog Post

We’re very happy to answer any questions you have about PSF sponsorship or PyCon US 2025– please feel free to reach out to us at sponsors@python.org.

On behalf of the PSF and the PyCon US 2025 Team, we look forward to receiving your proposals and seeing you in Pittsburgh next year 🥳 🐍

We’re super excited to announce that PyCon US 2025 is back in Pittsburgh! If you missed our first time here, please check out our PyCon US 2024 recap and video recordings.

We’re also excited to announce the launch of our conference website, along with Call for Proposals and our sponsorship program!We’re Coming Back to Pittsburgh!We will be hosting PyCon US 2025 again in Pittsburgh, Pennsylvania. We will continue to return in person, with Health and Safety Guidelines in place.

PyCon US 2025 will be held at the David L. Lawrence Convention Center in Pittsburgh, Pennsylvania, on the following dates:

• May 14-15, 2025 - Tutorials
• May 15, 2025 - Sponsor Presentations
• May 16-18, 2025 - Main Conference Days—Keynotes, Talks, Charlas, Expo Hall, and more
• May 19-22, 2025 - Sprints

Introducing PyCon US Co-Chair, and Future Conference Chair: Jon Banafato!:"Hi folks! I'm Jon, and I've been a Python community organizer for the past decade and change. You may recognize me from my work behind the scenes at PyGotham, volunteering at other Python conferences, or talking shop with other community organizers. I'm excited to be working with Elaine and the rest of the staff, organizers, and volunteers for the next few PyCon US conferences. Feel free to reach out to share your thoughts about the conference, and I look forward to seeing everyone back in Pittsburgh next year!"PyCon US 2025 Website is Live

You can now head over to the PyCon US 2025 website for all the conference details and more information about our sponsorship program.

Our design for this year’s event draws inspiration from the local fare of Pittsburgh infused with pops of colors and dynamic elements that reflect the uniqueness and character of our community.

We’re excited to collaborate with designers Malek Jerbi and Hamza Haj Taieb from Tunisia, whose illustrations and designs brought the PyCon US 2025 website to life. The design is brought together with the coordination of Georgi K and implemented by YupGup.
PyCon US Call for Proposals is Now OpenPyCon US 2025’s Call for Proposals is officially open for Talks, Tutorials, Posters, and Charlas! The deadline to submit for all tracks is December 19, 2024, 11:59 PM ET. You can view what time that is for you locally on our CfP countdown.

We need beginner, intermediate, and advanced proposals on all sorts of topics— and beginner, intermediate, and advanced speakers to give said presentations. You don’t need to be a 20-year veteran who has spoken at dozens of conferences. On all fronts, we need all types of people. Our community is comprised of a diverse set of people with unique skill sets, and we want our conference program to be a true reflection of that diversity.

For the new and first-time speakers, be sure to take advantage of the speaker mentorship program where you can be matched with experienced speakers who can help you with crafting your proposal. Check out the info on our Proposal Mentorship Program page.

For more information on where and how to submit your proposal, visit the Proposal Guidelines page on the PyCon US 2025 website.
Hatchery Program - Coming Soon!The PyCon US Hatchery is an effort to establish a path for the introduction of new tracks, summits, and demos at PyCon US. We are still finalizing the details of this program, so please stay tuned for more news about the PyCon US Hatchery 🐣 and start thinking of ideas you might want to contribute!
Sponsorship Has Tremendous ImpactSponsors are what make PyCon US and the Python Software Foundation possible. PyCon US is the main source of revenue for the PSF, the non-profit behind the Python language and the Python Packaging Index (PyPI), and the hub for the Python community.

PyCon US is the largest and longest-running Python gathering globally, with a diverse group of highly engaged attendees, many of whom you won’t find at other conferences. We’re excited to be able to provide our sponsors with opportunities to connect with and support the Python community. You’ll be face-to-face with talented developers, qualified recruits, and potential customers, access a large and diverse audience, as well as elevate your visibility and corporate identity within the Python community.

Check out our full menu of benefits,

What you can expect when you sponsor PyCon US and the PSF:

• Reach - Access to 2500+ attendees interested in your products and services and generate qualified leads.
• Brand strength - Be part of the biggest and most prestigious Python conference in the world and support the nonprofit organization behind the Python language.
• Connections - Networking with attendees in person to create connections and provide detailed information about your products and services.
• Recruiting - Access to qualified job candidates. If you’re hiring, there’s no better place to find Python developers than PyCon US.
• 12 months of benefits - Reach the Python community during PyCon US and beyond, with options for recognition on Python.org, PyPI.org, and more.

Because of the generosity of our sponsors, the Python Software Foundation is able to continually improve our support of Python and its community. Read more about the PSF here and in our most recent Annual Impact Report. You can help us ensure the Python ecosystem thrives for years to come. Apply to sponsor PyCon US and the PSF today! Head over to the Sponsorship Application page to learn more about the 2025 PSF sponsorship program and sign up to sponsor.

If you have any questions about sponsoring PyCon US and the PSF, please contact us at sponsors@python.org.Stay in the LoopAs we get closer to the event, the conference website is where you’ll find details for our call for proposals, registration launch, venue information, and everything PyCon US related! Be sure to subscribe here to the PyCon US Blog, follow @PyCon on Twitter, @pycon@fosstodon.org and @ThePSF@fosstodon.org on Mastodon, the PSF on LinkedIn, and subscribe to PyCon US 2025 News so you won’t miss a thing. Our official hashtag is #PyConUS.

Almost all of my Debian contributions this month were sponsored by Freexian.

You can also support my work directly via Liberapay.

Ansible

I noticed that Ansible had fallen out of Debian testing due to autopkgtest failures. This seemed like a problem worth fixing: in common with many other people, we use Ansible for configuration management at Freexian, and it probably wouldn’t make our sysadmins too happy if they upgraded to trixie after its release and found that Ansible was gone.

The problems here were really just slogging through test failures in both the ansible-core and ansible packages, but their test suites are large and take a while to run so this took some time. I was able to contribute a few small fixes to various upstreams in the process:

• test: Make git archive prefix fit in 32-bit ssize_t (though upstream went for a different approach)
• test: replace more deprecated assertEquals
• Fix import error on Python 3.13
• Make test_start_daemon_with_no_mock less flaky

This should now get back into testing tomorrow.

OpenSSH

Martin-Éric Racine reported that ssh-audit didn’t list the ext-info-s feature as being available in Debian’s OpenSSH 9.2 packaging in bookworm, contrary to what OpenSSH upstream said on their specifications page at the time. I spent some time looking into this and realized that upstream was mistakenly saying that implementations of ext-info-c and ext-info-s were added at the same time, while in fact ext-info-s was added rather later. ssh-audit now has clearer output, and the OpenSSH maintainers have corrected their specifications page.

I looked into a report of an ssh failure in certain cases when using GSS-API key exchange (which is a Debian patch). Once again, having integration tests was a huge win here: the affected scenario is quite a fiddly one, but I was able to set it up in the test, and thereby make sure it doesn’t regress in future. It still took me a couple of hours to get all the details right, but in the past this sort of thing took me much longer with a much lower degree of confidence that the fix was correct.

On upstream’s advice, I cherry-picked some key exchange fixes needed for big-endian architectures.

Python team

I packaged python-evalidate, needed for a new upstream version of buildbot.

The Python 3.13 transition rolls on. I fixed problems related to it in htmlmin, humanfriendly, postgresfixture (contributed upstream), pylint, python-asyncssh (contributed upstream), python-oauthlib, python3-simpletal, quodlibet, zope.exceptions, and zope.interface.

A trickier Python 3.13 issue involved the cgi module. Years ago I ported zope.publisher to the multipart module because cgi.FieldStorage was broken in some situations, and as a result I got a recommendation into Python’s “dead batteries” PEP 594. Unfortunately there turns out to be a name conflict between multipart and python-multipart on PyPI; python-multipart upstream has been working to disentangle this, though we still need to work out what to do in Debian. All the same, I needed to fix python-wadllib and multipart seemed like the best fit; I contributed a port upstream and temporarily copied multipart into Debian’s python-wadllib source package to allow its tests to pass. I’ll come back and fix this properly once we sort out the multipart vs. python-multipart packaging.

tzdata moved some timezone definitions to tzdata-legacy, which has broken a number of packages. I added tzdata-legacy build-dependencies to alembic and python-icalendar to deal with this in those packages, though there are still some other instances of this left.

I tracked down an nltk regression that caused build failures in many other packages.

I fixed Rust crate versioning issues in pydantic-core, python-bcrypt, and python-maturin (mostly fixed by Peter Michael Green and Jelmer Vernooij, but it needed a little extra work).

I fixed other build failures in entrypoints, mayavi2, python-pyvmomi (mostly fixed by Alexandre Detiste, but it needed a little extra work), and python-testing.postgresql (ditto).

I fixed python3-simpletal to tolerate future versions of dh-python that will drop their dependency on python3-setuptools.

I fixed broken symlinks in python-treq.

I removed (build-)depends on python3-pkg-resources from alembic, autopep8, buildbot, celery, flufl.enum, flufl.lock, python-public, python-wadllib (contributed upstream), pyvisa, routes, vulture, and zodbpickle (contributed upstream).

I upgraded astroid, asyncpg (fixing a Python 3.13 failure and a build failure), buildbot (noticing an upstream test bug in the process), dnsdiag, frozenlist, netmiko (fixing a Python 3.13 failure), psycopg3, pydantic-settings, pylint, python-asyncssh, python-bleach, python-btrees, python-cytoolz, python-django-pgtrigger, python-django-test-migrations, python-gssapi, python-icalendar, python-json-log-formatter, python-pgbouncer, python-pkginfo, python-plumbum, python-stdlib-list, python-tokenize-rt, python-treq (fixing a Python 3.13 failure), python-typeguard, python-webargs (fixing a build failure), pyupgrade, pyvisa, pyvisa-py (fixing a Python 3.13 failure), toolz, twisted, vulture, waitress (fixing CVE-2024-49768 and CVE-2024-49769), wtf-peewee, wtforms, zodbpickle, zope.exceptions, zope.interface, zope.proxy, zope.security, and zope.testrunner to new upstream versions.

I tried to fix a regression in python-scruffy, but I need testing feedback.

I requested removal of python-testing.mysqld.

What goes into building a spreadsheet application in Python that runs in the browser? How do you make it launch quickly, and where do you store the cells of data? This week on the show, we speak with Chris Laffra about his project, PySheets, and his book "Communication for Engineers."

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

PyCon US is the largest and longest-running annual gathering for the community using and developing the open-source Python programming language. It is produced and underwritten by the Python Software Foundation (PSF), the 501(c)(3) nonprofit organization dedicated to advancing and promoting Python and its community.

The PSF is a grant-giving non-profit and the revenue generated by PyCon US is essential to our continued community support and operations. Check out our Mission and our latest Annual Impact Report to get an idea of the scope of work our foundation is responsible for. Many misunderstand or are unaware of this relationship, but the PSF relies on maintaining a strict budget for the operation of PyCon US in order to continue its grants programs and other financial responsibilities.

With this in mind, PyCon US has taken on many expenses in the years following the pandemic, including the losses associated with canceling PyCon US 2020 and the move to PyCon US 2021 online. Additionally, inflation in event costs and lower projected sponsorships have combined to create a significant loss for PyCon US and the PSF in 2024. 

Because the PSF is a nonprofit, we keep PyCon US registration costs much lower than comparable technology conferences. Our goal is for PyCon US to remain accessible to the widest group possible. For us to continue to do so and ensure the sustainability of the event, we’ve had to make some changes to the structure of PyCon US. We know that an important piece of making changes to PyCon US is notice and awareness to our wonderful community– and we want to be transparent about the reasons behind them. 

Ticket Prices

To continue providing the content, program, and opportunities of PyCon US this year and in the future and to offset these rising costs, we’ve decided to raise ticket prices this year by $50 USD for the Individual and Corporate rates and $25 USD for the Student rate. 

New PyCon US 2025 registration prices: 

• Corporate: $800 USD
• Individual: $450 USD
• Student: $125 USD

By adjusting ticket prices, we hope to continue delivering a valuable and enriching conference experience while still being financially accessible to our community. 

PyCon US Online

PyCon US moved to online for the first year in 2021 during the pandemic and we’ve been proud to offer the Online attendance option for the past three years since. While we recognize the value that PyCon US Online provides in terms of accessibility, the challenges of delivering a seamless and engaging experience for both in-person and virtual attendees have been significant. From a cost perspective alone, cutting the online event will free up resources that can be put back into the in person and after-conference experience. 

PyCon US will record all Talk tracks, Keynotes and Lightning Talks on the main days of the conference (Friday - Sunday) and we plan to publish these recordings very quickly after the sessions take place so those who can’t join us in person can access them while the topics are still current. 

We appreciate the enthusiasm and participation of everyone who was part of the PyCon US Online experience and encourage everyone who can to join us in person in Pittsburgh this year. If you are unable to join, we welcome you to subscribe to our PyCon US YouTube channel to receive notifications of new content.

Tutorial Recordings

The pre-conference tutorials are a valuable part of the PyCon US conference and provide an opportunity for Pythonistas around the world to learn and grow their skills in hands-on sessions. 

Tutorial ticket sales are a large source of revenue for PyCon US, but in the past few years, the margin on the tutorials has declined due to rising costs associated with recording these sessions. 

Because tutorials are longer than talks and are in an interactive classroom-like setting, they require many resources such as power, better internet connectivity, and catering. Additionally, tutorial instructors are compensated due to the amount of work involved in preparing and providing these sessions. 

Since tutorial registration is at an additional cost, we want to prioritize the experience for those attending in person and to do so, we will be removing tutorial recordings this year. The value of the face-to-face learning, discussions, and hands-on engagement that occurs during the tutorials outweighs the benefits of capturing the sessions for later viewing and can’t be replicated in a recorded format. We hope this change will encourage more attendees to participate in the tutorials onsite and engage with the content the instructors work very hard to provide. 

Speaker Travel Grant Limit

The support of our community has allowed us to offer generous Travel Grants based on individual needs including travel, accommodation, and conference tickets. The Travel Grant program helps us shape a conference that is diverse on a number of axes from technical to geographic.

However, with decreased Travel Grant funds, PyCon US wants to ensure that we can still spread out our funding to support as many people as possible. Currently, a large part of the Travel Grant budget goes to speakers, and all speakers who are included on the proposal for an accepted talk are automatically eligible for a travel grant. For 2025, there will be a limit of two Travel Grant-funded speakers per accepted proposal across all tracks. We believe this will allow us to spread our funding capacity across our community and welcome as many Pythonistas to PyCon US as we can, while not impacting the quality and diversity of our programming.

Thank you!

PyCon US has been around for over 20 years, and we want to ensure it sticks around for many more! We recognize that these changes will impact many folks in a variety of ways, and we hope that our transparency about why these decisions were made will help bring understanding. 

By making these changes and being willing to adapt, PyCon US can continue to be a welcoming, community conference, where everyone can meet new people and learn new things, connect with old friends and tell people about their projects. We appreciate our community’s support and understanding regarding these changes. If you have any feedback or questions, please feel free to reach out to the organizing team.  

The Qt Quick 3D project was first announced and released in 2019. Since then, it has seen numerous enhancements. We have concentrated on boosting performance, introducing new features, and expanding the module's capabilities.