Feeds

KdeEcoTest

KdeEcoTest is an automation and testing tool which allows one to record and simulate user interactions with a Graphical User Interface. It is being developed as part of the FOSS Energy Efficiency Project to create usage scenario scripts for measuring the energy consumption of software in the KDE Eco initiative. Emmanuel Charruau is the original developer and lead maintainer of the project. In Season of KDE 2023 the tool was further improved by Mohamed Ibrahim under the tutelage of Karanjot Singh and Emmanuel Charruau.

One of the main goals in Season of KDE 2024 is to make the KdeEcoTest cross-platform so it can run on Windows and Linux systems (both Wayland and X11). Another necessary change in KdeEcoTest is to completely remove the use of libX11 and xdotool from the code base, which had issues with pixel coordinates while testing.

A benefit of a tool such as KdeEcoTest is that it is possible to create usage scenario scripts without having access to the source code of the software being tested. This is in contrast to a tool such as Selenium-AT-SPI, which requires access to the application's sources (QML file).

See how Selenium-AT-SPI helps with the KDE Eco project (Selenium-AT-SPI KDE Eco).

Getting Started: Platform Abstraction Layer

For the tool to be cross platform, it requires a platform abstraction layer, which is what Amartya Chakraborty and I, the SoK24 team, set out to write first. Any action, either simulated or read by the KdeEcoTest tool, can be separated into Window-based and Input device-based. The design goal is therefore to provide platform-independent interfaces for performing these actions.

We finished work on the abstraction layer and integrated it into the current code base with some minor changes. The layer provides two interfaces, a WindowHandler and an InputHandler (which have a platform-specific implementation), in order to access the underlying methods for taking window or input actions.

The team's next task was to implement these platform-specific classes. For the SoK24 project, Windows-specific classes were tasked to Amartya Chakraborty, while I set out to implement support for Wayland.

Adding Wayland Support: KDE's kdotool

From initial conversations between me, Emmanuel, and other members of the KdeEcoTest channel, we decided on using ydotool, a program written in C that simulates input devices with uinput. This enables the simulation of input devices on Wayland as well.

But ydotool was not enough as it did not support manipulating windows like xdotool did and we still required listening to input devices. For Window-based actions it requires communication with the underlying display server or compositor (in Wayland world), and Wayland currently did not have such a protocol. So we went back to the drawing board and tried different methods, even running the application as an X Client on Xwayland, which indeed worked but could hardly be called a solution to supporting Wayland.

Figure : Running as X client on Wayland using Xwayland. (Image from Wayland docs published under an MIT license.)

We thought that currently there did not exist any solution for Wayland, but we were wrong. It turns out that KWin, the KDE Plasma compositor for Wayland and X11, already had one! One fine day Emmanuel posted a message on the KDE Dev channel, and voilà we got our solution from a KDE developer in the form of a Rust tool, kdotool. The tool uses KWin’s dbus interface to upload a JavaScript file and access methods and properties provided by the KWin scripting API. Just love you, KDE!

Emmanuel brought kdotool’s author ‘genericity’ on board and we set sail. With some minor tweaks and upgrades we integrated kdotool into KdeEcoTest and we now have all window-related functionalities set up and working on X11 and Wayland. (Oh … and by the way that’s when I found out you can’t have gitsubmodules on invent.kde.org). The use of the KWin specific APIs means that KdeEcoTest will currently only work on KDE Plasma.

Special mention to ‘genericity’ for developing this great tool.

Adding Input Actions

With all window-related actions in, it was time to add support for input actions. I was testing this out with ydotool, but later dropped it and decided to directly access uinput. Reading through Linux kernel documentation, I found it's rather better to use the evdev interface, which also allows one to create and simulate devices and also to read inputs from existing ones.

evdev is the generic input event interface. It passes the events generated in the kernel straight to the program, with timestamps. The event codes are the same on all architectures and are hardware independent. The layout of the input event -

struct input_event { struct timeval time; unsigned short type; unsigned short code; unsigned int value; };

time is the timestamp, it returns the time at which the event happened. Type is for example EV_REL for relative movement, EV_KEY for a keypress or release. code is event code, for example REL_X or KEY_BACKSPACE. value is the value the event carries. Either a relative change for EV_REL, absolute new value for EV_ABS (joysticks ...), or 0 for EV_KEY for release, 1 for keypress and 2 for autorepeat.

You can find more details about the input event codes here, https://docs.kernel.org/input/event-codes.html.

libevdev sits below the process that handles input events, in between the kernel and that process. The stack would look like this:

• Kernel → libevdev → libinput → Wayland compositor → Wayland client

And for X11 using libinput:

• Kernel → libevdev → libinput → xf86-input-libinput → X server → X client

With the python-evdev binding for libevdev one could directly access evdev devices, but there is a catch: the script needs to be run as root, or the user needs permissions to read and write from /dev/input and /dev/uinput (and also /dev/console to run dumpkeys to get the keyboard drivers translation table). Considering the challenges, this was something we could live with.

My first plan was to directly use evdev to simulate and listen to input devices, but digging into it some more I found out that pynput had a uinput backend that, it turns out, was not fully implemented (it was partially implemented for keyboard). pynput is a cross-platform Python library that allows one to control and monitor input devices. Since pynput was already being used in KdeEcoTest and it supported all other systems (except Wayland), implementing a backend that uses libevdev would allow for using pynput and its already implemented methods and classes directly.

I finished writing a backend for pynput which allowed me to simulate and listen for mouse and keyboard events on Wayland. But there is one more problem with directly using libevdev. Input events generated from libevdev are passed to libinput before reaching the compositor.

Figure : libinput (Image from Wayland docs published under an MIT license.)

libinput processes these inputs and provides easy-to-use APIs to handle them. It also provides configurable interfaces, which means that inputs we simulate using uinput may not be interpreted the same way by the compositor. For, e.g. the delta values in EV_REL type events (Event type for relative movement) generated by evdev devices such as a mouse are converted into relative movement on screen using a function called pointer accelaration in libinput. This function depends on the pointer acceleration profiles configured by the user depending on the compositor. There are namely two: "adaptive and flat". We skip further low level details about this here, and just add that switching to "flat" profile provides 1:1 movement between the device and the pointer on-screen. This also means that the mouse acceleration profile needs to be set to flat when running KdeEcoTest.

Now all that was required was to alias InputHandler class's attributes to pynput classes. This may be improved in the future when individual level implementation is required.

With both now implemented and integrated, KdeEcoTest supports running on Wayland, and also with the contribution from Amartya Chakraborty it runs on Windows as well.

Finished Platform Abstraction layer

The platform abstraction layer in current form sets the required backend for pynput, imports the necessary backend and exposes the required interfaces WindowHandler and InputHandler. When running on KDE Plasma, since we require kdotool, it additionally adds kdotool to the path.

import platform as _platform import os from pathlib import Path import importlib def set_handlers(package): module = None if _platform.system() == 'Windows': module = '.win32' os.environ['PYNPUT_BACKEND']='win32' elif _platform.system() == 'Linux': if os.environ.get('XDG_SESSION_DESKTOP',None) == 'KDE': module = '.kwin' os.environ['PYNPUT_BACKEND']='uinput' #add kdotool to path KDOTOOL_PATH = os.path.join(Path(os.path.dirname(__file__)).parent,'bin/kdotool/release') os.environ['PATH'] = KDOTOOL_PATH + ':' + os.environ['PATH'] elif os.environ.get('XDG_SESSION_TYPE',None) == 'X11': print('Platform Not Supported') exit() try: return importlib.import_module(module,package) except ImportError: raise ImportError('Platform not supported') backend = set_handlers(__name__) WindowHandler = backend.WindowActionHandler InputHandler = backend.InputActionHandler del backend __all__ = [ WindowHandler, InputHandler ] Looking Forward

The listener for mouse events in pynput's uinput backend currently does not listen for touchpad events, because the event codes generated made it confusing to implement one. I think a better approach would be to use libinput for listening for events, as this would provide abstraction and also make it easier to write code for handling events, but libinput currently lacks a python binding.

Interested?

Of course it is far from complete. KdeEcoTest is still very much a work in progress and we're happy to have more developers looking to contribute. Hop on to the KdeEco and KdeEcoTest matrix channels to join the developers and follow the tool's development.

See https://invent.kde.org/teams/eco/feep for the FEEP repository, which hosts the KdeEcoTest tool among others.

Thank you to Emmanuel, 'genericity', Amartya, Joseph, as well as the whole KDE community and Wayland developers for helping out with the project.

We have a cabin out in the forest, and when I say "out in the forest" I mean "in a national forest subject to regulation by the US Forest Service" which means there's an extremely thick book describing the things we're allowed to do and (somewhat longer) not allowed to do. It's also down in the bottom of a valley surrounded by tall trees (the whole "forest" bit). There used to be AT&T copper but all that infrastructure burned down in a big fire back in 2021 and AT&T no longer supply new copper links, and Starlink isn't viable because of the whole "bottom of a valley surrounded by tall trees" thing along with regulations that prohibit us from putting up a big pole with a dish on top. Thankfully there's LTE towers nearby, so I'm simply using cellular data. Unfortunately my provider rate limits connections to video streaming services in order to push them down to roughly SD resolution. The easy workaround is just to VPN back to somewhere else, which in my case is just a Wireguard link back to San Francisco.

This worked perfectly for most things, but some streaming services simply wouldn't work at all. Attempting to load the video would just spin forever. Running tcpdump at the local end of the VPN endpoint showed a connection being established, some packets being exchanged, and then… nothing. The remote service appeared to just stop sending packets. Tcpdumping the remote end of the VPN showed the same thing. It wasn't until I looked at the traffic on the VPN endpoint's external interface that things began to become clear.

This probably needs some background. Most network infrastructure has a maximum allowable packet size, which is referred to as the Maximum Transmission Unit or MTU. For ethernet this defaults to 1500 bytes, and these days most links are able to handle packets of at least this size, so it's pretty typical to just assume that you'll be able to send a 1500 byte packet. But what's important to remember is that that doesn't mean you have 1500 bytes of packet payload - that 1500 bytes includes whatever protocol level headers are on there. For TCP/IP you're typically looking at spending around 40 bytes on the headers, leaving somewhere around 1460 bytes of usable payload. And if you're using a VPN, things get annoying. In this case the original packet becomes the payload of a new packet, which means it needs another set of TCP (or UDP) and IP headers, and probably also some VPN header. This still all needs to fit inside the MTU of the link the VPN packet is being sent over, so if the MTU of that is 1500, the effective MTU of the VPN interface has to be lower. For Wireguard, this works out to an effective MTU of 1420 bytes. That means simply sending a 1500 byte packet over a Wireguard (or any other VPN) link won't work - adding the additional headers gives you a total packet size of over 1500 bytes, and that won't fit into the underlying link's MTU of 1500.

And yet, things work. But how? Faced with a packet that's too big to fit into a link, there are two choices - break the packet up into multiple smaller packets ("fragmentation") or tell whoever's sending the packet to send smaller packets. Fragmentation seems like the obvious answer, so I'd encourage you to read Valerie Aurora's article on how fragmentation is more complicated than you think. tl;dr - if you can avoid fragmentation then you're going to have a better life. You can explicitly indicate that you don't want your packets to be fragmented by setting the Don't Fragment bit in your IP header, and then when your packet hits a link where your packet exceeds the link MTU it'll send back a packet telling the remote that it's too big, what the actual MTU is, and the remote will resend a smaller packet. This avoids all the hassle of handling fragments in exchange for the cost of a retransmit the first time the MTU is exceeded. It also typically works these days, which wasn't always the case - people had a nasty habit of dropping the ICMP packets telling the remote that the packet was too big, which broke everything.

What I saw when I tcpdumped on the remote VPN endpoint's external interface was that the connection was getting established, and then a 1500 byte packet would arrive (this is kind of the behaviour you'd expect for video - the connection handshaking involves a bunch of relatively small packets, and then once you start sending the video stream itself you start sending packets that are as large as possible in order to minimise overhead). This 1500 byte packet wouldn't fit down the Wireguard link, so the endpoint sent back an ICMP packet to the remote telling it to send smaller packets. The remote should then have sent a new, smaller packet - instead, about a second after sending the first 1500 byte packet, it sent that same 1500 byte packet. This is consistent with it ignoring the ICMP notification and just behaving as if the packet had been dropped.

All the services that were failing were failing in identical ways, and all were using Fastly as their CDN. I complained about this on social media and then somehow ended up in contact with the engineering team responsible for this sort of thing - I sent them a packet dump of the failure, they were able to reproduce it, and it got fixed. Hurray!

(Between me identifying the problem and it getting fixed I was able to work around it. The TCP header includes a Maximum Segment Size (MSS) field, which indicates the maximum size of the payload for this connection. iptables allows you to rewrite this, so on the VPN endpoint I simply rewrote the MSS to be small enough that the packets would fit inside the Wireguard MTU. This isn't a complete fix since it's done at the TCP level rather than the IP level - so any large UDP packets would still end up breaking)

I've no idea what the underlying issue was, and at the client end the failure was entirely opaque: the remote simply stopped sending me packets. The only reason I was able to debug this at all was because I controlled the other end of the VPN as well, and even then I wouldn't have been able to do anything about it other than being in the fortuitous situation of someone able to do something about it seeing my post. How many people go through their lives dealing with things just being broken and having no idea why, and how do we fix that?

(Edit: thanks to this comment, it sounds like the underlying issue was a kernel bug that Fastly developed a fix for - under certain configurations, the kernel fails to associate the MTU update with the egress interface and so it continues sending overly large packets)

comments

Today we are talking about CKEditor 4 End of Life, Moving to CKEditor 5, and what you can expect from CKEditor 5 now and in the future with guest Wim Leers. We’ll also cover CKEditor 5 Premium Features as our module of the week.

For show notes visit: www.talkingDrupal.com/438

Topics

• CKEditor 4 end of life June 2023
• Issues people might see if they are still on CKE4
• Why a third party library and not roll our own
• Are there other alternatives
• Why did Drupal decide on CKEditor
• Drupal 10 moved to CKE5 How should people update
• Upgrade gotchas
• What's new in CKE5
• What is on the roadmap regarding Drupal and CKE5
• Is there going to be a CKE6
• Native Web Components
• Does CKE in core affect Gutenberg

Resources

• CKEditor 4 End Of Life
  • Contrib
  • Core, prioritized upstream blockers
  • Core, all issues blocked on upstream (currently 29), if we look just at bugs there's 17 (1 of which is critical, 6 major)
• Drastically improve the linking experience in CKEditor 5
• Drupal Image ability to opt in to SVG image uploads
• Native
  and
  UX
  • https://www.drupal.org/project/drupal/issues/3274635
  • Extended html filter
  • Views / ckeditor custom elements
  Guests

  Wim Leers - wimleers.com Wim Leers

  Hosts

  Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Ivan Stegic - ten7.com ivanstegic

  MOTW Correspondent

  Martin Anderson-Clutz - mandclu

  • Brief description:
    • Have you ever wanted to offer your content creators advanced capabilities like real-time collaboration? There’s a module for that.
  • Module name/project name:
    • CKEditor 5 Premium Features
  • Brief history
    • How old: created in Sep 2022 by Wiktor Walc, although recent releases are by Wojciech (vOYchekh) Kukowski, both of CKSource, the company behind CKEditor (Wiktor was on episode 372 https://talkingdrupal.com/372)
    • Current version available: 1.2.5 which works with Drupal 9 and 10
  • Maintainership
    • Actively maintained, latest release in the past month
    • User Guide available, link is in the README
    • Number of open issues: 16, 8 of which are bugs
  • Usage stats:
    • 159 sites
  • Module features and usage
    • To me, the most compelling features enabled by this module are the ones that turn your Drupal WYSIWYG into a robust collaboration tool, similar to what users may be used to in tools like Google Docs or Office 365
    • Real-time inline comments and changes from multiple users
    • Track changes to suggest ways the content could be improved
    • A history of changes made in the WYSIWYG, independent of the saved Drupal revisions
    • Tag users with @ mentions to have them notified
    • There’s also a Productivity Pack to enhance your WYSIWYG, and again some of these will be familiar to users that also use popular online collaboration tools
    • A document outline that uses heading within your content to make navigation for moving quickly within the document
    • Can generate a linked Table of Contents, which will automatically update as headings are added or changed
    • Slash commands to execute actions
    • Enhanced Paste from Office, to preserve complex incoming content structures, but with clean HTML as the result
    • And more!
    • Another premium feature is the ability to export to Word or PDF, and it can also restore full screen editing, a feature that didn’t make the transition from CKEditor 4 to 5, as part of the open source offering
    • Finally, it also includes an AI Assistant that provides yet another interesting way to empower your content authors to leverage AI tools for their writing, including the ability to change the style, length, or tone of selected content using pre-made prompts, or generate content with custom queries. It also works with a number of different models out of the box, so you’re not restricted to ChatGPT
    • The module is open source but using these premium features does require a subscription. The pricing will depend on the number of active users and which features you need, so if you’d like more information you can use the contact form at ckeditor.com
    • Also worth mentioning here that the team at Palantir has released a YouTube video of an open source collaborative editor that they’re calling Edit Together. It’s based on the ProseMirror rich-text editor framework, and the blog where they announced it mentioned a mid-2024 release, but that was back in Jul 2023 and I haven’t been able to find any updates since then

The Pantheon Autopilot Toolbar module addresses navigation deficiencies within Pantheon's Dashboard UI, enabling efficient access to Autopilot. Kevin Reynen, the Principal Web Applications Developer at the University of Colorado, discusses the module's creation and its significance. Also, understand Pantheon's response to the issue. Explore the collaborative efforts and responses surrounding this crucial dashboard enhancement.

Interviewing is an art form. Live interviews are very much so. Written interviews with prepared questions rarely get to that level because seldom, and that too by random chance, a question would sprout based on a previous answer, which would have given scope for further explanations and explorations in a live setup.

Hello Readers,

Hope everyone had an amazing week! While casually scrolling through social media, I stumbled upon a random post featuring the timeless wisdom of Nelson Mandela: 

 "It always seems impossible until it's done."

 Reflecting on this, I couldn't help but relate it to my own experiences. Often, when I embark on a new task, it feels daunting and insurmountable. Yet, as I push through and complete it, I realize it wasn't as challenging as I initially imagined. I'm sure many of you can resonate with this feeling.

Often, our hesitation to take on responsibilities stems from a fear of making mistakes. This brings to my mind the words of Ruth Cheesley, the Project Lead of Mautic. 

"The hardest thing to do is to raise your hand and say 'I’d like to help,' so jump over that hurdle and see where it takes you!"

Ruth engaged in an email correspondence with me prior to India's first and APAC region's largest Mautic Conference, MautiCon India 2024. The interview delves into her journey from beneficiary to influential contributor within the open-source community. Sharing insights into her path, she recounts pivotal moments that led her to Mautic and discusses the project's evolution, community engagement, and future aspirations. 

As someone who lives with disbility and has gulped down many harsh experiences, Ruth with her ethos motivates us to challenge ourselves to overcome this fear and embrace opportunities to contribute. You never know where it might lead you!

Now, let's shift our focus and explore some of the latest news stories and articles we covered last week.

Alka Elizabeth discussed the newly developed Contribution Health Dashboard with Alex Moreno in a comprehensive interview. Alex discusses CHD development, technological underpinnings, and potential impact on the global diversity of the Drupal community. Discover the challenges faced, future steps, and how the Drupal community can contribute to the dashboards' enhancement with this conversation.

The DropTimes is embarking on a special project aimed at gathering insights directly from individuals within the Drupal community and its users. Your perspectives will play a crucial role in shaping a series of articles titled "Drupal's Innovation and Future: 2024 and Beyond." In our inaugural article, authored by Kazima Abbas, we engage with a variety of experts to delve into the challenges and opportunities present within the Drupal ecosystem.

An article by Irina Zaks, Co-Founder & CTO of Fibonacci Web Studio, published in TDT, delves into the topic of managing ambitious projects with limited budgets, specifically focusing on the transition from Drupal 7 to Backdrop CMS and streamlining the migration process for site-builders.

Kazima Abbas reached out to the organizers of the upcoming Drupal Delhi Meetup Group scheduled for February 24, 2024, to delve into what participants can anticipate at the event. Additionally, this weekend features one more notable gathering: Florida DrupalCamp 2024, happening from February 23rd to 25th.

Drupal enthusiasts are invited to participate as Scholarship Mentors for DrupalConPortland, offering support and guidance to scholarship recipients throughout the DrupalCon experience. Stay updated with this week's Drupal events by checking the published list here.

Victoria Spagnolo's blog underscores a noteworthy proposal within the Drupal community: the elimination of support for Windows in production environments in Drupal 11. Learn more here. Tim Doyle, CEO of the Drupal Association, unveiled substantial improvements to the Drupal Certified Partner program, slated to be implemented on April 1, 2024. Announced on February 9, 2024, nominations are now being accepted for the 2024 Aaron Winborn Award, presented by the Drupal Community Working Group. Learn more about this here.

In a notable development, Piyush Poddar, currently leading Sales and Partnerships at Axelerant and renowned for his open-source advocacy through foundational work with the Jaipur Drupal User Group and key contributions to the Drupal Association of India, has been appointed as the newest member of the Drupal Association Board, signaling a new era of global collaboration, as confirmed by the Association earlier this week.

Alex Moreno, announced the extension of The Drupal Association's Bounty Program, which aims to foster innovation and progress within the community by streamlining contributions in essential areas and addressing long-standing challenges.
Announcing a collaborative effort, The Drupal Association and Cloud-IAM are working together to enhance Drupal.org's user experience through the implementation of a secure, GDPR-compliant KeyCloak single sign-on (SSO) system. 

Selwyn Polit's latest update to Drupal at Your Fingertips introduces a revamped interface and improved functionality, utilizing the VitePress Vite & Vue Powered Static Site Generator. To delve deeper into this update, click here.

We acknowledge that there are more stories to share. However, due to constraints in selection, we must pause further exploration for now.

To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.

Thank you,

Sincerely
Elma John
Sub-editor, TheDropTimes.

When your Python project relies on external packages, you need to make sure you’re using the right version of each package. After an update, a package might not work as it did before. A dependency manager like Python Poetry helps you specify, install, and resolve external packages in your projects. This way, you can be sure that you always work with the correct dependency version on every machine.

In this tutorial, you’ll learn how to:

• Create a new project using Poetry
• Add Poetry to an existing project
• Configure your project through pyproject.toml
• Pin your project’s dependency versions
• Install dependencies from a poetry.lock file
• Run basic Poetry commands using the Poetry CLI

Poetry helps you create new projects or maintain existing projects while taking care of dependency management for you. It uses the pyproject.toml file, which has become the standard for defining build requirements in modern Python projects.

To complete this tutorial and get the most out of it, you should have a basic understanding of virtual environments, modules and packages, and pip.

While you’ll focus on dependency management in this tutorial, Poetry can also help you build a distribution package for your project. If you want to share your work, then you can use Poetry to publish your project on the Python Packaging Index (PyPI).

Free Bonus: Click here to get access to a free 5-day class that shows you how to avoid common dependency management issues with tools like Pip, PyPI, Virtualenv, and requirements files.

Take Care of Prerequisites

Before diving into the nitty-gritty of Python Poetry, you’ll take care of some prerequisites. First, you’ll read a short overview of the terminology that you’ll encounter in this tutorial. Next, you’ll install Poetry itself.

Learn the Relevant Terminology

If you’ve ever used an import statement in one of your Python scripts, then you’ve worked with modules and packages. Some of them might have been Python files you wrote on your own. Others could’ve been standard library modules that ship with Python, like datetime. However, sometimes, what Python provides isn’t enough. That’s when you might turn to external modules and packages maintained by third parties.

When your Python code relies on such external modules and packages, they become the requirements or dependencies of your project.

To find packages contributed by the Python community that aren’t part of the Python standard library, you can browse PyPI. Once you’ve found a package you’re interested in, you can use Poetry to manage and install that package in your project. Before seeing how this works, you need to install Poetry on your system.

Install Poetry on Your Computer

Poetry is distributed as a Python package itself, which means that you can install it into a virtual environment using pip, just like any other external package:

• Windows
• Linux + macOS

Windows PowerShell (venv) PS> python -m pip install poetry Copied! Shell (venv) $ python3 -m pip install poetry Copied!

This is fine if you just want to quickly try it out. However, the official documentation strongly advises against installing Poetry into your project’s virtual environment, which the tool must manage. Because Poetry depends on several external packages itself, you’d run the risk of a dependency conflict between one of your project’s dependencies and those required by Poetry. In turn, this could cause Poetry or your code to malfunction.

In practice, you always want to keep Poetry separate from any virtual environment that you create for your Python projects. You also want to install Poetry system-wide to access it as a stand-alone application regardless of the specific virtual environment or Python version that you’re currently working in.

There are several ways to get Poetry running on your computer, including:

1. A tool called pipx
2. The official installer
3. Manual installation
4. Pre-built system packages

In most cases, the recommended way to install Poetry is with the help of pipx, which takes care of creating and maintaining isolated virtual environments for command-line Python applications. After installing pipx, you can install Poetry by issuing the following command in your terminal window:

• Windows
• Linux + macOS

Windows PowerShell PS> pipx install poetry Copied! Shell $ pipx install poetry Copied!

While this command looks very similar to the one you saw previously, it’ll install Poetry into a dedicated virtual environment that won’t be shared with other Python packages.

Read the full article at https://realpython.com/dependency-management-python-poetry/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Kdenlive 23.08.5 has been released, featuring a multitude of bug fixes, including many issues related to nested sequences and same-track transitions. This release temporarily removes Movit effects until they are stable for production. However, the primary focus of this release was to continue the ongoing efforts in transitioning to Qt6 and KF6.

It’s important to note that, due to this transition, we regret to inform our Mac users that a package for this release won’t be available. We kindly request them to wait for the 24.02 release, expected by the end of the month.

Full changelog

• Fix undocked widgets don’t have a title bar to allow moving / re-docking. Commit.
• Multi guides export: replace slash and backslash in section names to fix rendering. Commit. Fixes bug #480845.
• Fix sequence corruption on project load. Commit. Fixes bug #480776.
• Fix multiple archiving issues. Commit. Fixes bug #456346.
• Fix possible sequence corruption. Commit. Fixes bug #480398.
• Fix sequences folder id not correctly restored on project opening. Commit.
• Fix Luma issue. Commit. See bug #480343.
• Fix subtitles not covering transparent zones. Commit. Fixes bug #480350.
• Group resize: don’t allow resizing a clip to length < 1. Commit. Fixes bug #480348.
• Fix crash cutting grouped overlapping subtitles. Don’t allow the cut anymore, add test. Commit. Fixes bug #480316.
• Fix clip monitor not updating when clicking in a bin column like date or description. Commit. Fixes bug #480148.
• Fix start playing at end of timeline. Commit. Fixes bug #479994.
• Fix save clip zone from timeline adding an extra frame. Commit. Fixes bug #480005.
• Fix clips with mix cannot be cut, add test. Commit. See bug #479875.
• Fix project monitor loop clip. Commit.
• Fix monitor offset when zooming back to 1:1. Commit.
• Fix sequence effects lost. Commit. Fixes bug #479788.
• Improved fix for center crop issue. Commit.
• Fix center crop adjust not covering full image. Commit. Fixes bug #464974.
• Disable Movit until it’s stable (should have done that a long time ago). Commit.
• Fix cannot save list of project files. Commit. Fixes bug #479370.
• Fix editing title clip with a mix can mess up the track. Commit. Fixes bug #478686.
• Fix audio mixer cannot enter precise values with keyboard. Commit.
• Prevent, detect and possibly fix corrupted project files, fix feedback not displayed in project notes. Commit. See bug #472849.
• Test project’s active timeline is not always the first sequence. Commit.
• Ensure secondary timelines are added to the project before being loaded. Commit.
• Ensure autosave is not triggered when project is still loading. Commit.
• Fix variable name shadowing. Commit.
• When switching timeline tab without timeline selection, don’t clear effect stack if it was showing a bin clip. Commit.
• Fix crash pressing del in empty effect stack. Commit.
• Ensure check for HW accel is also performed if some non essential MLT module is missing. Commit.
• Fix tests. Commit.
• Fix closed sequences losing properties, add more tests. Commit.
• Don’t attempt to load timeline sequences more than once. Commit.
• Fix timeline groups lost after recent commit on project save. Commit.
• Ensure we always use the correct timeline uuid on some clip operations. Commit.
• Add animation: remember last used folder. Commit. See bug #478688.
• Refresh effects list after downloading an effect. Commit.
• Fix audio or video only drag of subclips. Commit. Fixes bug #478660.
• Fix editing title clip duration breaks title (recent regression). Commit.
• Glaxnimate animations: use rawr format instead of Lottie by default. Commit. Fixes bug #478685.
• Fix timeline focus lost when dropping an effect on a clip. Commit.
• Fix dropping lots of clips in Bin can cause freeze on abort. Commit.
• Right click on a mix now shows a mix menu (allowing deletion). Commit. Fixes bug #442088.
• Don’t add mixes to disabled tracks. Commit. See bug #442088.
• Allow adding a mix without selection. Commit. See bug #442088.
• Remove line missing from merge commit. Commit.
• Fix proxied playlist clips (like stabilized clips) rendered as interlaced. Commit. Fixes bug #476716.
• Always keep all timeline models opened. Commit. See bug #478745.

The post Kdenlive 23.08.5 released appeared first on Kdenlive.

One of the major fields where Python shines is in data science. For data exploration and cleaning, Python has many powerful tools, such as pandas and polar. For visualization, Python has Matplotlib.

When you're building GUI applications with PyQt, you can have access to all those tools directly from within your app. While it is possible to embed matplotlib plots in PyQt, the experience doesn't feel entirely native. So, for highly integrated plots, you may want to consider using the PyQtGraph library instead.

PyQtGraph is built on top of Qt's native QGraphicsScene, so it gives better drawing performance, particularly for live data. It also provides interactivity and the ability to customize plots according to your needs.

In this tutorial, you'll learn the basics of creating plots with PyQtGraph. You'll also explore the different plot customization options, including background color, line colors, line type, axis labels, and more.

Table of Contents

• Installing PyQtGraph
• Creating a PlotWidget Instance
• Customizing PyQtGraph Plots
  • Background Color
  • Line Color, Width, and Style
  • Line Markers
  • Plot Titles
  • Axis Labels
  • Plot Legends
  • Background Grid
  • Axis Range
• Multiple Plot Lines
• Creating Dynamic Plots
• Conclusion

Installing PyQtGraph

To use PyQtGraph with PyQt6, you first need to install the library in your Python environment. You can do this using pip as follows:

sh $ python -m pip install pyqtgraph

Once the installation is complete, you will be able to import the module into your Python code. So, now you are ready to start creating plots.

Creating a PlotWidget Instance

In PyQtGraph, all plots use the PlotWidget class. This widget provides a canvas on which we can add and configure many types of plots. Under the hood, PlotWidget uses Qt's QGraphicsScene class, meaning that it's fast, efficient, and well-integrated with the rest of your app.

The code below shows a basic GUI app with a single PlotWidget in a QMainWindow:

python import pyqtgraph as pg from PyQt6 import QtWidgets class MainWindow(QtWidgets.QMainWindow): def __init__(self): super().__init__() # Temperature vs time plot self.plot_graph = pg.PlotWidget() self.setCentralWidget(self.plot_graph) minutes = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] temperature = [30, 32, 34, 32, 33, 31, 29, 32, 35, 30] self.plot_graph.plot(minutes, temperature) app = QtWidgets.QApplication([]) main = MainWindow() main.show() app.exec()

In this short example, you create a PyQt app with a PlotWidget as its central widget. Then you create two lists of sample data for time and temperature. The final step to create the plot is to call the plot() methods with the data you want to visualize.

The first argument to plot() will be your x coordinate, while the second argument will be the y coordinate.

In all the examples in this tutorial, we import PyQtGraph using import pyqtgraph as pg. This is a common practice in PyQtGraph examples to keep things tidy and reduce typing.

If you run the above application, then you'll get the following window on your screen:

Basic PyQtGraph plot: Temperature vs time.

PyQtGraph's default plot style is quite basic — a black background with a thin (barely visible) white line. Fortunately, the library provides several options that will allow us to deeply customize our plots.

In the examples in this tutorial, we'll create the PyQtGraph widget in code. To learn how to embed PyQtGraph plots when using Qt Designer, check out Embedding custom widgets from Qt Designer.

In the following section, we'll learn about the options we have available in PyQtGraph to improve the appearance and usability of our plots.

Customizing PyQtGraph Plots

Because PyQtGraph uses Qt's QGraphicsScene to render the graphs, we have access to all the standard Qt line and shape styling options for use in plots. PyQtGraph provides an API for using these options to draw plots and manage the plot canvas.

Below, we'll explore the most common styling features that you'll need to create and customize your own plots with PyQtGraph.

Background Color

Beginning with the app skeleton above, we can change the background color by calling setBackground() on our PlotWidget instance, self.graphWidget. The code below sets the background to white by passing in the string "w":

python from PyQt6 import QtWidgets import pyqtgraph as pg class MainWindow(QtWidgets.QMainWindow): def __init__(self): super().__init__() # Temperature vs time plot self.plot_graph = pg.PlotWidget() self.setCentralWidget(self.plot_graph) self.plot_graph.setBackground("w") minutes = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] temperature = [30, 32, 34, 32, 33, 31, 29, 32, 35, 30] self.plot_graph.plot(minutes, temperature) app = QtWidgets.QApplication([]) main = MainWindow() main.show() app.exec()

Calling setBackground() with "w" as an argument changes the background of your plot to white, as you can see in the following window:

PyQtGraph plot with a white background.

There are a number of colors available using single letters, as we did in the example above. They're based on the standard colors used in Matplotlib. Here are the most common codes:

Letter Code Color "b" Blue "c" Cian "d" Grey "g" Green "k" Black "m" Magenta "r" Red "w" White "y" Yellow

In addition to these single-letter codes, we can create custom colors using the hexadecimal notation as a string:

python self.plot_graph.setBackground("#bbccaa") # Hex

We can also use RGB and RGBA values passed in as 3-value and 4-value tuples, respectively. We must use values in the range from 0 to 255:

python self.plot_graph.setBackground((100, 50, 255)) # RGB each 0-255 self.plot_graph.setBackground((100, 50, 255, 25)) # RGBA (A = alpha opacity)

The first call to setBackground() takes a tuple representing an RGB color, while the second call takes a tuple representing an RGBA color.

We can also specify colors using Qt's QColor class if we prefer it:

python from PyQt6 import QtGui # ... self.plot_graph.setBackground(QtGui.QColor(100, 50, 254, 25))

Using QColor can be useful when you're using specific QColor objects elsewhere in your application and want to reuse them in your plots. For example, say that your app has a custom window background color, and you want to use it in the plots as well. Then you can do something like the following:

python color = self.palette().color(QtGui.QPalette.Window) # ... self.plot_graph.setBackground(color)

In the first line, you get the GUI's background color, while in the second line, you use that color for your plots.

Line Color, Width, and Style

Plot lines in PyQtGraph are drawn using the Qt QPen class. This gives us full control over line drawing, as we would have in any other QGraphicsScene drawing. To use a custom pen, you need to create a new QPen instance and pass it into the plot() method.

In the app below, we use a custom QPen object to change the line color to red:

python from PyQt6 import QtWidgets import pyqtgraph as pg class MainWindow(QtWidgets.QMainWindow): def __init__(self): super().__init__() # Temperature vs time plot self.plot_graph = pg.PlotWidget() self.setCentralWidget(self.plot_graph) self.plot_graph.setBackground("w") pen = pg.mkPen(color=(255, 0, 0)) time = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] temperature = [30, 32, 34, 32, 33, 31, 29, 32, 35, 45] self.plot_graph.plot(time, temperature, pen=pen) app = QtWidgets.QApplication([]) main = MainWindow() main.show() app.exec()

Here, we create a QPen object, passing in a 3-value tuple that defines an RGB red color. We could also define this color with the "r" code or with a QColor object. Then, we pass the pen to plot() with the pen argument.

PyQtGraph plot with a red plot line.

By tweaking the QPen object, we can change the appearance of the line. For example, you can change the line width in pixels and the style (dashed, dotted, etc.), using Qt's line styles.

Update the following lines of code in your app to create a red, dashed line with 5 pixels of width:

python from PyQt6 import QtCore, QtWidgets # ... pen = pg.mkPen(color=(255, 0, 0), width=5, style=QtCore.Qt.DashLine)

The result of this code is shown below, giving a 5-pixel, dashed, red line:

PyQtGraph plot with a red, dashed, and 5-pixel line

You can use all other Qt's line styles, including Qt.SolidLine, Qt.DotLine, Qt.DashDotLine, and Qt.DashDotDotLine. Examples of each of these lines are shown in the image below:

Qt's line styles.

To learn more about Qt's line styles, check the documentation about pen styles. There, you'll all you need to deeply customize the lines in your PyQtGraph plots.

Line Markers

For many plots, it can be helpful to use point markers in addition or instead of lines on the plot. To draw a marker on your plot, pass the symbol you want to use as a marker when calling plot(). The following example uses the plus sign as a marker:

python self.plot_graph.plot(hour, temperature, symbol="+")

In this line of code, you pass a plus sign to the symbol argument. This tells PyQtGraph to use that symbol as a marker for the points in your plot.

If you use a custom symbol, then you can also use the symbolSize, symbolBrush, and symbolPen arguments to further customize the marker.

The value passed as symbolBrush can be any color, or QBrush instance, while symbolPen can be any color or a QPen instance. The pen is used to draw the shape, while the brush is used for the fill.

Go ahead and update your app's code to use a blue marker of size 15, on a red line:

python pen = pg.mkPen(color=(255, 0, 0)) self.plot_graph.plot( time, temperature, pen=pen, symbol="+", symbolSize=20, symbolBrush="b", )

In this code, you pass a plus sign to the symbol argument. You also customize the marker size and color. The resulting plot looks something like this:

PyQtGraph plot with a plus sign as a point marker.

In addition to the + plot marker, PyQtGraph supports the markers shown in the table below:

Character Marker Shape "o" Circle "s" Square "t" Triangle "d" Diamond "+" Plus "t1" Triangle pointing upwards "t2" Triangle pointing right side "t3" Triangle pointing left side "p" Pentagon "h" Hexagon "star" Star "x" Cross "arrow_up" Arrow Up "arrow_right" Arrow Right "arrow_down" Arrow Down "arrow_left" Arrow Left "crosshair" Crosshair

You can use any of these symbols as markers for your data points. If you have more specific marker requirements, then you can also use a QPainterPath object, which allows you to draw completely custom marker shapes.

Plot Titles

Plot titles are important to provide context around what is shown on a given chart. In PyQtGraph, you can add a main plot title using the setTitle() method on the PlotWidget object. Your title can be a regular Python string:

python self.plot_graph.setTitle("Temperature vs Time")

You can style your titles and change their font color and size by passing additional arguments to setTitle(). The code below sets the color to blue and the font size to 20 points:

python self.plot_graph.setTitle("Temperature vs Time", color="b", size="20pt")

In this line of code, you set the title's font color to blue and the size to 20 points using the color and size arguments of setTitle().

You could've even used CSS style and basic HTML tag syntax if you prefer, although it's less readable:

python self.plot_graph.setTitle( '<span style="color: blue; font-size: 20pt">Temperature vs Time</span>' )

In this case, you use a span HTML tag to wrap the title and apply some CSS styles on to of it. The final result is the same as suing the color and size arguments. Your plot will look like this:

PyQtGraph plot with title.

Your plot looks way better now. You can continue customizing it by adding informative lables to both axis.

Axis Labels

When it comes to axis labels, we can use the setLabel() method to create them. This method requires two arguments, position and text.

python self.plot_graph.setLabel("left", "Temperature (°C)") self.plot_graph.setLabel("bottom", "Time (min)")

The position argument can be any one of "left", "right", "top", or "bottom". They define the position of the axis on which the text is placed. The second argument, text is the text you want to use for the label.

You can pass an optional style argument into the setLabel() method. In this case, you need to use valid CSS name-value pairs. To provide these CSS pairs, you can use a dictionary:

python styles = {"color": "red", "font-size": "18px"} self.plot_graph.setLabel("left", "Temperature (°C)", **styles) self.plot_graph.setLabel("bottom", "Time (min)", **styles)

Here, you first create a dictionary containing CSS pairs. Then you pass this dictionary as an argument to the setLabel() method. Note that you need to use the dictionary unpacking operator to unpack the styles in the method call.

Again, you can use basic HTML syntax and CSS for the labels if you prefer:

python self.plot_graph.setLabel( "left", '<span style="color: red; font-size: 18px">Temperature (°C)</span>' ) self.plot_graph.setLabel( "bottom", '<span style="color: red; font-size: 18px">Time (min)</span>' )

This time, you've passed the styles in a span HTML tag with appropriate CSS styles. In either case, your plot will look something like this:

PyQtGraph plot with axis labels.

Having axis labels highly improves the readability of your plots as you can see in the above example. So, it's a good practice that you should keep in mind when creating your plots.

Plot Legends

In addition to the axis labels and the plot title, you will often want to show a legend identifying what a given line represents. This feature is particularly important when you start adding multiple lines to a plot.

You can add a legend to a plot by calling the addLegend() method on the PlotWidget object. However, for this method to work, you need to provide a name for each line when calling plot().

The example below assigns the name "Temperature Sensor" to the plot() method. This name will be used to identify the line in the legend:

python self.plot_graph.addLegend() # ... self.plot_graph.plot( time, temperature, name="Temperature Sensor", pen=pen, symbol="+", symbolSize=15, symbolBrush="b", )

Note that you must call addLegend() before you call plot() for the legend to show up. Otherwise, the plot won't show the legend at all. Now your plot will look like the following:

PyQtGraph plot with legend.

The legend appears in the top left by default. If you would like to move it, you can drag and drop the legend elsewhere. You can also specify a default offset by passing a 2-value tuple to the offset parameter when calling the addLegend() method. This will allow you to specify a custom position for the legend.

Background Grid

Adding a background grid can make your plots easier to read, particularly when you're trying to compare relative values against each other. You can turn on the background grid for your plot by calling the showGrid() method on your PlotWidget instance. The method takes two Boolean arguments, x and y:

python self.plot_graph.showGrid(x=True, y=True)

In this call to the showGrid() method, you enable the grid lines in both dimensions x and y. Here's how the plot looks now:

PyQtGraph plot with grid.

You can toggle the x and y arguments independently, according to the dimension on which you want to enable the grid lines.

Axis Range

Sometimes, it can be useful to predefine the range of values that is visible on the plot or to lock the axis to a consistent range regardless of the data input. In PyQtGraph, you can do this using the setXRange() and setYRange() methods. They force the plot to only show data within the specified ranges.

Below, we set two ranges, one on each axis. The first argument is the minimum value, and the second is the maximum:

python self.plot_graph.setXRange(1, 10) self.plot_graph.setYRange(20, 40)

The first line of code sets the x-axis to show values between 1 and 10. The second line sets the y-axis to display values between 20 and 40. Here's how this changes the plot:

PyQtGraph plot with axis ranges

Now your plot looks more consistent. The axis show fix scales that are specifically set for the possible range of input data.

Multiple Plot Lines

It is common to have plots that involve more than one dependent variable. In PyQtGraph, you can plot multiple variables in a single chart by calling .plot() multiple times on the same PlotWidget instance.

In the following example, we plot temperatures values from two different sensors. We use the same line style but change the line color. To avoid code repetition, we define a new plot_line() method on our window:

python from PyQt6 import QtWidgets import pyqtgraph as pg class MainWindow(QtWidgets.QMainWindow): def __init__(self): super().__init__() # Temperature vs time plot self.plot_graph = pg.PlotWidget() self.setCentralWidget(self.plot_graph) self.plot_graph.setBackground("w") self.plot_graph.setTitle("Temperature vs Time", color="b", size="20pt") styles = {"color": "red", "font-size": "18px"} self.plot_graph.setLabel("left", "Temperature (°C)", **styles) self.plot_graph.setLabel("bottom", "Time (min)", **styles) self.plot_graph.addLegend() self.plot_graph.showGrid(x=True, y=True) self.plot_graph.setXRange(1, 10) self.plot_graph.setYRange(20, 40) minutes = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] temperature_1 = [30, 32, 34, 32, 33, 31, 29, 32, 35, 30] temperature_2 = [32, 35, 40, 22, 38, 32, 27, 38, 32, 38] pen = pg.mkPen(color=(255, 0, 0)) self.plot_line("Temperature Sensor 1", minutes, temperature_1, pen, "b") pen = pg.mkPen(color=(0, 0, 255)) self.plot_line("Temperature Sensor 2", minutes, temperature_2, pen, "r") def plot_line(self, name, minutes, temperature, pen, brush): self.plot_graph.plot( minutes, temperature, name=name, pen=pen, symbol="+", symbolSize=15, symbolBrush=brush, ) app = QtWidgets.QApplication([]) main = MainWindow() main.show() app.exec()

The custom plot_line() method on the main window does the hard work. It accepts a name to set the line name for the plot legend. Then it takes the time and temperature arguments. The pen and brush arguments allow you to tweak other features of the lines.

To plot separate temperature values, we'll create a new list called temperature_2 and populate it with random numbers similar to our old temperature, which now is temperature_1. Here's the plot looks now:

PyQtGrap plot with two lines.

You can play around with the plot_line() method, customizing the markers, line widths, colors, and other parameters.

Creating Dynamic Plots

You can also create dynamic plots with PyQtGraph. The PlotWidget can take new data and update the plot in real time without affecting other elements. To update a plot dynamically, we need a reference to the line object that the plot() method returns.

Once we have the reference to the plot line, we can call the setData() method on the line object to apply the new data. In the example below, we've adapted our temperature vs time plot to accept new temperature measures every minute. Note that we've set the timer to 300 milliseconds so that we don't have to wait an entire minute to see the updates:

python from random import randint from PyQt6 import QtCore, QtWidgets import pyqtgraph as pg class MainWindow(QtWidgets.QMainWindow): def __init__(self): super().__init__() # Temperature vs time dynamic plot self.plot_graph = pg.PlotWidget() self.setCentralWidget(self.plot_graph) self.plot_graph.setBackground("w") pen = pg.mkPen(color=(255, 0, 0)) self.plot_graph.setTitle("Temperature vs Time", color="b", size="20pt") styles = {"color": "red", "font-size": "18px"} self.plot_graph.setLabel("left", "Temperature (°C)", **styles) self.plot_graph.setLabel("bottom", "Time (min)", **styles) self.plot_graph.addLegend() self.plot_graph.showGrid(x=True, y=True) self.plot_graph.setYRange(20, 40) self.time = list(range(10)) self.temperature = [randint(20, 40) for _ in range(10)] # Get a line reference self.line = self.plot_graph.plot( self.time, self.temperature, name="Temperature Sensor", pen=pen, symbol="+", symbolSize=15, symbolBrush="b", ) # Add a timer to simulate new temperature measurements self.timer = QtCore.QTimer() self.timer.setInterval(300) self.timer.timeout.connect(self.update_plot) self.timer.start() def update_plot(self): self.time = self.time[1:] self.time.append(self.time[-1] + 1) self.temperature = self.temperature[1:] self.temperature.append(randint(20, 40)) self.line.setData(self.time, self.temperature) app = QtWidgets.QApplication([]) main = MainWindow() main.show() app.exec()

The first step to creating a dynamic plot is to get a reference to the plot line. In this example, we've used a QTimer object to set the measuring interval. We've connected the update_plot() method with the timer's timeout signal.

Theupdate_plot() method does the work of updating the data in every interval. If you run the app, then you will see a plot with random data scrolling to the left:

The time scale in the x-axis changes as the stream of data provides new values. You can replace the random data with your own real data. You can take the data from a live sensor readout, API, or from any other stream of data. PyQtGraph is performant enough to support multiple simultaneous dynamic plots using this technique.

Conclusion

In this tutorial, you've learned how to draw basic plots with PyQtGraph and customize plot components, such as lines, markers, titles, axis labels, and more. For a complete overview of PyQtGraph methods and capabilities, see the PyQtGraph documentation. The PyQtGraph repository on Github also has a complete set of plot examples.

In this tutorial, we will explore Pointwise Mutual Information (PMI), a valuable metric for identifying words that co-occur. You will also learn how to implement PMI in Python and R.

Table of Contents What is Pointwise mutual information?

PMI helps us to find related words. In other words, it explains how likely the co-occurrence of two words than we would expect by chance.

For example the word "Data Science" has a specific meaning when these two words "Data" and "Science" go together. Otherwise meaning of these two words are independent. Similarly "Great Britain" is meaningful since we know the word "Great" can be used with several other words but not so relevant in meaning like "Great UK, Great London, Great Dubai etc."

When words 'w1' and 'w2' are independent, their joint probability is equal to the product of their individual probabilities. Imagine when the formula of PMI as shown above returns 0, it means the numerator and denominator is same and then taking log of 1 returns 0. In simple words, it means the words together has NO specific meaning or relevance.

In general, we focus on finding pairs of words that have high pointwise mutual information. It means having high joint probability with the other word but having not so high probability if words are considered separately. It implies that this word pair has a specific meaning. Steps to Calculate PMI

Let's understand with an example. Suppose you have the following text and you are asked to calculate PMI scores based on that.

this is a foo bar bar black sheep foo bar bar black sheep foo bar bar black sheep shep bar bar black sentence To read this article in full, please click hereThis post appeared first on ListenData

Websites without servers or networking About • Blog • Newsletter • Links Websites without servers or networking

Published 2024-02-19 by Seth Larson
Reading time: minutes

Note that this article is about a feature that, at the time of writing, has been almost universally disabled or hampered on many platforms. Expect some or all things to not work.

Everyone in tech at one point has joked about serverless websites actually using servers behind the scenes. This isn't that, this is web content you can access with a URL that doesn't require any networking.

ServerServerInternetInternetNetworked WebNetworked WebBrowserBrowserHTTP ServerHTTP ServerParse URLParse URLHTTP GET /HTTP GET /Render ContentRender ContentHTTP 200 OKHTTP 200 OKLocal WebLocal WebBrowserBrowserParse URLParse URLRender ContentRender ContentText is not SVG - cannot display
Blue boxes are the sources of content in each scheme

I'll call the web as we know it today the "networked" web, because it relies on HTTP and the internet to deliver content to browsers and other user agents. The "local" web I'll be theory-crafting in this article doesn't require networking or HTTP at all, instead embedding content directly into the URL.

The basis of this is based on a web feature that still works on some platforms, below there's a tool for creating your own "local web" URL:

No HTTP needed!">No HTTP needed!"><marquee>No HTTP needed!</marquee> Copy URL

Note that pasting the URL into a mobile browser is unlikely to work. You might have to try on a desktop browser like Firefox, or simply trust that it works and continue onwards.

To use the tool, try making some edits in the text area, click the "Copy URL" button, and pasting it into a new browser tab navigation bar.

You should see whatever web content you wrote appear in your browser. Taking a look in the browser inspector network tab will show no activity.

What is a data URL?

Data URLs are URLs prefixed with the data: scheme and allow embedding content directly into the URL itself instead of indicating where a browser should request data from. Data URLs are capable are encoding any content type, including HTML content (i.e. text/html).

Data URLs are made of 3 components, a mime-type, an optional base64 encoding "flag", and the data value which is either encoded as text or base64-encoded bytes depending on the base64 flag:

data:<mime-type>[,base64];<data>

For example, below is the data URL encoding the data for Hello, world! with the mime type text/plain:

data:text/plain,Hello,%20world! What would a local web look like?

I love networking as much as the next person and there are obvious benefits, but there are also downsides to serving most websites across networks:

• Websites require a server somewhere and (unless your name is an IP address) likely a domain name. Both of these things cost money to run.
• URLs don't encode the information of the resource, only its potential location. This means resources referenced by URLs can be lost if the server at the other end ceases to exist.
• All web content requires electricity and connectivity for both client and server to work.
• Geographically distant users or users with worse network conditions have degraded performance.

The combination of browsers, HTML, CSS, and JavaScript are an incredible platform for building experiences, it'd be great if we could take advantage of those benefits without any of the downsides listed above. Let's build a local web with data URLs?

Referring to top-level data: URLs as being part of the "web" is an oxymoron, the web is meant to be an interconnected graph where our data URLs are functionally isolated nodes on that graph.

Sharing URLs

Right away data URLs have an ergonomics problem. Because they encode all the information in the URL itself it's tough to imagine typing one out with the same ease that you'd type out "example.com". If we're assuming that we're completely offline we also can't use a search index like DuckDuckGo or Google.

Luckily there's already a mechanism for getting long URLs into the browser: QR codes!

QR codes mean we can encode our data URLs into an image that can be scanned with a camera. Unfortunately this would mean that our data URLs are only shareable to devices with cameras (sorry laptops) but given the circumstances of "no internet" I think it's more likely interested users would be using a phone instead of a laptop.

QR code for the data URL of "example.com"
The content uses 1,290 of the maximum 2,953 bytes of information encodable in a binary QR code

You'll notice that the example.com QR code doesn't scan with your camera's built-in QR parser. Encoding a data URL into a QR code isn't against the QR standard (which allows arbitrary data, deferring to scanners for deciding what to do with the data) so this means the parser itself is refusing to process the data URL.

Dynamic content

One of the downsides of data URLs is they're static, whatever resources are embedded in them are there forever. New content requires either JavaScript and an internet connection or to create a new data URL.

This constraint is similar to physical media formats like cartridges, records, and disks. Creators of these physical media formats had to get everything right before they were distributed to users, you can't patch or update the media after!

If we carry on using QR codes as a means of distributing local web content we can use either screens to update QR codes automatically (after considering whether the information can be displayed directly to the screen) or using stickers or labels which are cheap to print if content needs to be updated on a semi-regular basis in a centralized location.

Encodes the above text area as a data URL QR code Size constraints

Websites can be any size, and frequently are on the order of megabytes (!) in terms of total content size. Data URLs encoded into QR codes can't be any length as they are limited in size in two ways.

The maximum amount of data a QR code can encode depends on multiple variables. QR codes can be different "versions" (i.e. "sizes"), have different error correction formats (L, M, Q, and H), and be a different "type" (Numeric, Alphanumeric, Binary, and Kanji).

Since we need more than alphanumerics to encode base64 data and the : character we'll need to use the "Binary" type. Optimizing for total amount of data with low error correction and version 40 we can fit a maximum of 2,953 characters into a QR code.

URLs have no maximum size according to standards, however in practice many browsers will limit URLs to a maximum of 2,000 characters, so to be on the safe side we'll be constrained to 2,000 characters in a data URL.

Base64 generally increases the size of a string by ~1.3x, so dividing 2,000 by ~1.3 we get ~1,538 bytes of data encodable into our data URLs. Quite a bit smaller than multi-megabytes websites of today.

Use-cases for data URLs

So given all of the above constraints, what niche would data URLs fill for real-life usage?

• Small amounts of static data that doesn't need to update frequently
• Users are likely to have mobile phones when encountering the data URL
• Users are unlikely to have network connectivity OR the distributor of information wants to minimize spend/operations by not running a website
• Users may want to access information after opening the data URL
• Provide additional information mixed with information your phone still has access to (think GPS, accelerometer, contacts, pictures)

Using the above constraints I can think of a few cases:

• Maps for remote locations or trails. Phones typically have GPS signal even without internet connectivity. QR codes could be placed at trailhead signs.
• Providing a decent initial web "landing page" for a QR code in areas where internet connectivity is spotty or unavailable so would provide a negative experience after scanning the QR code. For example advertisements on trains, subways, or airplanes. These data URL "landing pages" could automatically upgrade to the real-deal once internet connectivity is restored.
• Restaurant food and beverage menus are frequently encoded into a QR code. This requires serving a PDF or webpage from an actual website so costs money to maintain. Instead restaurant owners could encode their menu into a data URL for a zero-cost solution.

I'm sure there are many more example, send me any interesting ones you think of yourself.

Data URLs are dead, long live data URLs

So what's the biggest barrier to the local web being possible? Data URLs are dead.

Data URLs were essentially blocked from use as top-level domains (and thus in QR codes) by all major browsers in 2017 citing phishing, being confusing for users, and interacting strange ways with other standards (for example, should a data: URL be considered "secure" or "insecure"?) According to caniuse.com, the base href can't be data: for 92.75% of global browser users.

This ends our hypothetical local web journey, hope you had fun! 🥲

QR codes generated using the MIT-licensed 'qrcode-svg' project. Copyright (c) 2020 datalog.

&&&&&&&&&&&&&&&&&&&&&&&&

Thanks for reading! ♡ Did you find this article helpful and want more content like it? Get notified of new posts by subscribing to the RSS feed or the email newsletter.

This work is licensed under CC BY-SA 4.0

Posted on February 19, 2024
Tags: madeof:atoms, craft:sewing, FreeSoftWear

CW for body size change mentions

Just like the corset, I also needed a new pair of jeans.

Back when my body size changed drastically of course my jeans no longer fit. While I was waiting for my size to stabilize I kept wearing them with a somewhat tight belt, but it was ugly and somewhat uncomfortable.

When I had stopped changing a lot I tried to buy new ones in the same model, and found out that I was too thin for the menswear jeans of that shop. I could have gone back to wearing women’s jeans, but I didn’t want to have to deal with the crappy fabric and short pockets, so I basically spent a few years wearing mostly skirts, and oversized jeans when I really needed trousers.

Meanwhile, I had drafted a jeans pattern for my SO, which we had planned to make in technical fabric, but ended up being made in a cotton-wool mystery mix for winter and in linen-cotton for summer, and the technical fabric version was no longer needed (yay for natural fibres!)

It was clear what the solution to my jeans problems would have been, I just had to stop getting distracted by other projects and draft a new pattern using a womanswear block instead of a menswear one.

Which, in January 2024 I finally did, and I believe it took a bit less time than the previous one, even if it had all of the same fiddly pieces.

I already had a cut of the same cotton-linen I had used for my SO, except in black, and used it to make the pair this post is about.

The parametric pattern is of course online, as #FreeSoftWear, at the usual place. This time it was faster, since I didn’t have to write step-by-step instructions, as they are exactly the same as the other pattern.

Making also went smoothly, and the result was fitting. Very fitting. A big too fitting, and the standard bum adjustment of the back was just enough for what apparently still qualifies as a big bum, so I adjusted the pattern to be able to add a custom amount of ease in a few places.

But at least I had a pair of jeans-shaped trousers that fit!

Except, at 200 g/m² I can’t say that fabric is the proper weight for a pair of trousers, and I may have looked around online1 for some denim, and, well, it’s 2024, so my no-fabric-buy 2023 has not been broken, right?

Let us just say that there may be other jeans-related posts in the near future.

1. I had already asked years ago for denim at my local fabric shops, but they don’t have the proper, sturdy, type I was looking for.↩︎

I came across some conda packages that didn’t work properly on my M1 Mac (Apple Silicon – ARM processor) the other day. They installed fine, but gave segmentation faults when run. So, I wanted to run the x64 (Intel) versions of these packages instead.

I haven’t actually needed to do this since I got a M1 Mac (a testament to the quality and range of Arm conda packages available these days through conda-forge), so I wasn’t sure how to do it.

A bit of Googling and experimenting led to this nice simple set of instructions. The upshot of this is you don’t need to install another version of Anaconda/miniconda/etc – you can just create a x64 environment in your existing conda install.

So:

1. Run CONDA_SUBDIR=osx-64 conda create -n your_environment_name python

   This is a standard command to create a conda environment, but with CONDA_SUBDIR=osx-64 prepended to the command. This sets the CONDA_SUBDIR environment variable, and tells conda to use packages in the osx-64 subdirectory of the package server, rather than the standard osx-arm64 (M1 Mac) subdirectory. This is what gets you the x64 packages.

2. When this command finishes, you will have a new x64 conda environment. You can then activate it with conda activate your_environment_name
3. Now we need to tell conda to always use this CONDA_SUBDIR setting when using this environment, otherwise any future installs in this environment will use the default CONDA_SUBDIR and things will get very confused. We can do this by setting a conda environment config setting to tell conda to set a specific environment variable when you activate the environment. Do this by running: conda env config vars set CONDA_SUBDIR=osx-64
4. The output of that command will warn you to deactivate and reactivate the environment, so do this conda deactivate conda activate your_environment_name
5. That’s it! You now have a x64 environment which you can install packages in using standard conda install commands.

As I wrote a bit back, I had a really, really bad fourth quarter in 2023. As new years approached, and we were getting ready to go on a ski trip, I wasn’t even sure if and how much I’ll be able to ski.

And I felt so out of it that I didn’t even buy a ski pass for the whole week, just bought one day to see if a) I still like, and b) my knee can deal with it. And, of course, it was good.

It was good enough that I ended up skiing the entire week, and my knee got better during the week. WTH?! I don’t understand this anymore, but it was good. Good enough that this early year trip put me back on track and I started doing sports again.

But the main point is, that during this ski week, and talking to the teacher, I realised that my ski equipment is getting a bit old. I bought everything roughly ten years ago, and while they still hold up OK, my ski skills have improved since then. I said to myself, 10 years is a good run, I’ll replace this year the skis, next year the boot & helmet, etc.

I didn’t expect much from new skis - I mean, yes, better skis, but what does “better” mean? Well, once I’ve read enough forum posts, apparently the skis I selected are “that good”, which to me meant they’re not bad.

Oh my, how wrong I was! Double, triple wrong! Rather than fighting with the skis, it’s enough to think what I wand to do, and the skis do it. I felt OK-ish, maybe 10% limited by my previous skis, but the new skis are really good and also I know that I’m just at 30% or so of the new skis - so room to grow. For now, I am able to ski faster, longer, and I feel less tired than before. I’ve actually compared and I can do twice the distance in a day and feel slightly less tired at the end. I’ve moved from “this black is cool but a bit difficult, I’ll do another run later in the day when I’ve recovered” to “how cool, this blacks is quite empty of people, let’s stay here for 2-3 more rounds”.

The skis are new, and I haven’t used them on all the places I’m familiar with - but the upgrade is huge. The people on the ski forum were actually not exaggerating, I realise now. Stöckli++, and they’re also made in Switzerland. Can’t wait to get back to Saas Fee and to the couple of slopes that were killing me before, to see how they feel now.

So, very happy with this choice. I’d be even happier if my legs were less damaged, but well, you can’t win them all.

And not last, the skis are also very cool looking 😉

In 2008 I wrote about the idea of having a scheduled release for Debian and other distributions as Mark Shuttleworth had proposed [1]. I still believe that Mark’s original idea for synchronised release dates of Linux distributions (or at least synchronised feature sets) is a good one but unfortunately it didn’t take off.

Having been using Ubuntu a bit recently I’ve found the version numbering system to be really good. Ubuntu version 16.04 was release in April 2016, it’s support ended 5 years later in about April 2021, so any commonly available computers from 2020 should run it well and versions of applications released in about 2017 should run on it. If I have to support a Debian 10 (Buster) system I need to start with a web search to discover when it was released (July 2019). That suggests that applications packaged for Ubuntu 18.04 are likely to run on it.

If we had the same numbering system for Debian and Ubuntu then it would be easier to compare versions. Debian 19.06 would be obviously similar to Ubuntu 18.04, or we could plan for the future and call it Debian 2019.

Then it would be ideal if hardware vendors did the same thing (as car manufacturers have been doing for a long time). Which versions of Ubuntu and Debian would run well on a Dell PowerEdge R750? It takes a little searching to discover that the R750 was released in 2021, but if they called it a PowerEdge 2021R70 then it would be quite obvious that Ubuntu 2022.04 would run well on it and that Ubuntu 2020.04 probably has a kernel update with all the hardware supported.

One of the benefits for the car industry in naming model years is that it drives the purchase of a new car. A 2015 car probably isn’t going to impress anyone and you will know that it is missing some of the features in more recent models. It would be easier to get management to sign off on replacing old computers if they had 2015 on the front, trying to estimate hidden costs of support and lost productivity of using old computers is hard but saying “it’s a 2015 model and way out of date” is easy.

There is a history of using dates as software versions. The “Reference Policy” for SE Linux [2] (which is used for Debian) has releases based on date. During the Debian development process I upload policy to Debian based on the upstream Git and use the same version numbering scheme which is more convenient than the “append git date to last full release” system that some maintainers are forced to use. The users can get an idea of how much the Debian/Unstable policy has diverged from the last full release by looking at the dates. Also an observer might see the short difference between release dates of SE Linux policy and Debian release freeze dates as an indication that I beg the upstream maintainers to make a new release just before each Debian freeze – which is expactly what I do.

When I took over the Portslave [3] program I made releases based on date because there were several forks with different version numbering schemes so my options were to just choose a higher number (which is OK initially but doesn’t scale if there are more forks) or use a date and have people know that the recent date is the most recent version. The fact that the latest release of Portslave is version 2010.04.19 shows that I have not been maintaining it in recent years (due to lack of hardware as well as lack of interest), so if someone wants to take over the project I would be happy to help them do so!

I don’t expect people at Dell and other hardware vendors to take much notice of my ideas (I have tweeted them photographic evidence of a problem with no good response). But hopefully this will start a discussion in the free software community.

• [1] https://etbe.coker.com.au/2008/05/13/release-dates-for-debian/
• [2] https://github.com/SELinuxProject/refpolicy/releases
• [3] https://doc.coker.com.au/projects/portslave/

Related posts:

1. New Portslave release after 5 Years I’ve just uploaded Portslave version 2010.03.30 to Debian, it replaces...
2. Release Dates for Debian Mark Shuttleworth has written an interesting post about Ubuntu release...
3. New Debian release and new DPL Ingo Juergensmann has blogged in detail about the new release...

Get ready for the future with Drupfan's checklist for a fast and secure upgrade from Drupal 9 to Drupal 10. As Drupal 11's release approaches the end of 2024, ensuring your website is up-to-date and compatible is essential.

Check out this short tutorial to check the difference between the Python SQL libraries that are used in Python to connect to SQL databases. There are mostly 3 such libraries or call them adaptors namely sqlite3, pymysql, and mysql-connector-python. Introduction About Python SQL Libraries In the Python ecosystem, SQL libraries play a crucial role in […]

The post Difference Between 3 Python SQL Libraries appeared first on TechBeamers.