Feeds

Drupal's popularity is soaring because it's easy to use and flexible enough to fit any project. Businesses love it for its ability to grow with them, whether they're just starting out or already established. With each update, Drupal improves, offering more features and making it easier for everyone to build amazing websites.

One big thing to watch out for is the Starshot Initiative. It's all about making Drupal easier to use by adding some cool features from the start. Think of it as getting a new phone with all the apps you need installed. Starshot is like that for Drupal, making it simpler for everyone to get started and build something great.

The best part about Drupal is its community. People worldwide come together to share ideas, help each other, and improve Drupal. Whether you're a beginner or an expert, there's always something new to learn and someone to help you.

As Drupal grows, so does its potential. More and more businesses are choosing Drupal because it's reliable, adaptable, and just plain works. With initiatives like Starshot on the horizon, Drupal's future looks brighter.

With that said, let's now explore what updates and news we have covered last week:

In the interview with Alka Elizabeth, sub-editor at The Drop Times, Esmeralda Braad-Tijhoff, discusses her background, rise in the Drupal community, and ongoing efforts to promote inclusivity and collaboration within the tech industry. She shares insights into her roles with the Stichting Drupal Nederland and the Network of European Drupal Associations (NEDA), reflecting on her achievements and vision for the future of open-source solutions in public sector projects.

I had the opportunity to connect with some of the speakers of the upcoming DrupalJam to gain their insights on their sessions. Frederik Wouters sheds light on integrating RAG and OpenAI technologies within organizational infrastructures, emphasizing the significance of data quality and privacy considerations. Mikko Hämäläinen delves into the complexities of Digital Experience Platforms (DXP), elucidating their role in addressing customer expectations and internal silos in customer management. Finally, Mathias Bolt Lesniak advocates for the importance of open-source software in governmental policies, urging society to rediscover its benefits and advocate for its adoption.

Additionally, PHPCamp took place on 8th June 2024. I spoke with the organizers to provide insights into what attendees could expect. Amit Kumar Singh, a key organizer, shared his thoughts on why PHPCamp holds significance in the tech community.

Recently, Jay Callicott, an experienced Drupal Architect, introduced a new tool: DrupalX. DrupalX aims to simplify the process of building enterprise-level websites using Drupal. The Drop Times approached him for a comment, where he highlighted DrupalX's focus on improving the Drupal development experience, particularly for enterprise projects. Jay anticipates DrupalX will be beneficial in addressing common challenges developers face. Click here to read more about DrupalX.

The Drupal Starshot Initiative is picking up speed, and if you're interested in getting involved, now's the perfect time! The complete session calendar is now accessible, featuring a range of interactive Zoom calls. These sessions are tailored to offer updates and guidance on how you can contribute to this thrilling initiative.

After its unveiling at DrupalCon Portland 2024, the initial Drupal Starshot session, spearheaded by Dries Buytaert on May 31, 2024, attracted over 200 participants. The session discussed key topics such as community involvement, funding strategies, and governance plans. These discussions underscored the project's mission to streamline Drupal site construction through improved features and accessibility.

The Drupal Starshot project has introduced its new leadership team headed by Dries Buytaert. This team comprises key figures such as Technical Lead Tim Plunkett, User Experience Lead Cristina Chumillas, Product Owner Pamela Barone, and Contribution Coordinator Gábor Hojtsy. Together, they are committed to elevating the Drupal platform by prioritizing product vision, technical excellence, and user experience.

The Drupal Association has invited leaders of Local Associations from Asia to the inaugural Local Association Leaders meet-up on June 11, 2024. This initiative aims to strengthen the success of Drupal Local Associations by directly involving community leaders committed to advancing the Drupal project within their respective regions.

To celebrate Pride Month 2024, the Drupal Association is highlighting international LGBTQ+ organizations and donating proceeds from Pride-themed apparel sales. Community members can vote for the recipient organization, and they're encouraged to share their Pride stories for potential social media features.

The Drop Times has officially become a media partner for Drupaljam 2024, set to take place on June 12, 2024, at Fabrique Utrecht. We'll offer extensive coverage of all the latest updates throughout the event. But that's not all! The Drop Times has also been named the official Media Partner for DrupalCamp Spain 2024. This event is a cornerstone gathering for the Drupal community and is scheduled to be held from October 24 to 26 at the Hotel DeLoix Aqua Center. Stay tuned for comprehensive coverage of both Drupaljam 2024 and DrupalCamp Spain 2024!

Twin Cities Drupal Camp has opened the Call for Submissions for its upcoming event on September 12-13, 2024. With rolling session acceptance, early submissions have a higher chance of selection. The event will feature general sessions lasting 45 minutes, allowing for interactive audience Q&A sessions.

Registration for DrupalCamp Asheville 2024 is now open. The camp, which will take place from July 12 to 14, 2024, offers an exciting lineup of events. The camp includes three days of training, an unconference, various sessions, and community activities, all set against the scenic backdrop of Asheville. 

In other news, MidCamp will return with new dates in 2025, from May 20 to 22, at the DePaul University Student Center in Chicago. This change aligns with an earlier DrupalCon, providing the community a great opportunity to gather in the spring. Key events leading up to MidCamp include opening the call for speakers in the fall of 2024, with final selections and the schedule announcement expected in late January or early February 2025.

It summarises the week's major updates, though we couldn't include every story in this selection. Until next week... 

To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.

Thank you,
Sincerely
Kazima Abbas
Sub-editor, The Drop Times

May was packed with exciting updates and events in the Python community. This month saw the release of the first beta version of Python 3.13, the conclusion of PyCon US 2024, and the announcement of the keynote speakers for EuroPython 2024. Additionally, PEP 649 has been delayed until the Python 3.14 release, and the Python Software Foundation published its 2023 Annual Impact Report.

Get ready to explore the recent highlights!

Join Now: Click here to join the Real Python Newsletter and you'll never miss another Python tutorial, course update, or post.

The First Beta Version of Python 3.13 Released

After nearly a year of continuous development, the first beta release of Python 3.13 was made available to the general public. It marks a significant milestone in Python’s annual release cycle, officially kicking off the beta testing phase and introducing a freeze on new features. Beyond this point, Python’s core developers will shift their focus to only identifying and fixing bugs, enhancing security, and improving the interpreter’s performance.

While it’s still months before the final release planned for October 2024, as indicated by the Python 3.13 release schedule, third-party library maintainers are strongly encouraged to test their packages with this new Python version. The goal of early beta testing is to ensure compatibility and address any issues that may arise so that users can expect a smoother transition when Python 3.13 gets officially released later this year.

Although you shouldn’t use a beta version in any of your projects, especially in production environments, you can go ahead and try out the new version today. To check out Python’s latest features, you must install Python 3.13.0b1 using one of several approaches.

Note: If you’d like to share your feedback or file a bug against a pre-release development version, then open an issue on Python’s GitHub repository.

The quickest and arguably the most straightforward way to manage multiple Python versions alongside your system-wide interpreter is to use a tool like pyenv in the terminal:

Shell $ pyenv install 3.13.0b1 $ pyenv shell 3.13.0b1 $ python Python 3.13.0b1 (main, May 15 2024, 10:41:55) [GCC 13.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> Copied!

The highlighted line brings the first beta release of Python 3.13 onto your computer, while the following command temporarily sets the path to the python executable in your current shell session. As a result, the python command points to the specified Python interpreter.

Alternatively, you can use a Python installer, which you’ll find at the bottom of the downloads page, or run Python in an isolated Docker container to keep it completely separate from your operating system. However, for ultimate control, you can try building the interpreter from source code based on the instructions in the README file. This method will let you experiment with more advanced features, like turning off the GIL.

Unlike previous Python releases, which introduced a host of tangible syntactical features that you could get your hands on, this one mainly emphasizes internal optimizations and cleanup. That said, according to the official release summary document, there are a few notable new features that will be immediately visible to most Python programmers:

• A better standard Python REPL
• Improved error messages

Some of these features don’t work on Windows at the moment because they rely on Unix-specific libraries, so you won’t see any difference unless you’re a macOS or Linux user. The good news is that Windows support is coming in the second beta release, which will arrive soon, thanks to Real Python team member Anthony Shaw.

Read the full article at https://realpython.com/python-news-may-2024/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

It typically occurs when you click on a website and become disinterested in utilizing it because there are no language preferences. In addition, you will be more interested in browsing the website if you can easily navigate its user interface and select your preferred language, regardless of where you live. 

According to some reports: Multilingual websites can reach 75% more internet users and improve user experience, as 60% of global consumers prefer browsing in their native language. Businesses with such websites see a 70% average increase in conversion rates. That’s why it is preferred that you must create a multilingual website to help with website traffic and your business strategy.

How?

Another new-ish package that I’ve never got around to writing about on my blog is offline_folium. It has a somewhat niche use-case, but it seems like a few people have found it useful.

In brief, it allows you to use the folium package for creating interactive maps from Python, but without an internet connection. Folium is built on top of the Leaflet web mapping library – and it loads all the relevant JS and CSS files directly from CDNs. This means that if you try and use folium without an internet connection it just doesn’t work. Offline_folium works around this by downloading the files when you do have an internet connection, and then re-writing the links to the files to point to the offline versions.

I originally created this package when doing freelance work on a project for the UK Navy – they wanted to have interactive maps on an air-gapped computer, so I built a system that would allow this (in this case, the JS/CSS file download step was run as part of building the ‘app’ we produced). I should note at this point that without an internet connection the default OpenStreetMap background mapping will not work. In some situations that would be a significant problem – but we were using other background mapping, coastline vector files and so on, so it wasn’t a problem for us.

So, how do you use this package?

First, install it using pip install offline_folium. Then make sure you have an internet connection and run python -m offline_folium – this will download the JS/CSS files and store them in a sensible place (chosen automatically). Now you’re all set up.

Once you have no internet connection and want to use folium, you must import it in a slightly different way – first import offline from offline_folium, and then import folium, like this:

from offline_folium import offline import folium

Now you can use folium as usual, for example by creating a simple map object:

m = folium.Map()

So, that’s pretty-much it. People are actively submitting pull requests at the moment, so hopefully the functionality will expand to work with various folium plugins too.

For more information (or to submit a PR yourself), see the Github repo.

This year’s Board Election Nomination period opens tomorrow and closes on June 25th. Who runs for the board? People who care about the Python community, who want to see it flourish and grow, and also have a few hours a month to attend regular meetings, serve on committees, participate in conversations, and promote the Python community. Check out our Life as Python Software Foundation Director video to learn more about what being a part of the PSF Board entails. We also invite you to review our Annual Impact Report for 2023 to learn more about the PSF mission and what we do.

Current Board members want to share what being on the Board is like and are making themselves available to answer all your questions about responsibilities, activities, and time commitments via online chat. Please join us on the PSF Discord for the Board Election Office Hours (June 11th, 4-5 PM UTC and June 18th, 12-1 PM UTC)  to talk with us about running for the PSF Board. 

Board Election Timeline

Nominations open: Tuesday, June 11th, 2:00 pm UTC
Nominations close: Tuesday, June 25th, 2:00 pm UTC
Voter application cut-off date: Tuesday, June 25th, 2:00 pm UTC
Announce candidates: Thursday, June 27th
Voting start date: Tuesday, July 2nd, 2:00 pm UTC
Voting end date: Tuesday, July 16th, 2:00 pm UTC 

Not sure what UTC is for you locally? Check here!

Nomination details

You can nominate yourself or someone else. We encourage you to reach out to people before you nominate them to ensure they are enthusiastic about the potential of joining the Board. Nominations open on Tuesday, June 11th, 2:00 PM UTC, and end on June 25th, 2:00 PM UTC.

Please send nominations and questions regarding nominations to psf-board-nominations@pyfound.org. Include the name, email address, and an endorsement of the nominee's candidacy for the PSF Board. To get an idea of what a nomination looks like, check out the Nominees for 2023 PSF Board page. After the nomination period closes, we will request a statement and other relevant information from the nominees to publish for voter review.

Voting Reminder!

Every PSF Voting Member (Supporting, Managing, Contributing, and Fellow) needs to affirm their membership to vote in this year’s election. You should have received an email from "psf@psfmember.org <Python Software Foundation>" with subject "[Action Required] Affirm your PSF Membership voting status" that contains information on how to affirm your voting status.

You can see your membership record and status on your PSF Member User Information page. If you are a voting-eligible member and do not already have a login, please create an account on psfmembership.org first and then email psf-donations@python.org so we can link your membership to your account.

Ashish Kunwar has found that python-sql accepts any string in the offset or limit parameters when python is ran with -O which makes any system exposing those vulnerable to an SQL injection attack.

Impact

CVSS v3.0 Base Score: 9.1

• Attack Vector: Network
• Attack Complexity: Low
• Privileges Required: Low
• User Interaction: None
• Scope: Changed
• Confidentiality: High
• Integrity: Low
• Availability: Low

Workaround

Do not use the -O switch or PYTHONOPTIMIZE environment variable when executing python.

Resolution

All affected users should upgrade python-sql to the latest version.

Affected versions: <= 1.5.0
Non affected versions: >= 1.5.1

Reference

• SQL Injection vulnerability via offset parameter (#92) · Issues · Tryton / python-sql · GitLab

Concerns?

Any security concerns should be reported on the bug-tracker at https://bugs.tryton.org/python-sql with the confidential checkbox checked.

1 post - 1 participant

Read full topic

HL7 FHIR Integrations in Python 2024-06-10, by Dariusz Suchojad

HL7 FHIR, pronounced "fire", is a data model and message transfer protocol designed to facilitate the exchange of information among systems used in health care settings.

In such environments, a FHIR server will assume the role of a central repository of health records with other systems integrating with it, potentially in a hub-and-spoke fashion, thus letting the FHIR server become a unified and consistent source of data that would otherwise stay confined to a silo of each individual health information system.

While FHIR is the way forward, the current reality of health care systems is that much of the useful and actionable information is distributed and scattered among many individual data sources - paper-based directories, applications or data bases belonging to the same or different enterprises - and that directly hampers the progress towards delivering good health care. Anyone witnessing health providers copy-and-pasting the same information from one application to another, not having access to the already existing data, not to mention people not having an easy way to access their own data about themselves either, can understand what the lack of interoperability looks like externally.

The challenges that integrators face are two-fold. On the one hand, the already existing systems, including software as well as medical appliances, were often not, or are still not being, designed for the contemporary inter-connected world. On the other hand, FHIR in itself is a relatively new technology which means that it is not straightforward to re-use the existing skills and competencies.

Zato is an open-source platform that makes it possible to integrate systems with FHIR using Python. Specifically, its support for FHIR enables quick on-boarding of integrators who may be new to health care interoperability, who are coming to FHIR with previous experience or interest in web development technologies, and who need an easy way to get started with and to navigate the complex landscape of health care integrations.

Connecting to FHIR servers

Outgoing FHIR connections are what allows Python-based services to communicate with FHIR servers. Throughout the rest of the chapter, the following definition will be used. It connects to a live, publicly available FHIR server.

Filling out the form below will suffice, there is no need for any server restarts. This principle, that restarts are not needed, applies all throughout the platform, whenever you change any piece of configuration, it will be automatically propagated as necessary.

• Name: FHIR.Sample
• Address: https://simplifier.net/zato
• Security: No security definition (we will talk about security later)
• TLS CA Certs: Default bundle

Retrieving data from FHIR servers

In Python code, you obtain client connections to FHIR servers through self.out.hl7.fhir objects, as in the example below which first refers to the server by its name and then looks up all the patients in the server.

The structure of the Patient resource that we expect to receive can be found here.

# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class FHIService1(Service): name = 'demo.fhir.1' def handle(self) -> 'None': # Connection to use conn_name = 'FHIR.Sample' with self.out.hl7.fhir[conn_name].conn.client() as client: # This is how we can refer to patients patients = client.resources('Patient') # Get all active patients, sorted by their birth date result = patients.sort('active', '-birthdate') # Log the result that we received for elem in result: self.logger.info('Received -> %s', elem['name'])

Invoking the service will store in logs the data expected:

INFO - Received -> [{'use': 'official', 'family': 'Chalmers', 'given': ['Peter', 'James']}]

For comparison, this is what the FHIR server displays in its frontend. - the information is the same.

Storing data in FHIR servers

To save information in a FHIR server, create the required resources and call .save to permanently store the data in the server. Resources can be saved either individually (as in the example below) or as a bundle.

# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class CommandsService(Service): name = 'demo.fhir.2' def handle(self) -> 'None': # Connection to use conn_name = 'FHIR.Sample' with self.out.hl7.fhir[conn_name].conn.client() as client: # First, create a new patient patient = client.resource('Patient') # Save the patient in the FHIR server patient.save() # Create a new appointment object appointment = client.resource('Appointment') # Who will attend it participant = { 'actor': patient, 'status':'accepted' } # Fill out the information about the appointment appointment.status = 'booked' appointment.participant = [participant] appointment.start = '2022-11-11T11:11:11.111+00:00' appointment.end = '2022-12-22T22:22:22.222+00:00' # Save the appointment in the FHIR server appointment.save() Learning what FHIR resources to use

The "R" in FHIR stands for "Resources" and the sample code above uses resources such a Patient or Appointment but how does one learn what other resources exist and what they look like? In other words, how does one learn the underlying data model?

First, you need to get familiar with the spec itself which, in addition to textual information, offers visualizations of the data model. For instance, here is the description of the Observation object, including details such as all the attributes an Observation is composed of as well as their multiplicities.

Secondly, do spend time with FHIR servers such as Simplifier. Use Zato services to create test resources, look them up and compare the results with what the spec says. There is no substitute for experimentation when learning a new data model.

FHIR security

Outgoing FHIR connections can be secured in several ways, depending on what a given FHIR requires:

• With Basic Auth definitions
• With OAuth definitions
• With SSL/TLS. If the server is not a public one (e.g. it is in a private network with a private IP address), you may need to upload the server's certificate to Zato first if you plan to use SSL/TLS because, without it, the server's certificate may be rejected.

MLLP, HL7 v2 and v3

While FHIR is what new deployments use, it is worth to add that there are still other HL7 versions frequently seen in integrations:

• Version 2, using its own MLLP protocol
• Version 3, using XML

Both of them can be used in Zato services, in both directions. For instance, it is possible to both receive HL7 v2 messages as well as to send them to external applications. It is also possible to send v2 messages using REST in addition to MLLP.

• Read more about using Python in API integrations
• Start the tutorial which will guide you how to design and build Python API services for interoperability, integrations and automation

More blog posts➤

The Drop Times conducted an in-depth analysis of Drupal users across the world's largest cities, focusing on locations with populations over one million. Using LinkedIn data, this study identifies key urban centers with significant Drupal communities, highlighting 81 cities with more than 500 "Drupal People." This analysis aims to support local community growth and enhance global Drupal engagement. Discover the cities where Drupal is thriving and learn how to get involved in organizing and promoting local events.

The Drop Times aims to enhance and track the global growth of the Drupal community. By utilizing LinkedIn's public search capabilities, we can estimate the geographical spread of Drupal users. Despite certain limitations, this method provides a practical approach to gauge community growth and plan targeted events. Learn how we navigate data challenges and leverage LinkedIn for meaningful insights.

Hi, it's week 2 and my last scheduled week for research. I spent this week looking over more of Krita's code base and pixel-perfect algorithm. There was a large focus on looking at examples of how pixel-perfect is achieved (specifically in Aseprite)....

Today I’d like to share a new set of Human Interface Guidelines (HIG) for KDE’s software that I’ve written, replacing the old one. This work was done over the past several months in consultation with many KDE designers and developers; the merge request says 42 people were CCd, and almost 500 comments were posted during the 2+ month review process.

You can read the document at https://develop.kde.org/hig.

Wait, why does anyone need this at all?

Strictly speaking, we don’t need a HIG. Developers with an excellent eye for design can usually produce good results even without one. But for most of us, it’s useful to have guidelines to follow so you don’t have to think about the design side too much. Even for design-oriented developers, it’s useful to be able to have a quick reference for common patterns and rules.

And having a HIG is a good idea for any organization that wants for its software to share a similar look-and-feel and mode of operation, or any small individual developer who wants their software to match fit in well with a specific target platform (ours, in this case). When software fits into the visual and functional conventions of the platform with which it wants to integrate most closely, people already familiar with that platform learn to use it faster and like using it more. It feels at home to them. Comfortable, familiar, appealing.

Having and following a HIG makes all of this possible. And besides, we already had one anyway! But it needed major work.

Ok, so what was wrong with the old one?

To be honest, it wasn’t very good. I say this as a contributor to the old one! But it suffered from multiple problems that demanded a total rewrite:

• It was hard to find anything because the basic structure was confused, with excessive segmentation.
• Content was severely outdated and reflected design paradigms that KDE developers haven’t used in years, and that the industry in general has moved away from. It was also missing a lot of information relevant to how we write apps today.
• Far too wordy; hard to find the actionable recommendations amid all the filler text!
• Full of old mockups and screenshots that weren’t even needed to illustrate the point, and which would become out of date again quickly if we took the time to update them.
• Many basic style errors: misspellings, incorrect English grammar, awkwardly phrased sentences, etc.

For these reasons, very few KDE developers or designers still paid any attention to the old HIG, besides of a small number of pages that did still have a reasonably high threshold of information density. And for the same reason, the HIG got very few contributions. I think when something is in a sufficiently poor state, it can even discourage people from contributing, because any small improvement seems like a raindrop in the ocean of urgent need. It’s demoralizing.

How is this new one any better?

The new one was informed by research into how KDE apps look and work today. Some inspiration for structure and organization was taken from the ElementaryOS, GNOME, Apple, and Google HIGs — but not content! The content is all KDE.

The new HIG had the following design goals:

• Make 100% of the content actionable. No filler text, no rambling philosophy, no redundancy — just actionable recommendations for how to design your app. Short and sweet, and to the point!
• Reflect how KDE designs software today. For example we’ve seen a huge growth in new Kirigami-based apps using QtQuick as their UI technology (including some older KDE apps getting their UIs ported to Kirigami), and Kirigami is the UI platform that’s improving most quickly in KDE. So I made the decision to focus the recommendations on Kirigami as a platform. There’s still some content about QtWidgets, but it’s a Kirigami-focused document.
• Have flat navigation, so it’s easy to find everything. No deeply nested sub-pages and awkward categorization that makes you wonder why a page is in this category and not that one.
• Get the new content into good enough shape that people feel comfortable contributing. Hopefully people will start to submit tweaks, improvements, bug-fixes, and maintenance!

Is it 100% finished?

No, absolutely not! The HIG is intended to be a living document that evolves over time to better describe KDE’s design goals and software development paradigms.

Like any rewrite, there are bound to be rough edges and omissions compared to the old version. Maybe I missed a piece of useful information in the old HIG that had been buried somewhere but retained some value. Maybe there’s low-hanging fruit for improvement. Help out by contributing! It’s just some text with markdown styling; contributing to the HIG is waaaaaaay easier then contributing code.

In particular, there are some known omissions and limitations that you can help with:

• It needs more images. Multiple pages have embedded TODO comments where it would be nice to add a relevant image to illustrate a point.
• There’s nothing at all about visual style in general, simply delegating those decisions to the system’s active theme. To a certain extent this is intentional because we support visual theming, but it would also be good to add content about KDE’s default Breeze theme. I deliberately omitted this for now because a bunch of KDE’s designers are working on a fancy new style, and developers are working on a whole new theming engine to apply it. So the world around the HIG is in a state of flux. But if anyone wanted to add more about the current Breeze style anyway, that would be nice. Just know that it may be replaced once the new style is released.
• …With one exception: the entire section on Breeze icon design. This was kept from the old HIG content because it remains useful. Still, the presentation could use an overhaul to improve information density and collapse needless sub-pages into one parent page.
• The recommendation to use Kirigami is awkward for powerful apps that use features not currently available in Kirigami-based apps, such as dockable sidebars/panes, customizable toolbars, customizable keyboard shortcuts, and more. If you’re a developer, help add these features!

Ok cool, how do I contribute?

That’s great, it’s super duper appreciated! See these two links to learn how to contribute changes:

• https://develop.kde.org/hig/#contributing
• https://invent.kde.org/documentation/develop-kde-org/-/blob/master/README.md

If that’s too scary, we can help you get set up! Contact folks and we’ll see what we can do.

Still too scary? Then you can donate to KDE. Our budget is tiny, so your money genuinely does have an impact!

Today marks the release of  KDE Stopmotion 0.8.7!

About Stopmotion

Stopmotion is a Free Open Source application to create stop-motion animations. It helps you capture and edit the frames of your animation and export them as a single file.

Direct capture from webcams, MiniDV cameras, and DSLR cameras. It offers onion-skinning, import images from disk, and time lapse photography. Stopmotion supports multiple scenes, frame editing, basic sound track, animation playback at different frame rates, and GIMP integration for image. Movies can be exported to a file and to Cinelerra frame lists.

Technically, it is a C++ / Qt application with optional dependencies to camera capture libraries.

Changes in release 0.8.7

This release comes with no new features, but improvements to the project itself.

Changes

• The project is now officially called to KDE Stopmotion. The former name Linux Stopmotion is no longer used.
• Support for qmake has been removed. Use CMake instead.

Features

• Port serialization to libarchive. libtar is abandoned. (thanks to Bastian Germann)

Bugfixes

• The .sto files miss the tar trailer. (#16, thanks to Bastian Germann for providing a fix)

Improvements

• Use pkg-config to find dependencies vorbisfile and xml2 (thanks to Barak Pearlmutter)
• Remove code that relies on deprecations in Qt 5; this is a preparation to move to Qt 6.
  

Future plans

• Transition from Qt 5 to version 6. I am stuck with my port as QAudioDeviceInfo that was dropped in Qt 6. I need some help to port Stopmotion to the new way to handle audio with Qt 6 / Qt Mulimedia.
• We should integrate better to KDE's tech stack: Internationalization, using KDE libraries, update and reformat documentation.
  

Get involved!

If you are interested, give Stopmotion a try. Reach out to our mailing list stopmotion@kde.org or have a look into our project. Share your ideas or get involved!

Drupal's migration system allows the use of a number of different plugins to perform source, processing, and destination functions. 

Process plugins are responsible for copying and sometimes manipulating data into the destination. There are a number of different process plugins that allow you to get data in different ways and and apply it to your destination fields.

Both the core Migrate module and the excellent Migrate Plus module contain a number of different process plugins that you can use to process your data in different ways.

Out of the box, the default process plugin is the get plugin, which can be used like this in your migration scripts.

destination_field: plugin: get source: source_field

This is often shortened to the following, which has exactly the same functionality.

destination_field: source_field

Most of the time you will want to avoid creating custom plugins, but sometimes your migration requirements will necessitate their use. You might find that your source data is very messy and needs to be cleaned up before importing it into the site. Process plugins are a really good way of doing this, but it is essential that you write tests to cover every situation that you might encounter. 

In this article we will look at two custom migrate process plugins that are built in different ways and how to test them. This will dive into some concepts around Drupal plugin management, dependency injection, as well as unit testing and data providers with PHPUnit.

First, let's look at the migration script that we will be using in this article. All of the source code for this migration example is available on GitHub.

Read more

I’m a heavy user of Firefox profiles. Apart from using different profiles for different activities, I also have a few extra profiles that all run in the Default activity.

This means that I need to have different icons shown in Plasma’s panel in order to be able to easily differentiate which profile a window belongs to.

Sure, I use the tasks applet which shows the window title instead of the icon-only one (I prefer usability to minimalism), but still, it isn’t enough as sometimes the active tab in a Firefox window might not have the most informative title.

Plasma seems to rely on the application name and the window class when choosing the icon it will show in the panel. Which means that, by default, all Firefox instances end up having the same icon.

Librewolf with a custom profile icon

Fortunately, Firefox allows you to specify the window class it should use through command line arguments.

firefox -P ProfileName --class WindowClassName

And, to connect a launcher to a specific window class, you just need to add the following line to the .desktop file:

StartupWMClass=WindowClassName

So, in order to have a nicely supported Firefox profile, you can create a launcher with a desktop file similar to the following:

[Desktop entry] Exec=firefox -P SocialSites --class FirefoxSocialSites Icon=user-available-symbolic StartupWMClass=FirefoxSocialSites

It also works with Firefox derivatives such as Librewolf (which can be seen in the screenshot above) and others.

You can support my work on Patreon, or you can get my book Functional Programming in C++ at Manning if you're into that sort of thing. -->

Why write an update?
I wrote a blog post on Software Development with Generative AI last year, which was questioning the approach of the current AI software authoring assistants. I believe the bigger picture holds true that to fully utilize AI to write software, will require an entirely different approach. Changing the job of a software developer in a far more radical manner and perhaps making many of today's software languages redundant.

However I also raised the issue that I found the current generative AI helpers utility questionable for seasoned developers:
"The generative AI can help students and others who are learning to code in a computer language, but can it actually improve productivity for real, full time, developers who are fluent in that language?
I think that question is currently debatable... (but it is improving rapidly) ... We may reach that point within a year or two"

Well it hasn't been a year or two, just 6 months. But I believe the addition of the Chat window to CoPilot and an improvement in the accuracy of its models has already made a significant difference. 

On balance I would now say that even a fluent programmer may get some benefits from its use. Given the speed of improvement it is likely that all commercial programming will use an AI assistant within a few years. 

To delay the inevitable and not embed it in to your work process is like King Canute commanding the sea to retreat. There are increasing numbers of alternatives available too. However as the market leader I believe it is worth going in to slightly more depth as to the current state of play with CoPilot.

Copilot Features

The new Chat window within your IDE gives you a context sensitive version of Copilot ChatGPT that can act as a pair programmer and code reviewer for your work. 

If you have enabled auto-complete then you instigate that usage by writing functional comments, ie prompts then tabbing out to accept the suggestions it responds with.

To override these prompts, you instead can use dot and get real code completion options (as long as your IDE is configured correctly). Since code completion has your whole codebase as context, it complements CoPilot reasonably well. But whilst the code completion is always correct, CoPilot is less so, probably more like 75% now compared to its initial release level of 50%

It takes some time to improve the quality of your prompting. An effort must be made to eradicate any nuance, assumption, implication or subtlety from your English. Precise mechanical instructions are what are required. However its language model will have learnt common usage. So if you ask it to sort out your variables it will understand that you mean replace all hardcoded values in the body of your code with a set of constants defined at the top, explain that is what it thinks you mean and give you the code that does that.

You can ask it anything about the usage of the language you are working in, how something should be coded, alternatives to that etc. So taking a pair programming approach and explaining what you are about to code and why to CoPilot chat as you go,  can be very useful. Given rubber duck programming is useful, having an intelligent duck that can answer back ... is clearly more so. 

It excels as a learning tool, largely replacing Googling and Stack Overflow with an IDE embedded search for learning new languages. But even for a language you know well, there can be details and nuances of usage you have overlooked or changes in syntactic standards with new releases you have missed.

You can also ask it to give your file a code review. Where it will list out a series of suggested refactors that it judges would improve it.

Copilot Limitations

Currently however there are many limitations, understanding them, helps you know how to use CoPilot and not turn it off in frustration at its failings! 

The most important one is that CoPilot's context is extremely limited. There is no RAG enhancement yet, no learning from your usage. It may seem to improve with usage, but that is just you getting better at using it. It does not learn about you and your coding style as you might expect, given a dumb shopping site does that as standard.

It does not create a user context for you and populate it with your codebase. It simply grabs the content of the currently edited file and the Chat prompt text and the language version for the session as a big query. The same for the auto-suggestion. But here the chat text is from the comments or doc strings on the lines preceding. 

Posting the lot to a fixed CoPilot LLM that is some months out of date. Although apparently it has weekly updates from continuous retraining. 

This total lack of context can mean the only way you can get CoPilot to suggest what you actually want is to write very detailed prompts. It is often simpler to just cut and paste example code as comments into the file - please rewrite blah like this ... paste example. Since only if its in the file or latest Chat question will it get posted to inform the response.

At the time of writing CoPilot is due to at least retain and learn from Chat window history to extend its context a little. But currently it only knows about the currently open file and latest Chat message. Other providers have tools that do load the whole code base, for example Cody, plus there are open source tools to post more of your code base to ChatGPT or to an open source LLM.

As this blog post update indicates, the whole area is evolving at an extremely rapid pace.

The model it has for a language is fixed and dated. Less so for the core language but for example you may use a newer version of the leading 3rd party Postgres library that came out 2 years ago. But the majority of users are still on the previous one since it is still maintained. Their syntax differs. Copilot may only know the syntax for the old library because that is what it was trained with, even though a later version is being imported in the file, so is in Copilot's limited context. So any chat window or code prompts it suggests will be wrong.

I have yet to find it brings up anything useful that I didn't know about the code when using the code review feature, plus the suggestions can include things that are inapplicable or already applied. But I am sure it would be more useful for learning a new language.

AI prompting and commenting issue

Good practise for software teams around code commenting are that you should NOT stick in functional comments that just explain what the next few lines do.  The team are developers and they can read the code as quickly for its base functionality. Adding lots of functional commenting makes things unclear by excessive verbosity.
It is something that is only done for teaching people how to code in example snippets. It has no place in production code.

Comments should be added to give wider context, caveats, assumptions etc. So commenting is all about explaining the Why, not the How.

Doc strings at the head of methods and packages can contain a summary of what the function does in terms of the codebase. So more functional in orientation, but as a big scale summary. So again they are a What not a How.

It looks like current AI assistants may mess that up. Since they need comments that are basically as close to pseudo code as possible. Adding information about real world issues, roadmap, wider codebase, integration with other services ... ie all the Why is likely to  confuse them and degrade the auto-complete.

Unfortunately code comments are not AI prompts for generating code and vice versa.
Which suggests that you may want to write a temporary prompt as a comment to generate the code, then replace it with a proper comment once it has served its purpose.

Or otherwise introduce a separate form of hideable prompt marked comment that make it clear what is for the AI and what is for the Human!

Alternatively use the chat window for code generation then paste it in.

Copilot Translation

Translation is an area where Copilot can be very beneficial. As a non-native English speaker you can interact with it in your own language for prompting and comments and it will handle that and translate any comments in the file to English if asked to.

Code translation is more problematic, since the whole structure of a program and common libraries can be different. But if the code is doing some very encapsulated common process. For example just maths operations, or file operations. It can extract the comments and prompts and regenerate the code into another language for you.

One can imagine that one day the only language anyone will need will be a very high level, succinct English-like language, eg. Python.
When you want to write in a verbose or low-level language. You just write the simpler prompts in a spoken language, but use Python when it is faster to communicate explicitly than spoken. Since spoken languages are so unsuited to creating machine instructions.
Press a button and Copilot turns the lot into verbose C or Java code with English comments.

No dia 16 agosto é comemorado o aniversário do Projeto Debian, e todos os anos comunidades ao redor do mundo organizam encontros para celebrar esta data.

Chamado de Debian Day (Dia do Debian), o evento sempre conta com uma quantidade expressiva de comunidadades brasileiras organizando atividades nas suas cidades no dia 16 (ou no sábado mais próximo).

Em 2024 o Debian Day celebrará os 31 anos do Projeto Debian e o dia 16 de agosto será numa sexta-feira, por isso provavelmente a maioria das comunidades organizarão suas atividades no sábado, dia 17.

Estamos fazendo uma chamada de organizadores(as) para o Debian Day em 2024. A ideia é reunir, em um grupo no telegram, as pessoas interessadas em coordenar as atividades das suas comunidades locais para trocar experiências, ajudar os(as) novatos(as), e discutir a possibilidade do Projeto Debian ajudar financeiramente as comunidades.

O Debian Day na sua cidade pode ser desde um encontro em uma pizzaria/bar/restaurante para promover a reunião das pessoas, até um evento mais amplo com palestras/oficinas. Então não existe obrigatoriedade sobre como deve ser o encontro, tudo depende do que você e a sua comunidade querem e podem fazer.

Existe a possibilidade de solicitarmos ao líder do projeto Debian para reembolsar algumas despesas. Por exemplo, para produzir adesivos, pagar as pizzas, encomendar um bolo, etc.

Venha fazer parte do grupo Debian Day BR no telegram e discutir as ideias: https://t.me/debian_day_br

Se você topa esse desafio e vai organizar um Debina Day na sua cidade, não deixe de adicionar a sua cidade com as informações necessárias aqui.

No dia 16 agosto é comemorado o aniversário do Projeto Debian, e todos os anos comunidades ao redor do mundo organizam encontros para celebrar esta data.

Chamado de Debian Day (Dia do Debian), o evento sempre conta com uma quantidade expressiva de comunidadades brasileiras organizando atividades nas suas cidades no dia 16 (ou no sábado mais próximo).

Em 2024 o Debian Day celebrará os 31 anos do Projeto Debian e o dia 16 de agosto será numa sexta-feira, por isso provavelmente a maioria das comunidades organizarão suas atividades no sábado, dia 17.

Estamos fazendo uma chamada de organizadores(as) para o Debian Day em 2024. A ideia é reunir, em um grupo no telegram, as pessoas interessadas em coordenar as atividades das suas comunidades locais para trocar experiências, ajudar os(as) novatos(as), e discutir a possibilidade do Projeto Debian ajudar financeiramente as comunidades.

O Debian Day na sua cidade pode ser desde um encontro em uma pizzaria/bar/restaurante para promover a reunião das pessoas, até um evento mais amplo com palestras/oficinas. Então não existe obrigatoriedade sobre como deve ser o encontro, tudo depende do que você e a sua comunidade querem e podem fazer.

Existe a possibilidade de solicitarmos ao líder do projeto Debian para reembolsar algumas despesas. Por exemplo, para produzir adesivos, pagar as pizzas, encomendar um bolo, etc.

Venha fazer parte do grupo Debian Day BR no telegram e discutir as ideias: https://t.me/debian_day_br

Se você topa esse desafio e vai organizar um Debina Day na sua cidade, não deixe de adicionar a sua cidade com as informações necessárias aqui.

The Current State of AI Software GenerationThe user tries to describe what they want generated in terms of a snippet of high level programming language code using standard English. They submit it to the AI tool. So what are they asking the AI to generate and how does it do it?

The high level language

High level programming languages are human languages composed of english and maths symbols designed for the comprehension and composition of precise computer instructions. The language makes no more sense than English to a computer. It has to be compiled or interpreted to computer language for it to run. So it may compile to an intermediate bytecode language and then maybe to human readable assembly language - before final translation into the unreadable machine code that the computer runs.

A programmer learns the high level language and becomes fluent in it. They can read and understand the functionality of that code. With the complexity of the machine specific implementation stripped away.

Leaving just the precise functional maths and english / symbology that describes the computer functionality. They think in that code, in order to write it.
Even then, the majority of a programmers time is spent debugging the high level language - and fixing what they have written to be bug free. Because it is difficult to think clearly in code, pre-determining  all edge cases etc.

Unlike English language, it can succinctly describe computer functionality in a few lines. 

The AI

A detailed English language description of what functionality is required. Plus the name of a high level programming language, are submitted to the AI tool.

It does a search of the web, eg. stack overflow etc. for results for that code language. For Chatbot use (eg. ChatGPT) it applies an English language Large Language Model, LLM (a numeric encoding of learning of the English language) to generate a well phrased aggregation of the most popular results that match the English prompt. 

For software use (eg. CoPilot) it works just the same, but the LLM learns English to high level software language aggregate translation. From code examples data, eg. github, to generate what the code syntax might be to match the English description of it.

Finally it returns an untested snippet of generated high level code.

The Non-Developer

The non-developer pastes it in place and tries to run the program with it in.

They may be able to puzzle out the high level language - but don't naturally think in it, just as people without mathematics skills can only think as far as basic arithmetic and are dyslexic when it comes to complex equations.

It seems to work around 50% of the time. When it fails they, go back to square one and try to rephrase their English prompt. 

They patch together block after block of prompt created generated code. A crazy paving of a program that likely has a number of bugs and inappropriate features in it. But it kind of works, for the non-developer, that is good enough.

The code gets pushed out there with all its imperfections, and starts to populate the web of code data that is used to generate the next AI code snippet.

Or the Developer
The developer reads the code and understands it, determines if it should do what they want. Or if they just want to use some of it as an example.

They cut paste and rewrite it, using it as a hint tool. Or an extension to their IDE's existing auto-code generation tools that work using templated code and language / import library searches.

Hopefully their IDE is set up to clearer distinguish between real code completions and possible generative code completions. Since otherwise the percentage of nonsense code created by the generative AI pollutes the 100% reliability of IDE code completion, and harms productivity.

Then they run their code and debug as usual.

At least 75% of programming time is not on writing code, but on making sure that the high level instructions are exactly correct for generating bug free machine code. So iteratively refining the lines of code. With code a single comma out of place can break the whole program. When language has to be so carefully groomed, succinct minimal language is essential.

For many developers adding an imprecise, non mathematical language, that is entirely unsuited to defining machine code instructions, such as English, to generate such code is problematic. It introduces a whole layer of imprecision, complexity and bugs to the process. Slowing it right down. Along with requiring developers to write a lot lot more sentences (in English) rather than just quickly typing out the succinct lines of Python (or similar) programming language they have in their head.

The generative AI can help students and others who are learning to code in a computer language, but can it actually improve productivity for real, full time, developers who are fluent in that language?

I think that question is currently debatable. Because I believe the goal of adding yet another language to the stack of languages that need to be interpreted for humans authoring computer code, especially one as unsuited as English, is only useful for people who are far from fluent in the software language.

Once we move beyond error prone early releases of LLMs like ChatGPT-4 then tools such as CoPilot may start to become much more effective at authoring software, and actually produce code that is as likely to work first time and have the same amount of bugs as your average software developer's first cut of the code. We may reach that point within a year or two. At which point professional software developer will need to be adept at using it as part of their toolset.

Even so I believe the whole conception of the application of AI to writing software could benefit from more work engaged in a computer centric alternative approach to the current one focussed on generating plausible human language responses. It only dominates because of all the efforts related to NLP and human interaction. But taking that and sticking on to writing human software languages is more about creating a revenue stream than attempting to have AI do the main work of software development.

Until then, AI will never be able to replace me, as a software developer. Only be another IDE tool I need to learn ... in time when it improves sufficiently to increase productivity.

NOTE - June 2024 Update
Having come back to CoPilot 6 months later. I have come to appreciate some of its new features so have added a new blog post that accepts that it now provides utility even for the seasoned programmer.

Another WayCopilot and the like currently use the ChatGPT approach of a Chatbot front end tied to an English language LLM to generate aggregate search engine results in a human language. But there is no domain specific machine learning knowledge about the semantics of the content. So it doesn't understand, and certainly doesn't pre-check the code. Just as ChatGPT doesn't understand the search engine content. Since currently there are no domain specific trained models for the content in the loop. So if asked a question about pharmacy it doesn't plug in one of the AI models that has learnt pharmacy and is used by that industry to aid in the development of medicines. It understands nothing, it is a chatbot, just a constructor of plausible answers based on search popularity.
Similarly CoPilot has learnt how to predict what code somebody might be trying to write, but it hasn't learnt how to code.

This approach cannot lead to AI generating innovative new coding approaches, full self-coding computers, or remove the need for human readable high level programming languages.

There have been experiments with applying test driven development to AI generated code, but I have not heard of serious attempts to address the bigger picture...

• Move all functional code writing to be AI only.
• Remove the need for any high level computer language for humans to gain fluency in.
• Have AI develop software by hundreds of thousands of iterative composition  TDD cycles.
• Parallel refactoring thousands of solutions to arrive at the optimum one.
• Use AI that understands the machine code it is generating by training it on the results of running that code. 
• The ML training cycle must be running code not matching outputs to pre-ranked static result training sets.
• In addition to the static LLM that encodes the learning of machine code authoring, dynamic training cycles should be run as part of the code composition. Task based ephemeral training models.
• Get rid of the wasted effort training AI to understand English, Python, Java, Go or any other existing human language evolved for other tasks.
• Finally we are left with the job of telling the computer what its software should do.
  We do not want to use English for that, its way too verbose and inaccurate, similarly we don't want a full high level programming language to do it. We need a new half way house. A domain specific language (DSL) for defining functionality only, designed for giving software specification's to AI that it can use to generate automated test suites.

Self-Programming Computers

Exploring the last point in more detail...

Create a higher level pseudo-code language for describing the required functionality that is more English readable than even current high level languages such as Python.

Make that functional DSL focus on defining inputs and outputs - not creating the functionality, but creating the black box functional tests that describe what the working code should do.

Maybe add tools for a slightly no-code approach, with visual generators for the language, eg graphical pipeline builder tools. For people who find thinking visually easier than thinking symbolically.

The software creator uses the DSL to create an extensive set of functional definitions for a project.

The DSL language design and evolution is optimised for LLM interpretation.  So it has very tight grammatical and syntactical usage that promote accurate generative outputs.

A new non-developer friendly high level pseudo code language / rigorous AI prompt writing lingo.

Some basic characteristics of the DSL:

1. auto-formatting (like Go) minimizing syntactical variation
2. To quote Python's creator - 'There should be one-- and preferably only one --obvious way to do it.'
   But strictly applied, rather than as a vague principle as Python does
3. unlike any other high level language, the design needs to be optimized only for specifying functionality, a high level templating language from which test suites are generated.
   
4. the language will never be used to implement functionality
5. uses simple english vocabulary and ideally minimal mathematical symbology

These DSL prompts are written with a LLM for the DSL it helps create its own prompts and the code creator uses it to refine all the DSL definitions that specify the full functionality. 

The specification DSL auto generates all the required tests in a low level language.

Since the system should also have a generative AI LLM trained for C or assembly language.
This is what creates the actual functional code by iteratively running and rewriting it against the specification encoded into the tests.

The AI tool then generates the tests for that implementation and uses TDD to generate the actual functional code - eventually the system should improve to a level better than most software developers. The code it writes no longer needs to be read by a human - because a human will be unable to debug it at anything like the speed the AI tool can.

So we use generative AI to do the part of the job that actually takes all the time. Debugging, refactoring and maintaining the code, making sure it really does what is required functionally. Rather than the quick job of writing a first cut of it that might run without crashing.

Most importantly we don't introduce the use of the full English language, the language of Shakespeare, the language of puns, double meanings, multiple interpretations, shades of grey, implied feeling and emotions, into a binary world to which it is entirely unsuited.

Also we don't need English or high level computer languages in the stack of mistranslation at all.
Because we are not training the AI to understand human languages. We are training it to write its own machine code language based on defining what behaviour it should implement.
BDD / TDD generative AI if you like.

Human's no longer learn complex mathematical process based languages that can be translated into machine code. They learn a more generic language for specifying functional behaviour.

This gives more freedom to widen the DSL to mature into a general precise AI prompt language.

Whilst allowing computers to evolve more machine learning driven software architectures that are self maintaining and not so constrained into the models imposed by current human intelligence and coding practise based programming languages.

Could AI could take my job?Perhaps if all of the above were in place, then finally we would arrive at a place where AI could replace traditional software development and high level software languages.
With concerted effort it could be in 10 years, if some big companies put some serious investment in trying to replace traditional software development.
Code monkeys will all be automated. Only software architects would be required and they would use a new functional specification AI prompt language, not a programming language.

Of course if politicians are scared that dumb ChatGPT can already write as good a speech as they can. Plus replicate all the prejudices and errors of its training data and trainers.
Then setting AI free to fully write software, and itself ... will be way more scary in its long term implications.
Meanwhile we are currently at a place where it arguably doesn't even improve productivity for an experienced software developer, only allows non-developers, students and other language newbies to have a go at writing one of the many dialects of human languages, known as computer languages. 

Their mix of math, english, symbols, logic and process may appear more like English than Musical notation or pure maths, but sadly they are no more suited to creation by an English language Chatbot approach.

I built a little Python script called the Floyd-Warshall CSV Generator. It takes a CSV of graph edges as input, and generates a CSV of the edges that are the shortest paths between all pairs of vertices.

The script is a simple wrapper of the SciPy floyd_warshall function, which in turn implements the Floyd-Warshall Algorithm. Hope you find it useful for all your directed (or undirected) weighted graph needs.

Given an input CSV of the following graph edges:

point_a,point_b,cost a,b,5 b,c,8 c,d,23 d,e,6

When the script is called as follows:

floyd-warshall-csv-generator &bsol /path/to/input_data.csv &bsol --vertex-i-column-name point_a &bsol --vertex-j-column-name point_b &bsol --weight-column-name cost &bsol --no-directed &bsol --max-weight 35

It generates an output CSV that looks like this:

point_a,point_b,cost a,b,5.0 a,c,13.0 b,c,8.0 b,d,31.0 c,d,23.0 c,e,29.0 d,e,6.0

That is, it generates all the possible (indirect) paths from one point to all other points, based on the (direct) paths that are already known, with duplicate (undirected) paths filtered out, and with paths whose cost is more than max-weight filtered out.

I wrote this script in order to generate the "all edges" data that's shown in the World Locality Transit Graph, which I'll also be blogging about real soon. Let me know if you put this script to any other interesting uses!

Contrarian view on cursor-based pagination.

GraphQL documentation recommends cursor-based pagination, and it has subsequently become a popular standard.

In general, we’ve found that cursor-based pagination is the most powerful of those designed. Especially if the cursors are opaque, either offset or ID-based pagination can be implemented using cursor-based pagination (by making the cursor the offset or the ID), and using cursors gives additional flexibility if the pagination model changes in the future. As a reminder that the cursors are opaque and that their format should not be relied upon, we suggest base64 encoding them. …

{ hero { name friends(first: 2) { totalCount edges { node { name } cursor } pageInfo { endCursor hasNextPage } } } }

There are several oversights with this well-intentioned advice.

Cursors and state

Cursors imply state, at least they used to. A database cursor is used for iterating over a result set. Meaning it has transactional integrity to pick up where it left off.

The vast majority of GraphQL APIs are inherently stateless. The “cursor” is being decoded as input to a new request, and offers no guarantees. From this observation, the advice falls apart.

The problem with stateless pagination is inconsistency; items may shift, appear, or disappear. Which gives the client the perception of missing or duplicate items. This happens regardless of whether the pagination is offset or ID based. Arguably worse in the case of IDs, since the reference can move arbitrarily or be gone.

Cursors don’t solve the consistency problem; they give the client the false impression of solving the problem.

Opaqueness and compatibility

The claim is that an opaque cursor is compatible across changes. Changed to do what exactly, would be the more relevant question.

Taking a step back, what is the problem being solved here? We assume there is a list of items, with an inherent ordering, and too many to return to the client with acceptable performance.

Given those assumptions, the first obvious step is an optional size limit. That is not in dispute; the disagreement if over the “offset”. A simple and versatile solution is a range filter over whatever field(s) is relevant to ordering. This is not even remotely controversial when the field in question has a name like date. In other words, “pagination” is not necessarily the problem that needs solving.

Range filters with a size limit are sufficient to implement pagination, and new optional filters are always backwards compatible. They also offer the flexibility of search, whereas cursors can only be used iteratively. And what if the client does not want visibility into the range filters? That is exactly what offset is for; offset is a range filter over an implied index field.

There is a reason why the recommendation does not offer a useful example of this supposed compatibility; there isn’t one. The advice is equivocating on the ambiguity of an after: $ID filter. Is the ID field relevant to the ordering?

• If yes, then it is just another range filter
• If no, then it is just another placeholder for index

There is no third case. There is no future secret field that relates to ordering, is relevant to the client, but somehow still opaque to the client.

Stateless pagination is a combination of range filters and size limits. No matter what the input fields are called. A true stateful is cursor is opaque precisely because it does not represent any known field.

Next optimization

The “next” piece of advice is that the cursor implementation should indicate whether another request is worthwhile. Again, in a stateless API, the server can make no such guarantee.

If the server can provide a total count, by all means do so. It solves the “next” problem, and is more generally useful.

If it is not feasible for the server to provide a total count, how is it going to implement whether there are more items? At the data layer, it is going to stop processing at N + 1 items instead of the requested N. The client could do that too. Instead of requesting the next 10, it could go to 11.

Better yet, why stop at the server optimizing for N + 0? If it knows there is just 1 more item, why not go ahead and include that last one too. N + 2 anyone? Obsessing over the last “next” is a pointless micro-optimization, all the more so because it is irrelevant whenever the total count is not coincidentally a multiple of N. If N is arbitrary, then optimizing for a particular residue mod N is clearly arbitrary.

API design

Not only is there no good reason to blindly add opaque cursors, there is also no reason to add range filters before needed. A size limit alone solves the first order of magnitude of performance issues. If a client requests the first 10 items, then needs the next 10, actually pressure test whether it is unreasonable to request the first 20. The advantage is the client then has a consistent snapshot of the first 20 regardless of changes, which could provide a better user experience.

A simple strategy for pagination: start with none. Then proceed to next steps as performance warrants.

1. size limit
2. range filter on known field(s)
3. offset

In the unlikely event your API is stateful, you didn’t need this advice because you already had a cursor. Otherwise, cursors are an overly-complicated useless abstraction.