Feeds
Real Python: How to Create Pivot Tables With pandas
A pivot table is a data analysis tool that allows you to take columns of raw data from a pandas DataFrame, summarize them, and then analyze the summary data to reveal its insights.
Pivot tables allow you to perform common aggregate statistical calculations such as sums, counts, averages, and so on. Often, the information a pivot table produces reveals trends and other observations your original raw data hides.
Pivot tables were originally implemented in early spreadsheet packages and are still a commonly used feature of the latest ones. They can also be found in modern database applications and in programming languages. In this tutorial, you’ll learn how to implement a pivot table in Python using pandas’ DataFrame.pivot_table() method.
Before you start, you should familiarize yourself with what a pandas DataFrame looks like and how you can create one. Knowing the difference between a DataFrame and a pandas Series will also prove useful.
In addition, you may want to use the data analysis tool Jupyter Notebook as you work through the examples in this tutorial. Alternatively, JupyterLab will give you an enhanced notebook experience, but feel free to use any Python environment you wish.
The other thing you’ll need for this tutorial is, of course, data. You’ll use the Sales Data Presentation - Dashboards data, which is freely available for you to use under the Apache 2.0 License. The data has been made available for you in the sales_data.csv file that you can download by clicking the link below.
Get Your Code: Click here to download the free sample code you’ll use to create a pivot table with pandas.
This table provides an explanation of the data you’ll use throughout this tutorial:
Column Name Data Type (PyArrow) Description order_number int64 Order number (unique) employee_id int64 Employee’s identifier (unique) employee_name string Employee’s full name job_title string Employee’s job title sales_region string Sales region employee works within order_date timestamp[ns] Date order was placed order_type string Type of order (Retail or Wholesale) customer_type string Type of customer (Business or Individual) customer_name string Customer’s full name customer_state string Customer’s state of residence product_category string Category of product (Bath Products, Gift Basket, Olive Oil) product_number string Product identifier (unique) product_name string Name of product quantity int64 Quantity ordered unit_price double Selling price of one product sale_price double Total sale price (unit_price × quantity)As you can see, the table stores data for a fictional set of orders. Each row contains information about a single order. You’ll become more familiar with the data as you work through the tutorial and try to solve the various challenge exercises contained within it.
Throughout this tutorial, you’ll use the pandas library to allow you to work with DataFrames and the newer PyArrow library. The PyArrow library provides pandas with its own optimized data types, which are faster and less memory-intensive than the traditional NumPy types pandas uses by default.
If you’re working at the command line, you can install both pandas and pyarrow using python -m pip install pandas pyarrow, perhaps within a virtual environment to avoid clashing with your existing environment. If you’re working within a Jupyter Notebook, you should use !python -m pip install pandas pyarrow. With the libraries in place, you can then read your data into a DataFrame:
Python >>> import pandas as pd >>> sales_data = pd.read_csv( ... "sales_data.csv", ... parse_dates=["order_date"], ... dayfirst=True, ... ).convert_dtypes(dtype_backend="pyarrow") Copied!First of all, you used import pandas to make the library available within your code. To construct the DataFrame and read it into the sales_data variable, you used pandas’ read_csv() function. The first parameter refers to the file being read, while parse_dates highlights that the order_date column’s data is intended to be read as the datetime64[ns] type. But there’s an issue that will prevent this from happening.
In your source file, the order dates are in dd/mm/yyyy format, so to tell read_csv() that the first part of each date represents a day, you also set the dayfirst parameter to True. This allows read_csv() to now read the order dates as datetime64[ns] types.
With order dates successfully read as datetime64[ns] types, the .convert_dtypes() method can then successfully convert them to a timestamp[ns][pyarrow] data type, and not the more general string[pyarrow] type it would have otherwise done. Although this may seem a bit circuitous, your efforts will allow you to analyze data by date should you need to do this.
If you want to take a look at the data, you can run sales_data.head(2). This will let you see the first two rows of your dataframe. When using .head(), it’s preferable to do so in a Jupyter Notebook because all of the columns are shown. Many Python REPLs show only the first and last few columns unless you use pd.set_option("display.max_columns", None) before you run .head().
If you want to verify that PyArrow types are being used, sales_data.dtypes will confirm it for you. As you’ll see, each data type contains [pyarrow] in its name.
Note: If you’re experienced in data analysis, you’re no doubt aware of the need for data cleansing. This is still important as you work with pivot tables, but it’s equally important to make sure your input data is also tidy.
Tidy data is organized as follows:
- Each row should contain a single record or observation.
- Each column should contain a single observable or variable.
- Each cell should contain an atomic value.
If you tidy your data in this way, as part of your data cleansing, you’ll also be able to analyze it better. For example, rather than store address details in a single address field, it’s usually better to split it down into house_number, street_name, city, and country component fields. This allows you to analyze it by individual streets, cities, or countries more easily.
In addition, you’ll also be able to use the data from individual columns more readily in calculations. For example, if you had columns room_length and room_width, they can be multiplied together to give you room area information. If both values are stored together in a single column in a format such as "10 x 5", the calculation becomes more awkward.
The data within the sales_data.csv file is already in a suitably clean and tidy format for you to use in this tutorial. However, not all raw data you acquire will be.
It’s now time to create your first pandas pivot table with Python. To do this, first you’ll learn the basics of using the DataFrame’s .pivot_table() method.
Get Your Code: Click here to download the free sample code you’ll use to create a pivot table with pandas.
Take the Quiz: Test your knowledge with our interactive “How to Create Pivot Tables With pandas” quiz. You’ll receive a score upon completion to help you track your learning progress:
Interactive Quiz
How to Create Pivot Tables With pandasThis quiz is designed to push your knowledge of pivot tables a little bit further. You won't find all the answers by reading the tutorial, so you'll need to do some investigating on your own. By finding all the answers, you're sure to learn some other interesting things along the way.
How to Create Your First Pivot Table With pandasNow that your learning journey is underway, it’s time to progress toward your first learning milestone and complete the following task:
Calculate the total sales for each type of order for each region.
Read the full article at https://realpython.com/how-to-pandas-pivot-table/ »[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Maui Report 23
Today, we bring you a new report on the Maui Project’s progress after our previous 3.1.0 release, and the last one based on Qt5 – Here you will find detailed information on the new features, bug fixes, and improvements that have been made to the set of apps, frameworks, and shell environment.
To follow the Maui Project’s development or say hi, you can join us on Telegram @mauiproject.
We are present on Twitter and Mastodon:
Maui4 AppsThe complete set of Maui Applications has been fully ported to Qt6. In the migration process, some features have been disabled or removed; and after finalizing the initial porting, the work efforts are now placed on fixing typos, and new bugs introduced, making sure all the features are working correctly. More detailed information about that will be covered and listed below.
The previous stable release – 3.1.0 – is the last one based on Qt5 and MauiKit3. Although a new stable release was scheduled by this time, instead of a new stable version we present to you a beta release of the MauiKit4 Apps and Frameworks, fully based on Qt6 and KF6. A stable release is planned to be out for August 2024.
The ported versions of all the apps can be found in the qt6 branches, and testing packages will be published as they become available.
Porting & PendingEven though all of the Maui Apps have now been ported there is still pending work to review that all the features are working correctly and there are no regressions introduced.
List of notable changes:
- Vvave lost support for streaming remote files stored in NextCloud and gained mini-mode support.
- Index application has been fully ported, Pix, Buho, Nota, Station, and all the other apps.
- Arca has been ported, and the archive manager has been moved into a framework to be shared, for example, in Index and Shelf.
- Shelf is missing Comic book support coming from MauiKit-Documents, which is being refactored.
Another area of work is on the newest set of apps, aiming to reach parity of features with the older ones.
Note! The MauiKit4 apps have not yet been tested under Android, thus there is not yet a testing APK build. APK testing packages will be published on the Maui Telegram channel once they start becoming available.
https://x.com/cmhiguita/status/1785465786930184697
MauiKit4 Frameworks Porting & DocumentationThe porting of the frameworks has been finalized. All the frameworks are now only Qt6 compatible – thus Qt5 support has been dropped, and the last stable Qt5 version will remain at 3.1.0.
The only pending framework for completing the documentation is MauiKit-Documents, and the newly introduced MauiKit-Archiver.
The following is a list of fixes and new features introduced:
- Fixes to the Documents framework for opening locked PDF documents, and initial support for searching text
- MauiKit4 fine-tuning all the controls implementation and fixing small bugs all around.
- TextEditor is pending to be ported to a more powerful text rendering/layout engine
- Documents comic book support is to be refactored to solve Android crashing issues on multithreading
- Three new frameworks are still pending for a stable release MauiKit-SDK, MauiKit-Git, and MauiKit-Archiver. Arca, the archive file manager, is now to be ported to be using MauiKit-Archiver, and Index as well.
Maui Shell and its accompanying projects have long been ported over to Qt6, such as CaskServer, Maui-Settings. However, in the porting, a lot of small details broke and need some love and fixing, which takes us to the roadmap plans of making the first stable release before this year ends. So around November, the first stable release should be out.
And that’s all for this report.
New release scheduleThe post Maui Report 23 appeared first on MauiKit — #UIFramework.
LN Webworks: How to Fix Drupal Issues with Git Patches Using 'git apply patch' Command
Have you ever faced any problems on your Drupal websites coming from Drupal core, its contributed modules and themes or do you want to enhance their functionality as per your requirement which can't be possible through your custom modules? As you know we can't apply code directly there, then what could be the solution for this? Well, patching those codes might just be the solution you're looking for.
In this blog, we'll talk about the process of creating and applying patches using git diff and git apply commands and we will also apply patches through composer install command. By applying patches you can achieve all your requirements before this you must have good coding skills to understand Drupal code.
Python Bytes: #385 RESTing on Postgres
Zato Blog: Web scraping as an API service
In systems-to-systems integrations, there comes an inevitable time when we have to employ some kind of a web scraping tool to integrate with a particular application. Despite its not being our first choice, it is good to know what to use at such a time - in this article, I provide a gentle introduction to my favorite tool of this kind, called Playwright, followed by sample Python code that integrates it with an API service.
Naturally, in the context of backend integrations, web scraping should be avoided and, generally, it should be considered the last resort. The basic issue here is that while the UI term contains the "interface" part, it is not really the "Application Programming" Interface that we would like to have.
It is not that the UI cannot be programmed against. After all, a web browser does just that, it takes a web page and renders it as expected. Same goes for desktop or mobile applications. Also, anyone integrating with mainframe computers will recognize that this is basically what 3270 can be used for too.
Rather, the fundamental issue is that web scraping goes against the principles of separation of layers and roles across frontend, middleware and backend, which in turn means that authors of resources (e.g. HTML pages) do not really expect for many people to access them in automated ways.
Perhaps they actually should expect it, and web pages should finally start to resemble genuine knowledge graphs, easy to access by humans, be it manually or through automation tools, but the reality today is that it is not the case and, in comparison with backend systems, the whole of the web scraping space is relatively brittle, which is why we shun this approach in integrations.
Yet, another part of reality, particularly in enterprise integrations, is that people may be sometimes given access to a frontend application on an internal network and that is it. No API, no REST, no JSON, no POST data, no real data formats, and one is simply supposed to fill out forms as part of a business process.
Typically, such a situation will result in an integration gap. There will be fully automated parts in the business process preceding this gap, with multiple systems coordinated towards a specific goal and there will be subsequent steps in the process, also fully automated.
Or you may be given access only to a specific frontend and only through VPN via a single remote Windows desktop. Getting access to a REST API may take months or may be never realized because of some high level licensing issues. This is not uncommon in the real life.
Such a gap can be a jarring and sore point, truly ruining the whole, otherwise fluid, integration process. This creates a tension and to resolve the tension, we can, should all the attempts to find a real API fail, finally resort to web scraping.
It is mostly in this context that I am looking at Playwright below - the tool is good and it has many other uses that go beyond the scope of this text, and it is well worth knowing it, for instance for frontend testing of your backend systems, but, when we deal with API integrations, we should not overdo with web scraping.
Needless to say, if web scraping is what you do primarily, your perspective will be somewhat different - you will not need any explanation of why it is needed or when, and you may be only looking for a way to enclose up your web scraping code in API services. This article will explain that too.
Introducing PlaywrightThe nice part of Playwright is that we can use it to visually prepare a draft of Python code that will scrape a given resource. That is, instead of programming it in Python, we go to an address, fill out a form, click buttons and otherwise use everything as usually and Playwright generates for us code that will be later used in integrations.
That code will require a bit of clean-up work, which I will talk about below, but overall it works very nicely and is certainly useful. The result is not one of these do-not-touch auto-generated pieces of code that are better left to their own.
While there are better ways to integrate with Jira, I chose that application as an example of Playwright's usage simply because I cannot show you any internal application in a public blog post.
Below, there are two windows. One is Playwright's emulating a Blackberry device to open a resource. I was clicking around, I provided an email address and then I clicked the same email field once more. To the right, based on my actions, we can find the generated Python code, which I consider quite good and readable.
The Playwright Inspector, the tool that gave us the code, will keep recording all of our actions until we click the "Record" button which then allows us to click the button next to "Record" which is "Copy code to clipboard". We can then save the code to a separate file and run it on demand, automatically.
But first, we will need to install Playwright.
Installing and starting PlaywrightThe tools is written in TypeScript and can be installed using npx, which in turn is part of NodeJS.
Afterwards, the "playwright install" call is needed as well because that will potentially install runtime dependencies, such as Chrome libraries.
Finally, we install Playwright using pip as well because we want to access with Python. Note that if you are installing Playwright under Zato, the "/path/to/pip" will be typically "/opt/zato/code/bin/pip".
npx -g --yes playwright install playwright install /path/to/pip install playwrightWe can now start it as below. I am using BlackBerry as an example of what Playwright is capable of. Also, it is usually more convenient to use a mobile version of a site when the main window and Inspector are opened side by side, but you may prefer to use Chrome, Firefox or anything else.
playwright codegen https://example.atlassian.net/jira --device "BlackBerry Z30"That is practically everything as using Playwright to generate code in our context goes. Open the tool, fill out forms, copy code to a Python module, done.
What is still needed, though, is cleaning up the resulting code and embedding it in an API integration process.
Code clean-upAfter you keep using Playwright for a while with longer forms and pages, you will note that the generated code tends to accumulate parts that repeat.
For instance, in the module below, which I already cleaned up, the same "[placeholder=\"Enter email\"]" reference to the email field is used twice, even if a programmer developing this could would prefer to introduce a variable for that.
There is not a good answer to the question of what to do about it. On the one hand, obviously, being programmers we would prefer not to repeat that kind of details. On the other hand, if we clean up the code too much, this may result in too much of a maintenance burden because we need to keep it mind that we do not really want to invest to much in web scraping and, should there be a need to repeat the whole process, we do not want to end up with Playwright's code auto-generated from scratch once more, without any of our clean-up.
A good compromise position is to at least extract any kind of credentials from the code to environment variables or a similar place and to remove some of the code comments that Playwright generates. The result as below is what it should like at the end. Not too much effort without leaving the whole code as it was originally either.
Save the code below as "play1.py" as this is what the API service below will use.
# -*- coding: utf-8 -*- # stdlib import os # Playwright from playwright.sync_api import Playwright, sync_playwright class Config: Email = os.environ.get('APP_EMAIL', 'zato@example.com') Password = os.environ.get('APP_PASSWORD', '') Headless = bool(os.environ.get('APP_HEADLESS', False)) def run(playwright: Playwright) -> None: browser = playwright.chromium.launch(headless=Config.Headless) # type: ignore context = browser.new_context() # Open new page page = context.new_page() # Open project boards page.goto("https://example.atlassian.net/jira/software/projects/ABC/boards/1") page.goto("https://id.atlassian.com/login?continue=https%3A%2F%2Fexample.atlassian.net%2Flogin%3FredirectCount%3D1%26dest-url%3D%252Fjira%252Fsoftware%252Fprojects%252FABC%252Fboards%252F1%26application%3Djira&application=jira") # Fill out the email page.locator("[placeholder=\"Enter email\"]").click() page.locator("[placeholder=\"Enter email\"]").fill(Config.Email) # Click #login-submit page.locator("#login-submit").click() with sync_playwright() as playwright: run(playwright) Web scraping as a standalone activityWe have the generated code so the first thing to do with it is to run it from command line. This will result in a new Chrome window's accessing Jira - it is Chrome, not Blackberry, because that is the default for Playwright.
The window will close soon enough but this is fine, that code only demonstrates a principle, it is not a full integration task.
python /path/to/play1.pyIt is also useful that we can run the same Python module from our IDE, giving us the ability to step through the code line by line, observing what changes when and why.
Web scraping as an API service
Finally, we are ready to invoke the standalone module from an API service, as in the following code that we are also going to make available as a REST channel.
A couple of notes about the Python service below:
- We invoke Playwright in a subprocess, as a shell command
- We accept input through data models although we do not provide any output definition because it is not needed here
- When we invoke Playwright, we set the APP_HEADLESS to True which will ensure that it does not attempt to actually display a Chrome window. After all, we intend for this service to run on Linux servers, in backend, and such a thing will be unlikely to work in this kind of an environment.
Other than that, this is a straightforward Zato service - it receives input, carries out its work and a reply is returned to the caller (here, empty).
# -*- coding: utf-8 -*- # stdlib from dataclasses import dataclass # Zato from zato.server.service import Model, Service # ########################################################################### @dataclass(init=False) class WebScrapingDemoRequest(Model): email: str password: str # ########################################################################### class WebScrapingDemo(Service): name = 'demo.web-scraping' class SimpleIO: input = WebScrapingDemoRequest def handle(self): # Path to a Python installation that Playwright was installed under py_path = '/path/to/python' # Path to a Playwright module with code to invoke playwright_path = '/path/to/the-playwright-module.py' # This is a template script that we will invoke in a subprocess command_template = """ APP_EMAIL={app_email} APP_PASSWORD={app_password} APP_HEADLESS=True {py_path} {playwright_path} """ # This is our input data input = self.request.input # type: WebScrapingDemoRequest # Extract credentials from the input .. email = input.email password = input.password # .. build the full command, taking all the config into account .. command = command_template.format( app_email = email, app_password = password, py_path = py_path, playwright_path = playwright_path, ) # .. invoke the command in a subprocess .. result = self.commands.invoke(command) # .. if it was not a success, log the details received .. if not result.is_ok: self.logger.info('Exit code -> %s', result.exit_code) self.logger.info('Stderr -> %s', result.stderr) self.logger.info('Stdout -> %s', result.stdout) # ###########################################################################Now, the REST channel:
The last thing to do is to invoke the service - I am using curl from the command line below but it could very well be Postman or a similar option.
curl localhost:17010/demo/web-scraping -d '{"email":"hello@example.com", "password":"abc"}' ; echoThere will be no Chrome window this time around because we run Playwright in the headless mode. There will be no output from curl either because we do not return anything from the service but in server logs we will find details such as below.
We can learn from the log that the command took close to 4 seconds to complete, that the exit code was 0 (indicating success) and that is no stdout or stderr at all.
INFO - Command ` APP_EMAIL=hello@example.com APP_PASSWORD=abc APP_HEADLESS=True /path/to/python /path/to/the-playwright-module.py ` completed in 0:00:03.844157, exit_code -> 0; len-out=0 (0 Bytes); len-err=0 (0 Bytes); cid -> zcmdc5422816b2c6ff9f10742134We are now ready to continue to work on it - for instance, you will notice that the password is visible in logs and this should not be allowed.
But, all such works are extra in comparison with the main theme - we have Playwright, which is a a tool that allows us to quickly integrate with frontend applications and we can automate it through API services. Just as expected.
Next steps- Read more about using Python in API integrations
- Start the tutorial which will guide you how to design and build Python API services for automation and integrations
The Drop Times: Closing Chapter: Reflecting on My Time with The DropTimes
Dear Readers,
As I write my The DropTimes newsletter, I'm filled with a bittersweet blend of gratitude and nostalgia. When I first joined The DropTimes, my understanding of Drupal was minimal, but stepping into this expansive world, I was not only educated but deeply inspired by the robust spirit of our community. Throughout my tenure, I've had the unique privilege to connect with many of you—talented individuals from across the globe, each sharing the same passion and dedication.
Over these months, The DropTimes has stood as a never-fading testimony to the vibrant and ever-evolving Drupal world, chronicling its achievements, challenges, and the incredible community that drives its success. Today, I am sharing not just another update, but a personal farewell. May the coming chapters of my life lead me towards new beginnings, filled with personal and professional growth.
As I close this significant chapter at The DropTimes, I want to extend my deepest gratitude to all of you—my colleagues, our readers, and the entire Drupal community—for the support, inspiration, and camaraderie. It has been a profound journey, one that has enriched me beyond words, and I look forward to carrying these memories and lessons with me into my future endeavors.
So, with that said, let me, for the last time, take you through the stories we covered last week.
Kazima Abbas, a sub-editor at TDT, unveils insights from two significant events. Acquia Engage London 2024, which took place from May 21 to 22 marked the first European stop of the 2024 Digital Freedom Tour. It convened digital leaders who shared their expertise, insights, and practical tips on crafting impactful customer experiences. Learn more here. The next event is, Evolve Drupal Montreal 2024, organized by Evolving Web following the success of EvolveDrupal Atlanta. This upcoming summit, set for June 14, 2024, marks its return to Montreal where it debuted in May 2023. Read about this in detail here.
A few other important updates are; Drupal has launched the IXP Fellowship Initiative survey to bolster support for inexperienced developers looking to kickstart their careers in the Drupal ecosystem. By defining core competencies and gathering community input, this initiative aims to bridge the gap between training and practical experience, ultimately nurturing new talent within the community. Participate in shaping the future of Drupal development and read more about the initiative here.
Drupal 11 is set to remove several long-standing modules, such as Actions UI, Book, and Forum, in a bid to streamline its core functionality and focus on innovation. However, users need not fret as these features will still be accessible through contributed modules. This strategic move underscores Drupal's commitment to empowering site builders and ensuring a lean, efficient platform for ambitious digital experiences. Learn more about the changes and their implications here.
Michael Anello, on DrupalEasy, sheds light on the pressing need for fresh talent in the Drupal community, as evidenced by the concerning lack of new developers highlighted at DrupalCon Portland 2024. With only 9.1% of respondents under 30 in the 2024 Drupal Developer Survey, urgent action is needed to attract and retain young developers. Michael proposes several strategic measures, including modernizing Drupal's code and creating educational programs, to address this challenge. Get involved in shaping the future of Drupal development and read more about Michael's insights here.
New dates have been announced for DrupalCon Asia 2024, set to take place in Singapore from December 9th to 11th, 2024. Know more about the three-day event here. Applications are now open for grants and scholarships to attend DrupalCon Barcelona 2024 until June 28th, 2024. The initiative, led by the Drupal Association in partnership with Kuoni Tumlare Congress, aims to promote diversity and inclusivity within the open-source community.
The Drupal Brisbane meetup is scheduled to resume on June 18, 2024. This event offers both in-person attendance at Brisbane Square Library and the option to participate online, providing an opportunity for individuals to engage in discussions surrounding Drupal and contribute to the community. Interested participants are encouraged to submit their topic suggestions, fostering an inclusive environment for collaborative discourse. A complete list of events for the week is available here.
The Technical Working Group (TWG) has scheduled a final discussion on proposed changes to Drupal's coding standards for June 5, 2024, UTC. The focus of this discussion will be the coding style for PHP Enumerations, inviting community input to refine Drupal's coding practices.
The Drupal Association has appointed Simba Ndemera as its new Chief Financial and Operations Officer, effective April 2024. With nearly three decades of experience in finance and a strong background in nonprofit accounting, Simba brings valuable expertise to his role. His dedication to community service and advancing open-source technology aligns perfectly with the organization's mission, promising a collaborative effort toward progress and inclusivity in the tech industry.
Bluechip Tech Limited, headquartered in the UK, has unveiled a new training course focusing on Drupal responsive design. Geared towards educating participants on crafting responsive and adaptive designs with Drupal and its modules, the course covers essential principles and techniques. Learn more about this new training here. Additionally, Evolving Web is offering a series of in-person Drupal training sessions next month in Montreal, aimed at enhancing digital practices for teams and individuals. These full-day training sessions, scheduled for June 11 to 13, cover crucial aspects of Drupal, providing participants with expert knowledge and practical skills.
Frontkom has announced the imminent release of Drupal Gutenberg 3.0.0, promising enhanced customization options and improved support for content blocks in Drupal. This update, designed to simplify content creation with advanced style controls and user-defined patterns, aims to elevate the user experience within the Drupal ecosystem.
Introducing the Time Machine module for Drupal, crafted by Mandip Singh, offering administrators seamless site restoration capabilities to any desired point in time. With comprehensive rollback features covering content, configuration, and user data, this module ensures robust disaster recovery and facilitates safe experimentation. Read more about this new module here.
The Drupal Association has issued an update on its Global Accessibility Awareness Day (GAAD) Pledge for 2024, reaffirming Drupal's commitment to accessibility standards. Led by Mike Gifford, the Drupal accessibility maintainers are actively working to align with WCAG 2.2 AA standards, aiming for inclusivity across the platform. With ongoing efforts to address accessibility issues and promote community involvement, Drupal continues its mission to ensure accessibility for all users.
We acknowledge that there are more stories to share. However, due to selection constraints, we must pause further exploration for now.
To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.
For the Last Time,
Sincerely,
Elma John
Sub-editor, The DropTimes.
Quansight Labs Blog: Dataframe interoperability - what has been achieved, and what comes next?
FOSSASIA 2024: An Unforgettable Experience in Vietnam
Embarking on my journey to FOSSASIA 2024, I felt a mix of excitement and anticipation. As I boarded my flight, the reality of the adventure ahead began to sink in. I arrived early in the morning on April 7th, filled with enthusiasm and a bit of jet lag, ready to dive into the vibrant culture of Vietnam and the exhilarating events planned for the conference.
Arrival in HanoiLanding in Hanoi, I was greeted by the warm, humid air and a bustling airport scene. I quickly hopped onto Bus Express 86, which conveniently took me from the airport to the heart of the city. The ride itself was a mini-tour, offering glimpses of Hanoi’s unique blend of traditional and modern architecture. My destination was Hotel LakeSide, where I was warmly welcomed and offered an early check-in at no extra cost—a gesture that felt like a blessing after a long flight. The hotel staff’s generosity allowed me to freshen up and catch a few hours of much-needed sleep before the scheduled Hanoi City Tour organized by FOSSASIA.
A special shout-out goes to Lily’s Travel Agency for arranging an amazing stay throughout the conference days. For transportation, I relied on Grab, a widely used cab application in Vietnam similar to Uber. Despite the language barrier, the local drivers were exceptionally supportive and always willing to go the extra mile, which made commuting around the city a breeze.
Exploring HanoiBy 2 pm, I was ready for the city tour, excited to explore Hanoi’s rich history and culture. Our itinerary included visits to the One Pillar Pagoda, the Temple of Literature & National University, and the Imperial Citadel of Thang Long. Each site was more breathtaking than the last, steeped in history and surrounded by lush greenery. It was a day filled with fun, laughter, and new friendships. We shared stories, took countless photos, and even indulged in some silly antics that added a touch of whimsy to the day (try to find it in the photos!).
After the tour concluded around 7 pm, we split into groups to find dinner. I joined a few new friends for a delicious meal at a local restaurant, savoring the flavors of authentic Vietnamese cuisine. Later, I met Phu Nguyen, a KDE contributor from Hanoi, to collect a monitor for our booth. Phu, who works in Germany, couldn’t attend the event but was incredibly helpful in providing the display. With the monitor in hand, I returned to my hotel, reflecting on a day well spent and eagerly anticipating the start of the conference the next day.
Conference KickoffEach morning began with a hearty breakfast at the hotel, overlooking the serene Giang Vo Lake. Armed with hardware, promotional materials, and stickers, I set off for the Posts and Telecommunications Institute of Technology, where FOSSASIA 2024 was held. The venue buzzed with excitement, as hundreds of students and tech enthusiasts gathered around, curious about the event.
Our Booth: The Focal PointOur booth, strategically placed next to FSFE, COSCUP, and CalyxOS, quickly became the busiest spot at the venue. With the invaluable help of Aniqa and Paul from the KDE promo team, we had an impressive setup that drew in crowds continuously. Aniqa and Paul were instrumental in organizing the booth, ensuring we had everything we needed, and providing ongoing support throughout the event. Tomaz joined us on the first day, bringing a reMarkable tablet and his personal laptop to enhance our setup. Our booth was a vibrant hub of activity, attracting attendees with emulations of Nintendo games and Tomaz’s impressive origami skills. He exchanged his intricate origami creations for discussions about KDE, engaging many curious students.
Engaging with the CommunityThe conference was a vibrant hub of activity, featuring organizations such as FreeCAD, AlmaLinux, and fossunited. It was inspiring to see the diversity of projects and the passion driving each community. Our booth quickly became the most popular, with attendees lining up to learn about KDE’s latest projects and initiatives. Despite most attendees being college students with limited funds, their enthusiasm for KDE was overwhelming. Tomaz’s origami talent truly stood out, drawing significant attention and sparking numerous conversations about open-source software and community contributions.
Walking through the venue, I marveled at the variety of booths and the innovative projects on display. The FreeCAD team showcased their latest developments in open-source CAD software, while AlmaLinux representatives engaged attendees with their enterprise-grade Linux distribution. fossunited’s booth was a hive of activity, emphasizing their mission to promote and support open-source projects in India. Each interaction was a learning opportunity, and the camaraderie among the open-source communities was palpable.
Evening Get-TogethersEvenings were spent exploring the local culture, including a memorable Evening Get Together at Ta Hien Beer and Food Street. The lively atmosphere, coupled with delicious street food and refreshing drinks, made for perfect networking opportunities and deepened the bonds formed during the day. The conference days were a whirlwind of activity, leaving us with lasting impressions and numerous connections.
Sapa Valley Trek and FarewellAfter the conference, I ventured to Sapa Valley for a three-day trek. The journey was a stark contrast to the bustling city, offering a tranquil escape into nature. Walking among the rice paddies, often alongside local farmers, and soaking in the serene landscape was a rejuvenating experience. The trek left me with fond memories of the picturesque valley, the warmth of the local people, and the breathtaking beauty of Vietnam’s countryside.
I returned on April 15th, with a heart full of memories and a mind brimming with inspiration from FOSSASIA 2024. The conference not only highlighted the incredible work being done in the open-source community but also showcased the rich culture and hospitality of Vietnam. Reflecting on the experience, I felt a deep sense of gratitude and excitement for the future.
I want to extend my heartfelt thanks to KDE e.V. for sponsoring my attendance at this event. Their support made this enriching experience possible, and I am profoundly grateful for the opportunity to represent the KDE Community at FOSSASIA 2024. Until next time, FOSSASIA!
Gallery Tomaz Tomaz and me with the booth! Attendees at the KDE booth Even more attendees Tomaz multitasking with Origami! reMarkable tablet The booth bustling with attendees Tomaz and Origami Hong Phuc Dang with Konqi! Train Street! Train Street! Again! Ubuntu! Anuvrat at the Food Street! Food Street! Origami Horse by Tomaz! From the International Lounge at Delhi Airport An interesting mode of transport Assembly for the Ha Noi City Tour Temple of Literature! Souvenirs! Souvenirs! Artists on work! The Tomb The Citadel One Pillar Pagoda! Inside the One Pillar Pagoda! View from the One Pillar Pagoda! My guide during the trek, Mama Mao! Sa Pa Valley Getting down from the mountains Food at TrekRussell Coker: USB-A vs USB-C
USB-A is the original socket for USB at the PC end. There are 2 variants of it, the first is for USB 1.1 to USB 2 and the second is for USB 3 which adds extra pins in a plug and socket compatible manner – you can plug a USB-A device into a USB-A socket without worrying about the speeds of each end as long as you don’t need USB 3 speeds.
The differences between USB-A and USB-C are:
- USB-C has the same form factor as Thunderbolt and the Thunderbolt protocol can run over it if both ends support it.
- USB-C generally supports higher power modes for charging (like 130W for Dell laptops, monitors, and plugpacks) but there’s no technical reason why USB-A couldn’t do it. You can buy chargers that do 60W over USB-A which could power one of our laptops via a USB-A to USB-C cable. So high power USB-A is theoretically possible but generally you won’t see it.
- USB-C has “DisplayPort alternate mode” which means using some of the wires for DisplayPort.
- USB-C is more likely to support the highest speeds than USB-A sockets for “super speed” etc. This is not a difference in the standards just a choice made by manufacturers.
While USB-C tends to support higher power delivery modes in actual implementations for connecting to a PC the PC end seems to only support lower power modes regardless of port. I think it would be really good if workstations could connect to monitors via USB-C and provide power, DisplayPort, and keyboard, mouse, etc over the same connection. But unfortunately the PC and monitor ends don’t appear to support such things.
If you don’t need any of those benefits in the list above (IE you are using USB for almost anything we do other than connecting a laptop to a dock/monitor/charger) then USB-A will do the job just as well as USB-C. The choice of which type to use should be based on price and which ports are available, EG My laptop has 2*USB-C ports and 2*USB-A so given that one USB-C port is almost always used for the monitor or for charging I don’t really want to use USB-C for anything else to avoid running out of ports.
When buying USB devices you can’t always predict which systems you will need to connect them to. Currently there are a lot of systems without USB-C that are working well and have no need to be replaced. I haven’t yet seen a system where the majority of ports are USB-C but that will probably happen in the next few years. Maybe in 2027 there will be PCs on sale with only two USB-A sockets forcing people who don’t want to use a USB hub to save both of them for keyboard and mouse. Currently USB-C keyboards and mice are available on AliExpress but they are expensive and I haven’t seen them in Australian stores. Most computer users don’t wear out keyboards or mice so a lot of USB-A keyboard and mice will be in service for a long time. As an aside there are still many PCs with PS/2 keyboard and mouse ports in service so these things don’t go away for a long time.
There is one corner case where USB-C is convenient which is when you want to connect a mass storage device for system recovery or emergency backup, want a high speed, and don’t want to spend time figuring out which of the ports are “super speed” (which can be difficult at the back of a PC with poor lighting). With USB-C you can expect a speed of at least 5Gbit/s and don’t have to worry about accidentally connecting to a USB 2 port as is the situation with USB-A.
For my own use the only times that I prefer USB-C over USB-A are for devices to connect to phones. Eventually I’ll get a laptop that only has USB-C ports and this will change, but even then adaptors are possible.
For someone who doesn’t know the details of how things works it’s not unreasonable to just buy the newest stuff and assume it’s better as it usually is. But hopefully blog posts like this can help people make more informed decisions.
Related posts:
- Dell 32″ 4K Monitor and DisplayPort Switch After determining that the Philips 43″ monitor was too large...
- Cheap Peripherals for Work A problem with a lot of the purchase of peripherals...
- Thinkpad X1 Carbon Gen 6 In February I reviewed a Thinkpad X1 Carbon Gen 1...
XRechnung Viewer Release 1.0
Some time ago I quickly wrote a little utility to render XRechnung documents on the free desktop, called XRView. This is the initial Bogpost. It was a very fundamental Qt Widget app that shows e-invoice docs that come in the XRechnung XML format, in a human readable view.
It was never properly released, so recently I decided to wrap it up and finally cut a first release which people can find on the release page on Codeberg.
Technically it uses the XSLT stylesheets provided by Kosit and calls an external java process on the local machine to run that through a specific Saxon processor. For that, XRView requires a java runtime installed.
Since the setup of these dependencies is a bit cumbersome, the new release 1.0 does that for users. It downloads the stylesheets and also the saxon processor runtime from their upstream repositories and stores them on the local machine for future use. Of course it is strongly recommended to double check the downloaded resources for their validity and integrity and not to run software that some other code downloaded.
More new features in this first release are:
- internationalization, first available language is German
- a rudimentary application menu with about dialog and such
- the setup routine as described above
Note that this is the first release of the software. Yet, I think it is useful, and a interesting starting point for further activities in this area. As XRechnung will become a mandatory standard for all companies in Germany (at least) I think it is very important to have a free software alternative. There are already many commercial offerings.
However, I am not feeling to develop and maintain this as an “one man show” forever. Being kind of frustrated about the way how free software is often consumed nowadays, I will happily continue to contribute to it if there is more interest than “gimme for free” by other people or organizations.
Let’s see if this is heading somewhere :-)
Guido Günther: Don't unblank in my back pack please
Talk Python to Me: #463: Running on Rust: Granian Web Server
This week in KDE: Triple buffering and other sources of amazingness
We just branched for Plasma 6.1 and released the beta, which means the window to add new features has now closed. But before it did, a ton of amazing stuff snuck in! Plasma 6.1 promises to be a large and impressive release.
Probably the most impactful thing is triple buffering support on Wayland! This should make animations and screen rendering smoother in general–ideally up to the level of the X11 session, which already did triple buffering. This work by Xaver Hugl has been in progress for a long time and lands in Plasma 6.1. Link
That’s not all though… oh no, not by a long shot:
New FeaturesDolphin now includes a feature to move the selected items into a new folder, all at once (Ahmet Hakan Çelik, Dolphin 24.08. Link)
KDE’s desktop portal implementation now includes support for the Input Capture portal (David Redondo, Plasma 6.1. Link 1, link 2, and link 3)
Plasma now supports enabling and disabling the feature of some Lenovo IdeaPad and Legion laptops whereby the battery can be configured to only charge up to a specific fixed level (sometimes 60%, sometimes 80%; it depends on the machine) to maximize battery health (Fabian Arndt, Plasma 6.1. Link)
Plasma’s Edit Mode has a beautiful new zoom-out effect to help you notice and understand you’re in a separate mode, and also make it easier to get out once you’re done (Marco Martin, Plasma 6.1. Link 1 and link 2):
https://i.imgur.com/EryNKTq.mp4You can now configure the screen locker to unlock without a password, letting it be used as a traditional screensaver if you enable a visually attractive wallpaper plugin and disable the clock (Kristen McWilliam, Plasma 6.1. Link)
UI ImprovementsOur long national nightmare of jarring error beep sounds is now over!!!! Plasma now intercepts attempts to ring the system bell (which generally sounds so unpleasant that you feel the need to immediately commit an act of violence) and replaces them with a nice sound from the active sound theme (Nicolas Fella, Plasma 6.1. Link 1, link 2, and link 3)
KRunner search results already prioritized apps by default, but now they also prioritize System Settings pages too (Alexander Lohnau, Plasma 6.1, link 1, and link 2)
On System Settings’ Power Management page, a few UI controls that used spinboxes have been replaced with fancy comboboxes. This fixes some bugs and offers a faster interaction paradigm for the basic use case of choosing a common value — with an expert workflow of letting you select anything you want in a dialog box (Jakob Petsovits, Plasma 6.1. Link):
https://i.imgur.com/2YISTOz.mp4System Settings’ Printers page now guides you through the process of installing the system-config-printer package to improve printer detection, if it wasn’t pre-installed by your distro (Mike Noe, Plasma 6.1. Link)
Getting information from weather providers can sometimes be a bit flaky, so Plasma’s Weather Report widget now informs you to just try again in a little bit when this happens (me: Nate Graham, Plasma 6.1. Link):
The way Welcome Center presents KRunner has gotten a major overhaul, and now shows a fancy animated depiction of actually using it! In addition, the final page is now more streamlined and less demanding of your time and money (Oliver Beard, Plasma 6.1. Link 1 and link 2):
Plasma’s Weather Widget no longer shows the “Appearance” page in its config window when used on the desktop, since nothing on that page is applicable to the desktop form factor (Ismael Asensio, Plasma 6.1. Link)
KWin’s custom tile editor now uses clearer terminology for creating split views (me: Nate Graham, Plasma 6.1. Link):
System Settings’ Background Services page is no longer actually visible in System Settings by default; everything here is an implementation detail, and monkeying with its settings is an easy way to break your system. If you’re an expert, you can still get to it by searching for it in KRunner, but it won’t be shown in System Settings anymore (Nicolas Fella, Plasma 6.1. Link)
The remainder of the header messages in System Settings pages have been ported to the new frameless style, making them all consistent now (me: Nate Graham, Plasma 6.2. Link):
Improved the way SVG images render on screen when using a fractional scale factor, reducing blurriness (Marco Martin, Frameworks 6.3. Link)
Bug FixesFilelight no longer counts files stored in OneDrive cloud as local files that occupy space (Harald Sitter, Filelight 24.05.1. Link)
In KColorChooser, the “Pick Screen Color” button is no longer missing on Wayland (Thomas Weißschuh, KColorChooser 24.05.1. Link)
Made Plasma more robust against crashing when any widgets have malformed size values, which can happen under certain circumstances (Marco Martin, Plasma 6.0.5. Link)
When KWin falls back to using a software cursor after the graphics driver rejected the use of a hardware cursor, this can no longer lock up the entire screen under certain circumstances — such as with XWayland-using apps on an Apple Silicon Mac (Xaver Hugl, Plasma 6.0.5. Link)
Spectacle no longer takes blurry screenshots on systems with multi-screen plus mixed-scale-factor setups (Volodymyr Zolotopupov, Plasma 6.0.5. Link)
Global shortcuts are now more robust and stable on NixOS and other distros that regenerate the sycoca cache repeatedly in an automated manner (Vlad Zahorodnii, Plasma 6.0.5. Link)
Fixed multiple longstanding issues in System Settings whereby switching pages, clearing the search field, or opening a new page form outside of System Settings would cause the sub-category column to show the wrong thing (Nicolas Fella, Plasma 6.0.5. Link 1, link 2, and link 3)
Fixed a case where turning off an external monitor plugged into a laptop with its lid closed could cause KWin to crash (Xaver Hugl, Plasma 6.1. Link)
On Wayland, Plasma no longer quits when you open an enormous number of windows (Xaver Hugl, Plasma 6.1. Link)
The “Activation Gestures” category of System Settings’ Accessibility page is back, after being accidentally removed when the page was ported to QML (Nicolas Fella, Plasma 6.1. Link)
On Wayland, when any apps that have System Tray icons are running, there’s no longer a little invisible square in the top-left corner of the screen that eats input, and also no elevated CPU usage with certain screen arrangements (David Edmundson, Plasma 6.1. Link 1 and link 2)
Pressing Meta+B repeatedly no longer opens multiple Power Profile chooser OSDs, and therefore no longer represents a way for you to exhaust your system’s memory by generating an infinite stack of them (Fabian Arndt, Plasma 6.1. Link)
Made KWin more reliable about detecting screens’ physical sizes (Jakub Piecuch, Plasma 6.1. Link)
When using a Plasma Panel in “Fit content” mode with only an Icons-Only Task Manager on it, there’s no longer unnecessary empty space on the right side of it on login (Akseli Lahtinen, Plasma 6.1. Link)
In the dialog that lets you choose windows and screens to share, clicking on the checkboxes to select items now works. Previously you had to click on the whole items themselves, but the checkboxes didn’t work; now both work (me: Nate Graham, Frameworks 6.3. Link)
Fixed several issues preventing certain Breeze icons from adjusting their colors properly when run with a dark color scheme, as well as issues with generation of static dark-theme-compatible icons (Corbin Schwimmbeck, Frameworks 6.3. Link 1 and link 2)
Re-spun the KWidgetsAddons framework to include a bugfix for an issue that caused OBS to crash when selecting files, and also one that caused KMessageWidgets to sometimes show incorrect background colors (Joshua Goins and Albert Astals Cid, KWidgetsAddons frameworks 6.2.2. Link 1 and link 2)
Re-spun the KWallet framework to include a bugfix for an issue that caused the Secrets portal to not work in Flatpak apps (Nicolas Fella, KWallet 6.2.1. Link)
Context menus should now be a lot less likely to appear as odd standalone windows with titlebars when activated on an inactive window (Vlad Zahorodnii, Qt 6.7.2. Link)
Other bug information of note:
- 3 Very high priority Plasma bugs (same as last week). Current list of bugs
- 36 15-minute Plasma bugs (down from 39 last week). Current list of bugs
- 105 KDE bugs of all kinds fixed over the last week Full list of bugs
Reduced frame drop on a variety of hardware (Xaver Hugl, Plasma 6.1. Link)
Improved the speed with which Discover launches and how responsive it is when scrolling through long app lists while the Flatpak backend is active (Aleix Pol Gonzalez, Plasma 6.1. Link)
Automation & SystematizationAdded some new autotests for Plasma panels and containments to make sure they get sized and located correctly (Marco Martin and Akseli Lahtinen. Link 1, link 2, and link 3)
…And Everything ElseThis blog only covers the tip of the iceberg! If you’re hungry for more, check out https://planet.kde.org, where you can find more news from other KDE contributors.
How You Can HelpThe KDE organization has become important in the world, and your time and labor have helped to bring it there! But as we grow, it’s going to be equally important that this stream of labor be made sustainable, which primarily means paying for it. Right now the vast majority of KDE runs on labor not paid for by KDE e.V. (the nonprofit foundation behind KDE, of which I am a board member), and that’s a problem. We’ve taken steps to change this with paid technical contractors — but those steps are small due to growing but still limited financial resources. If you’d like to help change that, consider donating today!
Otherwise, visit https://community.kde.org/Get_Involved to discover other ways to be part of a project that really matters. Each contributor makes a huge difference in KDE; you are not a number or a cog in a machine! You don’t have to already be a programmer, either. I wasn’t when I got started. Try it, you’ll like it! We don’t bite!
Gunnar Wolf: How computers make books • from graphics rendering, search algorithms, and functional programming to indexing and typesetting
If we look at the age-old process of creating books, how many different areas can a computer help us with? And how can each of them be used to teach computer science (CS) fundamentals to a nontechnical audience? This is the premise of John Whitington’s enticing book and the result is quite amazing.
The book immediately drew my attention when looking at the titles available for review. After all, my initiation into computing as a kid was learning the LaTeX typesetting system while my father worked on his first book on scientific language and typography [1]. Whitington picks 11 different technical aspects of book production, from how dots of ink are transferred to a white page and how they are made into controllable, recognizable shapes, all the way to forming beautiful typefaces and the nuances of properly addressing white-space to present aesthetically pleasing paragraphs, building it all into specific formats aimed at different ends.
But if we dig beyond just the chapter titles, we will find a very interesting book on CS that, without ever using technical language or notation, presents aspects as varied as anti-aliasing, vector and raster images, character sets such as ASCII and Unicode, an introduction to programming, input methods for different writing systems, efficient encoding (compression) methods, both for text and images, lossless and lossy, and recursion and dithering methods. To my absolute surprise, while the author thankfully spared the reader the syntax usually associated with LISP-related languages, the programming examples clearly stem from the LISP school, presenting solutions based on tail recursion. Of course, it is no match for Donald Knuth’s classic book on this same topic [2], but could very well be a primer for readers to approach it.
The book is light and easy to read, and keeps a very informal, nontechnical tone throughout. My only complaint relates to reading it in PDF format; the topic of this book, and the care with which the images were provided by the author, warrant high resolution. The included images are not only decorative but an integral part of the book. Maybe this is specific to my review copy, but all of the raster images were in very low resolution.
This book is quite different from what readers may usually expect, as it introduces several significant topics in the field. CS professors will enjoy it, of course, but also readers with a humanities background, students new to the field, or even those who are just interested in learning a bit more.
References-
Sánchez y Gándara, A.; Magariños Lamas, F.; Wolf, K. B., Manual de lenguaje y tipografía científica en castellano. Trillas, Mexico City, Mexico, 1986, https://www.fis.unam.mx/~bwolf/manual.html
-
Knuth, D. E. Digital typographyCSLI Lecture Notes: CSLI Lecture Notes. CSLI Publications, Stanford, CA, 1999, https://www-cs-faculty.stanford.edu/~knuth/dt.html
ImageX: The Benefits of a Composable CMS (And How Drupal Fits the Bill)
This article was updated in May 2024.
As a marketing leader, you need to drive traffic to your site and create a superior user experience. But you also need to push your content out to a variety of channels so you can reach your audience where they are.
To achieve your goals, you need a content management system (CMS) that’s flexible, scalable, and efficient. And if you’re researching your options, you’ve probably heard a lot about composable CMSs.
1xINTERNET blog: CMS features every editor and marketer needs
Every marketer and content editor deserves solutions that deliver outstanding results. Check how a preconfigured Drupal CMS can make your daily work easier!
Real Python: Quiz: How to Create Pivot Tables With pandas
In this quiz, you’ll test your understanding of how to create pivot tables with pandas.
By working through this quiz, you’ll review your knowledge of pivot tables and also expand beyond what you learned in the tutorial. For some of the questions, you’ll need to do some research outside of the tutorial itself.
[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]
Web Review, Week 2024-21
Let’s go for my web review for the week 2024-21.
Gender bias in open source: Pull request acceptance of women versus menTags: tech, foss, bias
A bit too GitHub centric for my taste. Still it shows some unwarranted bias, especially when outsiders to a project are identified as women. We should do better.
Tags: tech, foss, version-control, linux, git, history
The often forgotten history behind the creation of Git. This article does a good job summarizing it.
https://graphite.dev/blog/bitkeeper-linux-story-of-git-creation
Tags: tech, ai, machine-learning, gpt, data
The training dataset crisis is looming in the case of large language models. They’ll sooner or later run out of genuine content to use… and the generated toxic waste will end up in training data, probably leading to dismal results.
https://pluralistic.net/2024/03/14/inhuman-centipede/#enshittibottification
Tags: tech, ai, machine-learning, gpt, google, data, quality
No, your model won’t get smarter just by throwing more training data at it… on the contrary.
Tags: tech, ai, machine-learning, art, social-media, criticism
This is indeed sad to see another platform turn against its users. This was once a place to nurture young artists… it’s now another ad driven platform full of AI made scams.
https://slate.com/technology/2024/05/deviantart-what-happened-ai-decline-lawsuit-stability.html
Tags: tech, ai, machine-learning, gpt, criticism
Open is unsurprisingly only in the name… this company is really just a cult.
Tags: tech, microsoft, windows, security, privacy
This is completely nuts… they really want to unleash a security and privacy nightmare. The irony is that it does respect DRM content on the other hand, we can see where the priorities are.
Tags: tech, ai, machine-learning, gpt, hype
Definitely this, it’s not the first time we see such a hype cycle around “AI”. When it bursts the technology which created it is just not called “AI” anymore. I wonder how long this one will last though.
https://blog.glyph.im/2024/05/grand-unified-ai-hype.html
Tags: tech, ai, machine-learning, gpt, hype, criticism
Definitely too much hype around large models right now. This over shadows the more useful specialized models.
https://www.dbreunig.com/2024/05/16/sober-ai.html
Tags: tech, google, microsoft, web, search
We’re still fairly dependent on just two major web indices… time for an index built as a common for everyone to use?
Tags: tech, web, search
Looks like an interesting new search engine.
https://github.com/StractOrg/stract?tab=readme-ov-file
Tags: tech, email, debugging
Fascinating bug… the fine details of mundane protocols like SMTP can sometimes be surprising.
https://tjaart.substack.com/p/the-curious-case-of-the-missing-period
Tags: tech, firefox, bookmarks
Interesting Firefox feature I didn’t notice. Looks fairly nice, I’ll use it more.
https://blog.meain.io/2024/firefox-bookmark-keywords
Tags: tech, web, frontend, cad, physics, mathematics
This gives a good idea of the important parts in a CAD program. It also list a few of the usable libraries to build one such program in the browser.
https://mattferraro.dev/posts/cadmium
Tags: tech, webassembly, server
Where WebAssembly is, and where WebAssembly on the server is going… let’s hope it doesn’t become another CORBA.
https://kerkour.com/webassembly-wasi-preview2
Tags: tech, git, ci
Neat trick, especially useful for CI uses.
https://blog.hartwork.org/posts/clone-arbitrary-single-git-commit/
Tags: tech, version-control, writing, communication
Very extensive guide on writing better commit messages. This is important, it’s a very central communication mechanism with other developers.
https://www.chiark.greenend.org.uk/~sgtatham/quasiblog/commit-messages/
Tags: tech, gui, ux
Interesting discussion about UI density. What are we talking about? Is there value to is? Which aspects of a UI are impacting it? The conclusion makes it all very clear.
https://matthewstrom.com/writing/ui-density/
Bye for now!
KDDockWidgets 2.1 Released
KDDockWidgets has launched its latest version 2.1. This release comes packed with over 500 commits, offering enhanced stability over its predecessor, version 2.0, without introducing any breaking changes.
KDDockWidgets is a versatile framework for custom-tailored docking systems in Qt written by KDAB’s Sérgio Martins. For more information about its rich set of features, have a look at its GitHub repository.
What’s changed in version 2.1?Here are the main highlights:
Bug Fixes:For starters, KDDW 2.1 introduces a range of bug fixes aimed at enhancing stability and user experience. Less popular features like nested main-windows and auto-hide received lots of attention and window manager specific bugs such as restoring maximized windows were addressed.
KDDW is now memory-leak free, several singletons were leaking before. We’ve added a valgrind GitHub Actions workflow to prevent regressions regarding leaks.
QtQuick:KDDW 2.1 also introduces improvements in QtQuick. New features include an API for setting affinities via QML, enabling mixing MDI with docking similar to QtWidgets, and fixing DPI issues of icons in TitleBar.qml for better scaling at 150% and 200%. Additionally, it resolves issues such as MDI widgets not raising when clicked on and various crashes related to MDI mode.
QtWidgets:For QtWidgets, we’ve improved handling of the preferredSize argument when adding dock widgets. It was being ignored in some cases. Overriding DockWidget::closeEvent() can now be done to prevent closing. Several crashes were fixed and we’ve added a GitHub Actions workflow which runs the tests against a Qt built with AddressSanitizer.
These enhancements improve the functionality and stability of KDDW 2.1 across different Qt environments.
Miscellaneous:KDDW 2.1 brings miscellaneous updates, including an upgrade to nlohmann json v3.11.3, the addition of a standalone layouting example using the UI toolkit Slint, and extensive testing on CI. Additionally, Config::setLayoutSpacing(int) has been added for increased customization.
Learn about all the changes here. Let us know what you think.
More information about KDDockWidgets-
- A manual on installation and usage written by KDAB’s Sérgio Martins.
- Find out more about KDAB’s Qt tools.
- A YouTube video about KDDockWidgets:
About KDAB
If you like this article and want to read similar material, consider subscribing via our RSS feed.
Subscribe to KDAB TV for similar informative short video content.
KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.
The post KDDockWidgets 2.1 Released appeared first on KDAB.
Julian Andres Klode: Observations in Debian dependency solving
In my previous blog, I explored The New APT 3.0 solver. Since then I have been at work in the test suite making tests pass and fixing some bugs.
You see for all intents and purposes, the new solver is a very stupid naive DPLL SAT solver (it just so happens we don’t actually have any pure literals in there). We can control it in a bunch of ways:
- We can mark packages as “install” or “reject”
- We can order actions/clauses. When backtracking the action that came later will be the first we try to backtrack on
- We can order the choices of a dependency - we try them left to right.
This is about all that we really want to do, we can’t go if we reach a conflict, say “oh but this conflict was introduced by that upgrade, and it seems more important, so let’s not backtrack on the upgrade request but on this dependency instead.”.
This forces us to think about lowering the dependency problem into this form, such that not only do we get formally correct solutions, but also semantically correct ones. This is nice because we can apply a systematic way to approach the issue rather than introducing ad-hoc rules in the old solver which had a “which of these packages should I flip the opposite way to break the conflict” kind of thinking.
Now our test suite has a whole bunch of these semantics encoded in it, and I’m going to share some problems and ideas for how to solve them. I can’t wait to fix these and the error reporting and then turn it on in Ubuntu and later Debian (the defaults change is a post-trixie change, let’s be honest).
apt upgrade is hardThe apt upgrade commands implements a safe version of dist-upgrade that essentially calculates the dist-upgrade, and then undoes anything that would cause a package to be removed, but it (unlike its apt-get counterpart) allows the solver to install new packages.
Now, consider the following package is installed:
X Depends: A (= 1) | BAn upgrade from A=1 to A=2 is available. What should happen?
The classic solver would choose to remove X in a dist-upgrade, and then upgrade A, so it’s answer is quite clear: Keep back the upgrade of A.
The new solver however sees two possible solutions:
- Install B to satisfy X Depends A (= 1) | B.
- Keep back the upgrade of A
Which one does it pick? This depends on the order in which it sees the upgrade action for A and the dependency, as it will backjump chronologically. So
-
If it gets to the dependency first, it marks A=1 for install to satisfy A (= 1). Then it gets to the upgrade request, which is just A Depends A (= 2) | A (= 1) and sees it is satisfied already and is content.
-
If it gets to the upgrade request first, it marks A=2 for install to satisfy A (= 2). Then later it gets to X Depends: A (= 1) | B, sees that A (= 1) is not satisfiable, and picks B.
We have two ways to approach this issue:
- We always order upgrade requests last, so they will be kept back in case of conflicting dependencies
- We require that, for apt upgrade a currently satisfied dependency must be satisfied by currently installed packages, hence eliminating B as a choice.
See if you have a X Recommends: A (= 1) and a new version of A, A (= 2), the solver currently will silently break the Recommends in some cases.
But let’s explore what the behavior of a X Recommends: A (= 1) in combination with an available upgrade of A (= 2) should be. We could say the rule should be:
- An upgrade should keep back A instead of breaking the Recommends
- A dist-upgrade should either keep back A or remove X (if it is obsolete)
This essentially leaves us the same choices as for the previous problem, but with an interesting twist. We can change the ordering (and we already did), but we could also introduce a new rule, “promotions”:
A Recommends in an installed package, or an upgrade to that installed package, where the Recommends existed in the installed version, that is currently satisfied, must continue to be satisfied, that is, it effectively is promoted to a Depends.
This neatly solves the problem for us. We will never break Recommends that are satisfied.
Likewise, we already have a Recommends demotion rule:
A Recommends in an installed package, or an upgrade to that installed package, where the Recommends existed in the installed version, that is currently unsatisfied, will not be further evaluated (it is treated like a Suggests is in the default configuration).
Whether we should be allowed to break Suggests with our decisions or not (the old autoremover did not, for instance) is a different decision. Should we promote currently satisified Suggests to Depends as well? Should we follow currently satisified Suggests so the solver sees them and doesn’t autoremove them, but treat them as optional?
tightening of versioned dependenciesAnother case of versioned dependencies with alternatives that has complex behavior is something like
X Depends: A (>= 2) | B X Recommends: A (>= 2) | BIn both cases, installing X should upgrade an A < 2 in favour of installing B. But a naive SAT solver might not. If your request to keep A installed is encoded as A (= 1) | A (= 2), then it first picks A (= 1). When it sees the Depends/Recommends it will switch to B.
We can solve this again as in the previous example by ordering the “keep A installed” requests after any dependencies. Notably, we will enqueue the common dependencies of all A versions first before selecting a version of A, so something may select a version for us.
version narrowing instead of version choosingA different approach to dealing with the issue of version selection is to not select a version until the very last moment. So instead of selecting a version to satisfy A (>= 2) we instead translate
Depends: A (>= 2)into two rules:
-
The package selection rule:
Depends: AThis ensures that any version of A is installed (i.e. it adds a version choice clause, A (= 1) | A (= 2) in an example with two versions for A.
-
The version narrowing rule:
Conflicts: A (<< 2)This outright would reject a choice of A (= 1).
So now we have 3 kinds of clauses:
- package selection
- version narrowing
- version selection
If we process them in that order, we should surely be able to find the solution that best matches the semantics of our Debian dependency model, i.e. selecting earlier choices in a dependency before later choices in the face of version restrictions.
This still leaves one issue: What if our maintainer did not use Depends: A (>= 2) | B but e.g. Depends: A (= 3) | B | A (= 2). He’d expect us to fall back to B if A (= 3) is not installable, and not to B. But we’d like to enqueue A and reject all choices other than 3 and 2. I think it’s fair to say: “Don’t do that, then” here.
Implementing strict pinning correctlyAPT knows a single candidate version per package, this makes the solver relatively deterministic: It will only ever pick the candidate, or an installed version. This also happens to significantly reduce the search space which is good - less backtracking. An uptodate system will only ever have one version per package that can be installed, so we never actually have to choose versions.
But of course, APT allows you to specify a non-candidate version of a package to install, for example:
apt install foo/oracular-proposedThe way this works is that the core component of the previous solver, which is the pkgDepCache maintains what essentially amounts to an overlay of the policy that you could see with apt-cache policy.
The solver currently however validates allowed version choices against the policy directly, and hence finds these versions are not allowed and craps out. This is an interesting problem because the solver should not be dependent on the pkgDepCache as the pkgDepCache initialization (Building dependency tree...) accounts for about half of the runtime of APT (until the Y/n prompt) and I’d really like to get rid of it.
But currently the frontend does go via the pkgDepCache. It marks the packages in there, building up what you could call a transaction, and then we translate it to the new solver, and once it is done, it translates the result back into the pkgDepCache.
The current implementation of “allowed version” is implemented by reducing the search space, i.e. every dependency, we outright ignore any non-allowed versions. So if you have a version 3 of A that is ignored a Depends: A would be translated into A (= 2) | A (= 1).
However this has two disadvantages. (1) It means if we show you why A could not be installed, you don’t even see A (= 3) in the list of choices and (2) you would need to keep the pkgDepCache around for the temporary overrides.
So instead of actually enforcing the allowed version rule by filtering, a more reasonable model is that we apply the allowed version rule by just marking every other version as not allowed when discovering the package in the from depcache translation layer. This doesn’t really increase the search space either but it solves both our problem of making overrides work and giving you a reasonable error message that lists all versions of A.
pulling up common dependencies to minimize backtracking costOne of the common issues we have is that when we have a dependency group
`A | B | C | D`we try them in order, and if one fails, we undo everything it did, and move on to the next one. However, this isn’t perhaps the best choice of operation.
I explained before that one thing we do is queue the common dependencies of a package (i.e. dependencies shared in all versions) when marking a package for install, but we don’t do this here: We have already lowered the representation of the dependency group into a list of versions, so we’d need to extract the package back out of it.
This can of course be done, but there may be a more interesting solution to the problem, in that we simply enqueue all the common dependencies. That is, we add n backtracking levels for n possible solutions:
- We enqueue the common dependencies of all possible solutions deps(A)&deps(B)&deps(C)&deps(D)
- We decide (adding a decision level) not to install D right now and enqueue deps(A)&deps(B)&deps(C)
- We decide (adding a decision level) not to install C right now and enqueue deps(A)&deps(B)
- We decide (adding a decision level) not to install B right now and enqueue A
Now if we need to backtrack from our choice of A we hopefully still have a lot of common dependencies queued that we do not need to redo. While we have more backtracking levels, each backtracking level would be significantly cheaper, especially if you have cheap backtracking (which admittedly we do not have, yet anyway).
The caveat though is: It may be pretty expensive to find the common dependencies. We need to iterate over all dependency groups of A and see if they are in B, C, and D, so we have a complexity of roughly
#A * (#B+#C+#D)
Each dependency group we need to check i.e. is X|Y in B meanwhile has linear cost: We need to compare the memory content of two pointer arrays containing the list of possible versions that solve the dependency group. This means that X|Y and Y|X are different dependencies of course, but that is to be expected – they are. But any dependency of the same order will have the same memory layout.
So really the cost is roughly N^4. This isn’t nice.
You can apply various heuristics here on how to improve that, or you can even apply binary logic:
- Enqueue common dependencies of A|B|C|D
- Move into the left half, enqueue of A|B
- Again divide and conquer and select A.
This has a significant advantage in long lists of choices, and also in the common case, where the first solution should be the right one.
Or again, if you enqueue the package and a version restriction instead, you already get the common dependencies enqueued for the chosen package at least.
Pages
Recent Publications
- Managing Hidden Dependencies in OO Software: a Study Based on Open Source Projects
- Open Source Communities as Liminal Ecosystems
- Investigating developers' email discussions during decision-making in Python language evolution
- Developers, Quality Control and Download Volume in Open Source Software (OSS) Projects
More...FLOSS Project Planets
- Debug Academy: Evaluating Acquia storage limits - "Emergency upsize" notification
- GNU Taler news: Video interview with Mikolai Gütschow on payments for the Internet of Things
- The Drop Times: The Chief Who Drives and Is Driven by Drupal: A Talk with Dries Buytaert
- mark.ie: My LocalGov Drupal contributions for week-ending July 19th, 2024
- Enrico Zini: meson, includedir, and current directory
FLOSS Research
- Cailean Osborne: voices of the Open Source AI Definition
- The Open Source Initiative joins CMU in launching Open Forum for AI: A human-centered approach to AI development
- Open Source AI Definition – Weekly update July 15
- Mer Joyce: voices of the Open Source AI Definition
- Beyond SPDX: expanding licenses identified by ClearlyDefined