FLOSS Project Planets

Building on Part 1 of this PSF Grants Program Update, we are pleased to share updates to the Grants Workgroup (workgroup) Charter. We have outlined all the changes below in a chart, but there are a couple of changes that we’d like to highlight to grant applicants. These updates in particular will change how and when you apply, and hopefully reduce blockers to getting those applications in and ready for review. Because we are just sharing these updates, we are happy to be flexible on these changes but hope to see all applicants adhere to the changes starting around January 2025. 

• Increase overall process time frame to 8 weeks (formerly 6 weeks). We want to be realistic about how long the process takes and we know that going over our projection can cause pain for applicants. We hope to turn around applications in 6 weeks in most cases, but planning for the extra two weeks can make a big difference for everyone involved!
• Our application form requires that you set the event date out to 6 weeks in advance. We will wait to update that to 8 weeks in advance until January 2025.
• It’s important to note that this time frame begins only once all required information has been received, not exactly from the day the application is submitted. Make sure to check the email you provided on the application to see if the workgroup Chair has any questions regarding your request!
• Add a statement of support for accessibility services. In line with the PSF’s mission to support and facilitate the growth of a diverse community, we are explicitly stating in the charter that we will consider funding accessibility services. For established events (have 2 or more events in the past with more than 200 participants at the last event), we are open to considering accessibility-related requests such as live captioning, sign language interpretation, or certified child care.
• To review these types of requests, we will need sufficient documentation such as quotes, certifications, or any other relevant information.
• Add guidelines around program/schedule review. Previously undocumented, we were checking event programs/schedules to ensure a Python focus as well as a diversity of speakers. Because of event organizing time frames, we often received grant requests before the schedule was available. Moving forward we are accepting 1 of 3 options:
1. The program/schedule for the event
2. A tentative schedule or list of accepted speakers/sessions for the event
3. Programs from previous editions of the event if available, a link to the event’s call for proposals, which should state a required Python focus for the event as well as a statement in support of a diverse speaker group, and a description of the efforts that are being made to ensure a diversity of speakers.

Grants Workgroup Charter Updates   Update Summary Former Charter Projected Benefit Observations Establish fast-track grants: Grants that meet pre-approved criteria skip the review period with the workgroup and go straight to a vote Did not exist previously Resolutions reach applicants sooner, reduce load on workgroup Not many events meet the initial criteria we set to qualify for fast-track review, so this is mostly untested Establish workgroup participation criteria: workgroup members must participate in 60% of the votes to remain active Did not exist previously Resolutions reach applicants sooner, set out clear guidelines on the meaning of active participation, reduce load on Chair Reduction of workgroup membership to only active members has resulted in shorter voting periods by removing blockers to meeting quorum Increase $ amount for PSF Board review: Grant requests over 15K are reviewed by PSF Board Grant requests over 10K were reviewed by PSF Board Resolutions reach applicants sooner, reduces load on PSF Board to ensure they are focused on high level efforts Resolutions have reached applicants sooner, some reduction in load for PSF Board as we are still receiving applications over 15K Increase process timeframe: 8 week processing time from when all information has been received 6 week processing time from when all information has been received Improves community satisfaction and sets realistic expectations, reduces stress on workgroup & Chair We are just sharing this update so it has yet to be tested- come to our Grants Office Hour session to discuss it with us! Establish schedule for Grant review process: 10 day review period and 10 day voting period Did not exist previously Improve community satisfaction by ensuring requests are moving through the process promptly This has worked great to keep things moving as the workgroup has a set expectation of how long they have to comment Establish guideline for workgroup process: no discussion after the vote has begun Did not exist previously Improve community satisfaction by ensuring requests are moving hrough the process promptly While untested, this has set an expectation for the workgroup to comment during the review period Update voting mechanics: Votes will last for 10 days, a majority is reached, or when all voting members have voted, whichever comes first. For a proposal to be successful, it must have ayes in the majority totalling 30% of the WG Decisions were made by a majority rule (50%+1), with no time limit Improve community satisfaction by ensuring votes take place promptly, reduce stress on the workgroup and Chair if members are absent or unable to participate This has worked wonderfully! The Chair no longer has to track down votes. Paired with the participation guideline, voting periods no longer present a bottleneck Removed stated set budget: The annual budget is set by the PSF Board and is subject to change The previously documented budget was $120,000 (regularly exceeded) Removes an inaccurate description of the Grants Program budget and the need to update this line yearly A practical update, no observations to note Update workgroup officer roles: one Chair, one Vice Chair, one Appointed Board Director One Chair and two Vice Chairs Correct an unusual and discouraged practice of having two vice chairs and ensures PSF Board participation A practical update, no observations to note Add a statement of support for accessibility services: for mature events, consideration of granting funds for accessibility services Did not exist previously Establishes criteria for the workgroup and Board to consider accessibility-related requests We are just sharing this update so it has yet to be tested- come to our Grants Office Hour session to discuss it with us! Additional guidelines around grant reviews: tentative schedules OR previous schedules, CfP that shows a Python focus, as well as a description of the efforts being made to ensure a diversity of speakers Did not exist previously in documented form, though we checked for a program Improve community satisfaction with the process, remove delays in the grant review process This has been a great addition, and blockers for many applications have been removed! .table { display: block; overflow-y: hidden; overflow-x: auto; scroll-behavior: smooth; } .table thead { display: table-header-group; vertical-align: middle; border-color: inherit; color: inherit; background: darkcyan; } table, th, td { border: 1px solid black; border-collapse: collapse; } tr { display: table-row; vertical-align: inherit; border-color: inherit; } table th { padding: 16px; text-align: inherit; border-bottom: 1px solid black; color:white!important; } tbody { display: table-row-group; vertical-align: middle; border-color: inherit; } table:not(.tr-caption-container) { min-width: 100%; border-radius: 3px; }  What’s next?

Still on our Grants Program refresh to-do list is:

• Mapping Board-mandated priorities for the Grants Program to policy
• Charter adjustments as needed, based on the priority mapping
  
• Main documentation page re-write
• Budget template update
• Application form overhaul
• Transparency report for 2024
• Exploration and development of other resources that our grant applicants would find useful

Our community is ever-changing and growing, and we plan to be there every step of the way and continue crafting the Grants Program to serve Pythonistas worldwide. If you have questions or comments, we welcome and encourage you to join us at our monthly Grants Program Office Hour sessions on the PSF Discord.

Time has flown by since we received the community call last December for greater transparency and better processes around our Grants Program. PSF staff have produced a Grants Program Transparency Report and begun holding monthly Grants Program Office Hours. The PSF Board also invested in a third-party retrospective and launched a major refresh of all areas of our Grants program.

To provide the Grants Program more support, we assigned Marie Nordin, PSF Community Communications Manager, to support the Grants Program alongside Laura Graves, Senior Accountant. Marie has stepped into the Grants Workgroup Chair role to relieve Laura after 3+ years– thank you, Laura! Marie has been leading the initiatives and work related to the Grants Program in collaboration with Laura.

Behind the scenes, PSF staff has been working with the PSF Board and the Grants Workgroup (workgroup) to translate the feedback we’ve received and the analysis we’ve performed into action, starting with the Grants Workgroup Charter. A full breakdown of updates to the charter can be found in Part 2 of this update.

The PSF Board spent time on their recent retreat to explore priorities for the program going forward. We also ran a more thorough workgroup membership renewal process based on the updated charter to support quicker grant reviews and votes through active workgroup engagement. We’re excited to share refresh progress, updates, and plans for the future of the program later on in this post!

Something wonderful, bringing more changes

Meanwhile, the attention our Grants Program has received in the past year has resulted in something wonderful: we’re getting more requests than ever. Our call to historically underrepresented regions to request funds has been answered in some areas- and we are thrilled! For example, in the African region, we granted around 65K in 2023 and over 140K already this year! And, year to date in 2024 we have awarded more grant funding than we did in all of 2023. The other side of this coin presents us with a new issue– the budget for the program.

Up until this year, we’ve been able to grant at least partial funding to the majority of requests we’ve received while staying within our guidelines and maintaining a feasible annual budget. With more eligible requests incoming, every “yes” brings us closer to the ceiling of our grant budget. In addition to the increased quantity of requests, we are receiving requests for higher amounts. Inflation and the tech crunch have been hitting event organizers everywhere (this includes the PSF-produced PyCon US), and we are seeing that reflected in the number and size of the grant requests we are receiving.

Moving forward, with the increased quantity and amount of eligible grant requests, we will need to take steps to ensure we are balancing grant awards with sustainability for our Grants Program, and the Foundation overall. We know that the most important part of any changes to the Grants Program is awareness and two-way communications with the community. We aim to do that as early and transparently as we possibly can. That means we aren’t changing anything about how we award grants today or even next week– but within the next couple of months. Please keep an eye on our blog and social accounts (Mastodon, X, LinkedIn) for news about upcoming changes, and make sure to share this post with your fellow Python event and initiative organizers.

Grants Workgroup Charter update process

The purpose of the PSF Grants Workgroup (workgroup) is to review, approve, and deny grant funding proposals for Python conferences, training workshops, Meetups, development projects, and other related Python initiatives. The workgroup charter outlines processes, guidelines, and membership requirements for the workgroup. Small changes have been made to the charter over the years, but it’s been some time since any significant changes were implemented.
 
During the summer of 2024, Marie, workgroup chair (hi 👋 it’s me writing this!), and Laura worked on updates for the charter. The updates focused on how to make the Grants Program processes and guidelines work better for the workgroup, the PSF Board, and most especially, the community we serve.

After many hours of discussing pain points, running scenarios, exploring possible guidelines, and drafting the actual wording, Marie and Laura introduced proposed updates for the charter to the Board in July. After a month of review and 1:1 meetings with the PSF Board and workgroup members, the updated charter went to a vote with the PSF Board on August 14th and was approved unanimously.

The workgroup has been operating under its new charter for a couple of months. Before we shared broadly with the community, we wanted to make sure the updates didn’t cause unintended consequences, and we were ready to walk back anything that didn’t make sense. Turns out, our hard work paid off, and the updates have been mostly working as we hoped. We will continue to monitor the impact of the changes and make any adjustments in the next Charter update. Read up on the Grants Workgroup Charter updates in Part 2 of this blog post!

A gift from my brother. Coincidentally I’ve had John Carpenter’s “Halloween” echoing around my my head for weeks: I’ve been deconstructing it and trying to learn to play it.

I've used Drush - the Drupal shell - to interact with my Drupal applications on the command line since I started around 2008.

It's always been part of my Drush experience.

From installing Drupal and performing routine actions such as enabling modules and clearing caches to, in newer Drupal versions, performing migrations and generating Storybook stories.

Many projects I work on have custom Drush commands to perform tasks from the command line.

This week, I created a new Drupal 11 project for a client using the drupal/core-recommended package and initially forgot to install Drush so I could install Drupal.

I'm surprised Drush isn't in Drupal core or a dependency of the recommended package.

There is a basic Drupal CLI at core/scripts/drupal, but I wonder if we'll see a fully-featured CLI tool like Drush included with Drupal core, similar to Symfony's console or Laravel's artisan commands.

For me, including Drush would be an obvious choice.

Over 180 individual programs plus dozens of programmer libraries and feature plugins are released simultaneously as part of KDE Gear.

Today they all get new bugfix source releases with updated translations, including:

• neochat: Adjustments to make it work with the newly released libquotient 0.9 (Commit)
• kdevelop: MesonManager: remove test suites when a project is closing, (Commit, fixes bug #427157)
• kdevelop: Fix a qml crash building timeline with Qt 6.8 (Commit, fixes bug #495335)

Distro and app store packagers should update their application packages.

• 24.08 release notes for information on tarballs and known issues.
• Package download wiki page
• 24.08.3 source info page
• 24.08.3 full changelog

If you are paying close attention to the Drupal CMS roadmap, you may have noticed that our focus has mostly been on CMS features and the administrative user interface. Many people have asked: What about themes?

Drupal CMS will initially ship with Olivero, which is the default theme for Drupal core in the Standard profile. Of course, Experience Builder will completely change the way we build sites, and that includes support for design systems and single-directory components. In order to support this initially, the Starshot Demo Design System was developed (very quickly!) to show how design systems can be integrated with XB. We will also develop some components for Olivero so that Drupal CMS and eventually core have something to demo with XB.

Now, we are planning for what comes next. So we are seeking a strategic partner to collaborate on designing and implementing a comprehensive design system for our post-v1 integration with Experience Builder for Drupal CMS. 

The goal for this initiative is to create a modern and versatile design system that provides designers and front-end developers tools to accelerate their adoption of Drupal as their digital platform, by enabling them to easily adapt it to their own brand. This design system will enable content marketers to efficiently build landing pages and campaigns, allowing them to execute cohesive marketing strategies while maintaining the brand integrity.

Since it’s a big commitment for anyone, we are dividing the scope of work between design and implementation. We welcome applicants with expertise in one area who wish to specialize, as well as those who are equipped to handle the complete lifecycle of the design system, from initial design to full technical implementation and integration.

For more details, including information on how to apply, check out the full brief.

Interested partners should submit the following by 6 December, and we will announce the selected proposal(s) the week of 16 December. If you have questions before that, we’ll host a webinar the week of 19 November. You can also find us on Slack in #starshot or #experience-builder in the meantime.

We are looking forward to seeing your proposals!

Dear Debian community,

this is Bits from DPL for October. In addition to a summary of my recent activities, I aim to include newsworthy developments within Debian that might be of interest to the broader community. I believe this provides valuable insights and foster a sense of connection across our diverse projects. Also, I welcome your feedback on the format and focus of these Bits, as community input helps shape their value.

Ada Lovelace Day 2024

As outlined in my platform, I'm committed to increasing the diversity of Debian developers. I hope the recent article celebrating Ada Lovelace Day 2024–featuring interviews with women in Debian–will serve as an inspiring motivation for more women to join our community.

MiniDebConf Cambridge

This was my first time attending the MiniDebConf in Cambridge, hosted at the ARM building. I thoroughly enjoyed the welcoming atmosphere of both MiniDebCamp and MiniDebConf. It was wonderful to reconnect with people who hadn't made it to the last two DebConfs, and, as always, there was plenty of hacking, insightful discussions, and valuable learning.

If you missed the recent MiniDebConf, there's a great opportunity to attend the next one in Toulouse. It was recently decided to include a MiniDebCamp beforehand as well.

FTPmaster accepts MRs for DAK

At the recent MiniDebConf in Cambridge, I discussed potential enhancements for DAK to make life easier for both FTP Team members and developers. For those interested, the document "Hacking on DAK" provides guidance on setting up a local DAK instance and developing patches, which can be submitted as MRs.

As a perfectly random example of such improvements some older MR, "Add commands to accept/reject updates from a policy queue" might give you some inspiration.

At MiniDebConf, we compiled an initial list of features that could benefit both the FTP Team and the developer community. While I had preliminary discussions with the FTP Team about these items, not all ideas had consensus. I aim to open a detailed, public discussion to gather broader feedback and reach a consensus on which features to prioritize.

• Accept+Bug report

Sometimes, packages are rejected not because of DFSG-incompatible licenses but due to other issues that could be resolved within an existing package (as discussed in my DebConf23 BoF, "Chatting with ftpmasters"[1]). During the "Meet the ftpteam" BoF (Log/transcription of the BoF can be found here), for the moment until the MR gets accepted, a new option was proposed for FTP Team members reviewing packages in NEW:

Accept + Bug Report

This option would allow a package to enter Debian (in unstable or experimental) with an automatically filed RC bug report. The RC bug would prevent the package from migrating to testing until the issues are addressed. To ensure compatibility with the BTS, which only accepts bug reports for existing packages, a delayed job (24 hours post-acceptance) would file the bug.

• Binary name changes - for instance if done to experimental not via new

When binary package names change, currently the package must go through the NEW queue, which can delay the availability of updated libraries. Allowing such packages to bypass the queue could expedite this process. A configuration option to enable this bypass specifically for uploads to experimental may be useful, as it avoids requiring additional technical review for experimental uploads.

Previously, I believed the requirement for binary name changes to pass through NEW was due to a missing feature in DAK, possibly addressable via an MR. However, in discussions with the FTP Team, I learned this is a matter of team policy rather than technical limitation. I haven't found this policy documented, so it may be worth having a community discussion to clarify and reach consensus on how we want to handle binary name changes to get the MR sensibly designed.

• Remove dependency tree

When a developer requests the removal of a package – whether entirely or for specific architectures – RM bugs must be filed for the package itself as well as for each package depending on it. It would be beneficial if the dependency tree could be automatically resolved, allowing either:

a) the DAK removal tooling to remove the entire dependency tree after prompting the bug report author for confirmation, or b) the system to auto-generate corresponding bug reports for all packages in the dependency tree.

The latter option might be better suited for implementation in an MR for reportbug. However, given the possibility of large-scale removals (for example, targeting specific architectures), having appropriate tooling for this would be very beneficial.

In my opinion the proposed DAK enhancements aim to support both FTP Team members and uploading developers. I'd be very pleased if these ideas spark constructive discussion and inspire volunteers to start working on them--possibly even preparing to join the FTP Team.

On the topic of ftpmasters: an ongoing discussion with SPI lawyers is currently reviewing the non-US agreement established 22 years ago. Ideally, this review will lead to a streamlined workflow for ftpmasters, removing certain hurdles that were originally put in place due to legal requirements, which were updated in 2021.

Contacting teams

My outreach efforts to Debian teams have slowed somewhat recently. However, I want to emphasize that anyone from a packaging team is more than welcome to reach out to me directly. My outreach emails aren't following any specific orders--just my own somewhat naïve view of Debian, which I'm eager to make more informed.

Recently, I received two very informative responses: one from the Qt/KDE Team, which thoughtfully compiled input from several team members into a shared document. The other was from the Rust Team, where I received three quick, helpful replies–one of which included an invitation to their upcoming team meeting.

Interesting readings on our mailing lists

I consider the following threads on our mailing list some interesting reading and would like to add some comments.

Sensible languages for younger contributors

Though the discussion on debian-devel about programming languages took place in September, I recently caught up with it. I strongly believe Debian must continue evolving to stay relevant for the future.

"Everything must change, so that everything can stay the same." -- Giuseppe Tomasi di Lampedusa, The Leopard

I encourage constructive discussions on integrating programming languages in our toolchain that support this evolution.

Concerns regarding the "Open Source AI Definition"

A recent thread on the debian-project list discussed the "Open Source AI Definition". This topic will impact Debian in the future, and we need to reach an informed decision. I'd be glad to see more perspectives in the discussions−particularly on finding a sensible consensus, understanding how FTP Team members view their delegated role, and considering whether their delegation might need adjustments for clarity on this issue.

Kind regards Andreas.

Authored by Nadiia Nykolaichuk.

AI is shifting our perception of the impossible. It does things that past generations would have never imagined. Indeed, it has long since become a routine to ask AI assistants to play music, turn on the lights, or even order groceries. With the advance of generative AI, boosting content management through various AI-driven tasks has also become increasingly common.

Authored by Nadiia Nykolaichuk.

AI is shifting our perception of the impossible. It does things that past generations would have never imagined. Indeed, it has long since become a routine to ask AI assistants to play music, turn on the lights, or even order groceries. With the advance of generative AI, boosting content management through various AI-driven tasks has also become increasingly common.

LocalGov Drupal is the epitome of the principles of a Digital Public Good.

MARINA BAY, Singapore, 6 November, 2024—Drupal CMS, the groundbreaking package built on Drupal core with the marketer in mind, will launch on 15 January 2025. Conference attendees at DrupalCon Singapore 2024 will have the exclusive opportunity to be the first to learn more about Drupal CMS directly from Drupal’s founder, Dries Buytaert.

Learn how Drupal CMS will enable site builders without any Drupal experience to easily create a new site using their browser, marking one of the most significant launches in Drupal history.

Alongside the Drupal Association leadership team, Dries will unveil key features of Drupal CMS, making DrupalCon Singapore 2024 a can’t-miss event for anyone in the Open Source community. Occurring one month before the release of Drupal CMS, DrupalCon Singapore 2024 is an exclusive opportunity for attendees to join in the conversation surrounding Drupal CMS directly with its creators.

“The product strategy is for Drupal CMS to be the gold standard for no-code website building,” said Dries. “Our goal is to empower non-technical users like digital marketers, content creators, and site-builders to create exceptional digital experiences without requiring developers.”

DrupalCon Singapore 2024, 9-11 December 2024, is a premier gathering of Drupal and Open Source professionals. Over three days, the conference will showcase the latest Drupal trends, facilitate networking opportunities, and offer a platform for thought leadership in the Open Source landscape.

Key features of DrupalCon Singapore 2024 include:

• Keynotes, sessions, and panels: The Driesnote and Drupal CMS Panel are two highlights amongst a packed schedule of insightful sessions.
• Contribution Day: Contribution Day is where attendees grow and learn by helping to make Drupal even better. Giving back to the project is crucial in an Open Source community, as the Drupal project is developed by a community of people who work together to innovate the software.
• Birds of a Feather (BoFs): BoFs provide the perfect setting for connecting with like-minded attendees who share your interests.
• Splash Awards: Celebrate the work and creativity of the global Drupal community with this awards ceremony, which recognises outstanding projects built with Drupal.
• Networking Opportunities: Network with experts from around the globe who create ambitious digital experiences.

Register for DrupalCon Singapore 2024 at https://events.drupal.org/singapore2024 and join the next chapter in Drupal’s evolution!

In this tutorial, you’ll learn how to reset a pandas DataFrame index, the reasons why you might want to do this, and the problems that could occur if you don’t.

Before you start your learning journey, you should familiarize yourself with how to create a pandas DataFrame. Knowing the difference between a DataFrame and a pandas Series will also prove useful to you.

In addition, you may want to use the data analysis tool Jupyter Notebook as you work through the examples in this tutorial. Alternatively, JupyterLab will give you an enhanced notebook experience, but feel free to use any Python environment you wish.

As a starting point, you’ll need some data. To begin with, you’ll use the band_members.csv file included in the downloadable materials that you can access by clicking the link below:

Get Your Code: Click here to download the free sample code you’ll use to learn how to reset a pandas DataFrame index.

The table below describes the data from band_members.csv that you’ll begin with:

Column Name PyArrow Data Type Description first_name string First name of member last_name string Last name of member instrument string Main instrument played date_of_birth string Member’s date of birth

As you’ll see, the data has details of the members of the rock band The Beach Boys. Each row contains information about its various members both past and present.

Note: In case you’ve never heard of The Beach Boys, they’re an American rock band formed in the early 1960s.

Throughout this tutorial, you’ll be using the pandas library to allow you to work with DataFrames, as well as the newer PyArrow library. The PyArrow library provides pandas with its own optimized data types, which are faster and less memory-intensive than the traditional NumPy types that pandas uses by default.

If you’re working at the command line, you can install both pandas and pyarrow using the single command python -m pip install pandas pyarrow. If you’re working in a Jupyter Notebook, you should use !python -m pip install pandas pyarrow. Regardless, you should do this within a virtual environment to avoid clashes with the libraries you use in your global environment.

Once you have the libraries in place, it’s time to read your data into a DataFrame:

Python >>> import pandas as pd >>> beach_boys = pd.read_csv( ... "band_members.csv" ... ).convert_dtypes(dtype_backend="pyarrow") Copied!

First, you used import pandas to make the library available within your code. To construct the DataFrame and read it into the beach_boys variable, you used pandas’ read_csv() function, passing band_members.csv as the file to read. Finally, by passing dtype_backend="pyarrow" to .convert_dtypes() you convert all columns to pyarrow types.

If you want to verify that pyarrow data types are indeed being used, then beach_boys.dtypes will satisfy your curiosity:

Python >>> beach_boys.dtypes first_name string[pyarrow] last_name string[pyarrow] instrument string[pyarrow] date_of_birth string[pyarrow] dtype: object Copied!

As you can see, each data type contains [pyarrow] in its name.

If you wanted to analyze the date information thoroughly, then you would parse the date_of_birth column to make sure dates are read as a suitable pyarrow date type. This would allow you to analyze by specific days, months or years, and so on, as commonly found in pivot tables.

The date_of_birth column is not analyzed in this tutorial, so the string data type it’s being read as will do. Later on, you’ll get the chance to hone your skills with some exercises. The solutions include the date parsing code if you want to see how it’s done.

Now that the file has been loaded into a DataFrame, you’ll probably want to take a look at it:

Python >>> beach_boys first_name last_name instrument date_of_birth 0 Brian Wilson Bass 20-Jun-1942 1 Mike Love Saxophone 15-Mar-1941 2 Al Jardine Guitar 03-Sep-1942 3 Bruce Johnston Bass 27-Jun-1942 4 Carl Wilson Guitar 21-Dec-1946 5 Dennis Wilson Drums 04-Dec-1944 6 David Marks Guitar 22-Aug-1948 7 Ricky Fataar Drums 05-Sep-1952 8 Blondie Chaplin Guitar 07-Jul-1951 Copied!

DataFrames are two-dimensional data structures similar to spreadsheets or database tables. A pandas DataFrame can be considered a set of columns, with each column being a pandas Series. Each column also has a heading, which is the name property of the Series, and each row has a label, which is referred to as an element of its associated index object.

The DataFrame’s index is shown to the left of the DataFrame. It’s not part of the original band_members.csv source file, but is added as part of the DataFrame creation process. It’s this index object you’re learning to reset.

The index of a DataFrame is an additional column of labels that helps you identify rows. When used in combination with column headings, it allows you to access specific data within your DataFrame. The default index labels are a sequence of integers, but you can use strings to make them more meaningful. You can actually use any hashable type for your index, but integers, strings, and timestamps are the most common.

Note: Although indexes are certainly useful in pandas, an alternative to pandas is the new high-performance Polars library, which eliminates them in favor of row numbers. This may come as a surprise, but aside from being used for selecting rows or columns, indexes aren’t often used when analyzing DataFrames. Also, row numbers always remain sequential when rows are added or removed in a Polars DataFrame. This isn’t the case with indexes in pandas.

Read the full article at https://realpython.com/pandas-reset-index/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Right now, I am on a never ending quest that requires me to think of building a full fledge MVC controller : an anti-jira tracker that would favours HARD CHECKED facts over wishful thinking.

For this to begin, I am not really motivated in beginning with a full fledged MVC (Model View Controller) à la django because there is a lot of boilerplates and actions to do before a result. But, it has a lot of feature I want, including authentication, authorization and handling security.

For prototypes we normally flavours lightweight framework (à la flask), and CRUD.

CRUD approach is a factorisation of all framework in a single dynamic form that adapts itself to the model to generate HTML forms to input data, tabulate, REST endpoints and search them from the python class declaration and generate the database model. One language to rule them all : PYTHON. You can easily generate even the javascript to handle autocompletion on the generated view from python with enough talent.

But before using a CRUD framework, we need a cruder one, ugly, disgusting but useful for a human before building the REST APIs, writing the class in python, the HTML form, and the controlers.

I call this the crudest CRUD of them all.

Think hard at what you want when prototyping ...

• to write no CONTROLLERS ; flask documentation has a very verbose approach to exposing routes and writing them, writing controller for embasing and searching databases is boring
• to write the fewer HTML views possible, one and only onle would be great ;
• to avoid having to fiddle the many files reflecting separation of concerns : the lesser python files and class you touch the better;
• to avoid having to write SQL nor use an ORM (at least a verbose declarative one) ;
• show me your code and you can mesmerize and even fool me, however show me your data structure and I'll know everthing I have to know about your application : data structure should be under your nose in a readable fashion in the code;/
• to have AT LEAST one end point for inserting and searching so that curl can be used to begin automation and testing, preferably in a factorisable fashion;
• only one point of failure is accepted

Once we set these few condition we see whatever we do WE NEED a dynamic http server at the core. Python being the topic here, we are gonna do it in python.

What is the simplest dynamic web server in python ?

The reference implementation of wsgi that is the crudest wsgi server of them all : wsgiref. And you don't need to download it since it's provided in python stdlib.

First thing first, we are gonna had a default view so that we can serve an HTML static page with the list of the minimal HTML we need to interact with data : sets of input and forms.

Here, we stop. And we see that these forms are describing the data model.

Wouldn't it be nice if we could parse the HTML form easily with a tool from the standard library : html.parser and maybe deduce the database model and even more than fields coud add relationship, and well since we are dreaming : what about creating the tables on the fly from the form if they don't exists ?

The encoding of the relationship do require an hijack of convention where when the parser cross a name of the field in the form whatever_id it deduces it is a foreign key to table « whatever », column « id ».
Once this is done, we can parse the html, do some magick to match HTML input types to database types (adapter) and it's almost over. We can even dream of creating the database if it does not exists in a oneliner for sqlite.

We just need to throw away all the frugality of dependencies by the window and spoil our karma of « digital soberty » by adding the almighty sqlalchemy the crudest (but still heavy) ORM when it comes of the field of the introspective features of an ORM to map a database object to a python object in a clear consistent way. With this, just one function is needed in the controller to switch from embasing (POST method) and searching (GET).

Well, if the DOM is passed in the request. So of course I see the critics here :

• we can't pass the DOM in the request because the HTML form ignores the DOM
• You are not scared of error 415 (request too large) in the get method if you pass the DOM ?

That's where we obviously need two important tools : 1) javascript, 2) limitations.

Since we are human we would also like the form to be readable when served, because, well, human don't read the source and can't see the name attributes of the input. A tad of improving the raw html would be nice. It would also give consistency. It will also diminishes the required size of the formular to send. Here, javascript again is the right anwser. Fine, we serve the static page in the top of the controller. Let's use jquery to make it terse enough. Oh, if we have Javascript, wouldn't il be able to clone the part of the invented model tag inside every form so now we can pass the relevant part of the DOM to the controller ?

I think we have everything to write the crudest CRUD server of them all :D

Happy code reading : import multipart from wsgiref.simple_server import make_server from json import dumps from sqlalchemy import create_engine, MetaData, Table, Column from sqlalchemy import Integer, String, Float, Date, DateTime,UnicodeText, ForeignKey from html.parser import HTMLParser from sqlalchemy.ext.automap import automap_base from sqlalchemy.orm import Session from sqlalchemy import select from sqlalchemy import create_engine from sqlalchemy_utils import database_exists, create_database from urllib.parse import parse_qsl, urlparse engine = create_engine("postgresql://jul@192.168.1.32/pdca") if not database_exists(engine.url): create_database(engine.url) tables = dict() class HTMLtoData(HTMLParser): def __init__(self): global engine, tables self.cols = [] self.table = "" self.tables= [] self.engine= engine self.meta = MetaData() super().__init__() def handle_starttag(self, tag, attrs): attrs = dict(attrs) if tag == "input": if attrs.get("name") == "id": self.cols += [ Column('id', Integer, primary_key = True), ] return try: if attrs.get("name").endswith("_id"): table,_=attrs.get("name").split("_") self.cols += [ Column(attrs["name"], Integer, ForeignKey(table + ".id")) ] return except Exception as e: print(e) if attrs["type"] in ("email", "url", "phone", "text"): self.cols += [ Column(attrs["name"], UnicodeText ), ] if attrs["type"] == "number": if attrs["step"] == "any": self.cols+= [ Columns(attrs["name"], Float), ] else: self.cols+= [ Column(attrs["name"], Integer), ] if attrs["type"] == "date": self.cols += [ Column(attrs["name"], Date) ] if attrs["type"] == "datetime": self.cols += [ Column(attrs["name"], DateTime) ] if attrs["type"] == "time": self.cols += [ Column(attrs["name"], Time) ] if tag== "form": self.table = urlparse(attrs["action"]).path[1:] def handle_endtag(self, tag): if tag=="form": self.tables += [ Table(self.table, self.meta, *self.cols), ] tables[self.table] = self.tables[-1] self.table = "" self.cols = [] with engine.connect() as cnx: self.meta.create_all(engine) cnx.commit() html = """ <!doctype html> <html> <head> <script src="https://ajax.googleapis.com/ajax/libs/jquery/3.7.1/jquery.min.js"></script> <script> $(document).ready(function() { $("form").each((i,el) => { $(el).wrap("<fieldset>"+ el.action + "</fieldset>" ); $(el).append("<input type=submit value=insert formmethod=post ><input type=submit value=search formmethod=get />"); }); $("input:not([type=hidden],[type=submit])").each((i,el) => { $(el).before("<label>" + el.name+ "</label><br/>"); $(el).after("<br>"); }); }); </script> </head> <body> <form action=/user > <input type=number name=id /> <input type=text name=name /> <input type=email name=email > </form> <form action=/event > <input type=number name=id /> <input type=date name=date /> <input type=text name=text /> <input type=number name=user_id /> </form> </body> </html> """ router = dict({"" : lambda fo: html,}) def simple_app(environ, start_response): fo,fi=multipart.parse_form_data(environ) fo.update(**{ k: dict( name=fi.filename, content=fi.file.read().decode('utf-8', 'backslashreplace'), content_type=fi.content_type, ) for k,v in fi.items()}) table = route = environ["PATH_INFO"][1:] fo.update(**dict(parse_qsl(environ["QUERY_STRING"]))) start_response('200 OK', [('Content-type', 'text/html; charset=utf-8')]) try: HTMLtoData().feed(html) except KeyError: pass metadata = MetaData() metadata.reflect(bind=engine) Base = automap_base(metadata=metadata) Base.prepare() if route in tables.keys(): with Session(engine) as session: Item = getattr(Base.classes, table) if environ.get("REQUEST_METHOD", "GET") == "POST": new_item = Item(**{ k:v for k,v in fo.items() if v and not k.startswith("_")}) session.add(new_item) ret=session.commit() fo["insert_result"] = new_item.id if environ.get("REQUEST_METHOD") == "GET": result = [] for elt in session.execute( select(Item).filter_by(**{ k : v for k,v in fo.items() if v and not k.startswith("_")})).all(): result += [{ k.name:getattr(elt[0],k.name) for k in tables[table].columns}] fo["search_result"] = result return [ router.get(route,lambda fo:dumps(fo.dict, indent=4, default=str))(fo).encode() ] print("Crudest CRDU of them all on port 5000...") make_server('', 5000, simple_app).serve_forever()

The end of life of Drupal 7 is just around the corner and selecting the right migration partner is crucial for a smooth, cost-effective, and future-proof transition. Find out how 1xINTERNET and Pantheon’s unique solution can support your organisation!

The end of life of Drupal 7 is just around the corner and selecting the right migration partner is crucial for a smooth, cost-effective, and future-proof transition. Find out how 1xINTERNET and Pantheon’s unique solution can support your organisation!

November 2024 will be known as the time when killing peanut, a pet squirrel, by the New York State DEC swung the US presidential elections and shaped history forever.

The hundreds of millions of dollars spent on each side, the tireless campaigning by the candidates, the celebrity endorsements ... all made for an open race for months. Investments evened each other out.

But an OnlyFans producer showing people an overreaching, bureaucracy driven State raiding his home to confiscate a pet squirrel and kill it ... swung enough voters to decide the elections.

That is what we need to understand in times of instant worldwide publication and a mostly attention driven economy: Human fates, elections, economic cycles and wars can be decided by people killing squirrels.

RIP, peanut.

P.S.: Trump Media & Technology Group Corp. (DJT) stock is up 30% pre-market.

While I do have a Qt git build on my machine that I use for development, I usually only test individual applications and functionality but hardly ever run my full Plasma session on it. This means that for day-to-day use I typically only get to enjoy new Qt features once they have actually been released.

Proper modal dialogs under Wayland (note the darkened editor window) thanks to XDG Dialog and the new Qt 6.8

One feature I talked about in the very last issue of “On the road to Plasma 6” is a nice API for XDG Foreign. To recap: it’s a Wayland protocol that lets an application export a window to another one so it can can attach a window to it. For example, the XDG Desktop Portal wants to attach the “Open File” dialog as if it were coming from the application that requested it.

Of course we don’t want to write low-level Wayland code and instead have an easy to use API for it. The KWindowSystem::setMainWindow function does just that: hand in a window and the token you received from the other application (created through KWaylandExtras::exportWindow) and it takes care of everything else. Presumably, you want to set the parent window before showing your dialog to make absolutely sure it’s set up properly.

However, Qt did not have an API to tell us when the underlying XDG Toplevel (think: a regular desktop-y window with a title bar and what not) had been created. We were only told when the basic wl_surface was created, which was too early, or the window was exposed/shown, at which point it was already flashing up in the user’s task bar. Hence, I added a new QWaylandWindow::surfaceRoleCreated (and corresponding surfaceRoleDestroyed) signal. Utilizing that, the aforementioned KWindowSystem API now works perfectly.

Another major addition to Qt Wayland that I have been looking forward to very much is support for the XDG Dialog protocol. While a window could have always had a parent (e.g. a popup menu or settings dialog parented to the application’s main window), there was no concept of a “modal” dialog. Therefore, we did not support the “dim parent” effect under Wayland that darkens a window to indicate it cannot be interacted with. More importantly, KWin couldn’t take it into account for its focus handling either. It happily let you focus a blocked window but the application would then just ignore your input.

There’s only one Dolphin running here!

This was most noticeable for me when Alt+Tab’ing back and forth, for example using the “Open File” dialog in one application and then trying to switch to the other to verify where the file was actually located. Instead of cycling between the file dialog and the other application, it would alternate between the file dialog and the blocked main window.

Sadly, even when I upgraded to Qt 6.8 the situation didn’t improve. I noticed that Alt+Tab actually showed the dialog twice. This looked like a bug and sure enough comparing it to the Plasma 5.27 LTS session on my other computer proved that it used to work at some point. At first I didn’t spot anything obvious until I noticed a small typo that must have slipped in during some major refactoring. Instead of not including the main window when it had a modal child, it included the modal child once again! Sure enough, adding an exclamation mark (the logical NOT operator in C++) did the trick.

If you want to support more good people such as myself, consider donating to the KDE End of Year Fundraiser!

Low-code + Decoupled Drupal: The Power of Custom Elements 3.0 no-code.png wolfgang.ziegler Wed, 11/06/2024 - 04:54 Custom Elements UI empowers frontend developers to configure Drupal's data processing for API responses, enabling precise tailoring of output.

Kamal offers zero-downtime deploys, rolling restarts, asset bridging, remote builds, accessory service management, and everything else you need to deploy and manage your web app in production with Docker. Originally built for Rails apps, Kamal will work with any type of web app that can be containerized. We dig into Kamal, how it works, and how you could use it on your next project.

This one required a few other features to be implemented first, so let’s jump right in.

Matching reference luminances

A big part of what a desktop compositor needs to get right with HDR content is to show SDR and HDR content properly side by side. KWin 6.0 added an SDR brightness slider for that purpose, but that’s only half the equation - what about the brightness of HDR content?

When we say “HDR”, usually that refers to a colorspace with the rec.2020 primaries and the perceptual quantizer (PQ) transfer function. A transfer function describes how to calculate a real brightness value from the “electrical” signal encoded in the content - PQ specifically has encoded values from 0 to 1 and brightness values from 0 to 10000 nits. For reference, your typical office monitor does around 300 or 400 nits at maximum brightness setting, and many newer phones can go a bit above 1000 nits.

Now if we want to show HDR content on an HDR screen, the most straight forward thing to do would be to just calculate the brightness values, write them to the screen and be done with it, right? That’s what KWin did up to Plasma 6.1, but it’s far from ideal. Even if your display can show the full range of requested brightness values, you might want to adjust the brightness to match your environment - be it brighter or darker than the room the content was optimized for - and when there’s SDR things in HDR content, like subtitles in a video, that should ideally match other SDR content on the screen as well.

Luckily, there is a preexisting relationship between HDR and SDR that we can use: The reference luminance. It defines how bright SDR white is - which is why another name for it is simply “SDR white”.

As we want to keep the brightness slider working, we won’t map SDR content to the reference luminance of any HDR transfer function though, but instead we map both SDR and HDR content to the SDR brightness setting. If we have an HDR video that uses the PQ transfer function, that reference luminance is 203 nits. If your SDR brightness setting is at 406 nits, KWin will just multiply the brightness of the HDR video with a factor of 2.

This doesn’t only mean that we can make SDR and HDR content fit together nicely on HDR screens, but it also means we now know what to do when we have HDR content on an SDR screen: We map the reference luminance from the video to SDR white on the screen. That’s of course not enough to make it look nice though…

Tone mapping

Especially with HDR presented on an SDR screen, but also on many HDR screens, it will happen that the content brightness exceeds the display capabilities. To handle this, starting with Plasma 6.2, whenever the HDR metadata of the content says it’s brighter than the display can go, KWin will apply tone mapping.

Doing this tone mapping in RGB can result in changing the content quite badly though. Let’s take a look by using the most simple “tone mapping” function there is, clipping. It just limits the red, green and blue values separately to the brightness that the screen can show.

If we have a pixel with the value [2.0, 0.0, 2.0] and a maximum brightness of 1.0, that gets mapped to [1.0, 0.0, 1.0] - which is the same purple, just in darker. But if the pixel has the values [2.0, 0.0, 1.0], then that gets mapped to [1.0, 0.0, 1.0], even though the source color was significantly more red!

To fix that, KWin’s tone mapping uses ICtCp. This is a color space developed by Dolby, in which the perceived brightness (aka Intensity) is separated from the chroma components (Ct = blue-yellow, Cp = red-green), which is perfect for tone mapping. KWin’s shaders thus transform the RGB content to ICtCp, apply a brightness mapping function to only the intensity component, and then convert back to RGB.

The result of that algorithm looks like this:

RGB clipping KWin 6.2’s tone mapping MPV’s tone mapping

As you can see, there’s still some color changes going on in comparison to MPV’s algorithm; this is partially because the tone mapping curve still needs some more adjustments, and partially because we also still need to do similar mapping for colors that the screen can’t actually show. It’s already a large improvement though, and does better than the built-in tone mapping functionality in many HDR screens.

When tone mapping HDR content on SDR screens, we always end up reducing the brightness of the overall image, so that we have some brightness values to map the really bright highlights in the video to - otherwise everything just slightly over the reference luminance would look like an overexposed blob of color, as you can see in the “RGB clipping” image. There are ways around that though…

HDR on SDR laptop displays

To explain the reasoning behind this, it helps to first have a look at what even makes a display “HDR”. In many cases it’s just marketing nonsense, a label that’s put on displays to make them seem more fancy and desirable, but in others there’s an actual tangible benefit to it.

Let’s take OLED displays as an example, as it’s considered one of the display technologies where HDR really shines. When you drive an OLED at high brightness levels, it becomes quite inefficient, it draws a lot of power and generates a lot of heat. Both of these things can only be dealt with to a limited degree, so OLED displays can generally only be used with relatively low average brightness levels. They can go a lot brighter than the average in a small part of the screen though, and that’s why they benefit so much from HDR - you can show a scene that’s on average only 200 nits bright, with the sky in the image going up to 300 nits, the sun going up to 1000 nits and the ground only doing 150 nits.

Now let’s compare that to SDR laptop displays. In the case of most LCDs, you have a single backlight LED for the whole screen, and when you move the brightness slider, the power the backlight is driven at is changed. So there’s no way to make parts of the screen brighter than the rest on a hardware level… but that doesn’t mean there isn’t a way to do it in software!

When we want to show HDR content and the brightness slider is below 100%, KWin increases the backlight level to get a peak brightness that matches the relative peak brightness of that content (as far as that’s possible). At the same time it changes the colorspace description on the output to match that change: While the reference luminance stays the same, the maximum luminance of the transfer function gets increased in proportion to the increase in backlight brightness.

The results is that SDR white gets mapped to a reduced RGB value, which is at least supposed to exactly counteract the increase of brightness that we’re applying with the backlight, while HDR content that goes beyond the reference luminance gets to use the full brightness range.

Increasing the backlight power of course doesn’t come without downsides; black levels and power usage both get increased, so this is only ever active if there’s HDR content on the screen with valid HDR metadata that signals brightness levels going beyond the reference luminance.

As always, capturing HDR content with a phone camera is quite difficult, but I think you can at least sort of see the effect:

without backlight adjustment with backlight adjustment

This feature has been merged into KWin’s git master branch and will be available on all laptop displays starting with Plasma 6.3. I really recommend trying it for yourself once it reaches your distribution!