Feeds

Krita Monthly Update – Edition 15

Planet KDE - Mon, 2024-05-06 20:00

It is time for the monthly news update brought to you by the Krita-promo team. Let us take a look at the highlights of krita community and development for this month.

Development report

  • Our users on chromebooks faced a nasty bug which crashed krita on startup. So we made a 5.2.2.1 hotfix release for Android Play Store only to fix this bug. It also contains other fixes from the stable branch, but be warned there is a known crash regression with importing audio.

  • A proper 5.2.3 release for all supported platforms will be made as soon as possible, hopefully in the next few weeks.

  • At the time of writing, nightly builds for macOS are still blocked by a signing-related issue. Once that is resolved, automated builds for all supported platforms will be up and running again. That is the culmination of months of work by lead developer Dmitry Kazakov, together with macOS developer Iván Yossi, Android developer Sharaf Zaman, Windows contributor Simon Ra, and others, in a refactor of Krita’s build system.

  • Feature Request: Palette in Toolbar has been marked “solved” by freyalupen’s most recently merged code. Add docker box toolbar widget allows the user to add any docker to the toolbar in a temporary popup widget similar to the “choose brush preset” one in Painter’s Tools.

  • A problem with certain RBGA brushes has been solved and will be part of the next release. Users were experiencing lagging and freezing when accessing these brushes. The thread makes an interesting read as it’s a “live” look at an issue being revealed and it shows how helpful it is when users conduct testing. You can read the thread here.

  • Ken_Lo has been accepted as a student for Google Summer of Code, to work on pixel perfect hand-drawn lines.

  • In addition to various recorder related fixes by @freyalupen, the FFmpeg profiles in the recorder docker are improved by @Ralek. We congratulate @Ralek on their first contribution to Krita.

  • When entering canvas-only mode, the document used to jump abruptly and reposition itself. @YRH helped in solving this issue

  • Deif_Lou has improved performance of the fill tool making it faster.

  • Ken Lo added an option in the settings to pick default export file type.

  • Grum999 has looked into improving Krita’s API for python plugins and as a start, chose to implement a scratchpad API that adds functionality to the scratchpad.

  • Emir Sari sent patches to help Krita build on Haiku OS.

Community report March 2024 Monthly Art Challenge

The April Monthly Art Challenge, Animal Curiosity, inspired submissions from 26 artists. @jimplex was voted the winner with this creative piece: Firefly by jimplex

The theme for the May 2024 challenge is “reflection.” You can get all the details here. We already have some ideas and pre-work flying around in the discussion and WIP thread. Have a look – something might inspire your creativity.

Featured artwork

Krita-Artists members nominated 9 images for the featured artwork banner. When the mid-month poll ended, these are the 5 that won a place on the banner. All 5 will be entered into the Best of Krita-Artists 2024 competition next January.

Cabin in the woods-RH by Rohit Hela

Detailed Portrait by denjay5

Nier Automata by IvanGilbertt

Alien Senator by DavB

My uni project by smollbirb

Nominations for the April/May poll are open until May 11, 2024.

Noteworthy plugin

Blender-Krita link plugin for texture editing by heisenshark

This plugin has a fresh update that the author describes as a “big overhaul of how the plugin works.” Check out the thread on Krita-artists.org here.

Tutorial of the month

Krita’s newest tutorial by Ramon Miranda features an interview with Rakurri, the creator of Rakurri’s brush pack containing more than 200 brushes made just for Krita. Ramon demonstrates his favorite ones such Glow FX, Liquid Bristle and the vegetation brushes.

Notable changes in code

This section has been compiled by freyalupen.. Apr 3 - May 2, 2024

Stable branch (5.2.2+):

Bugfixes:

Nightly build regression bugfixes:

  • [Layer Stack] Fix wrong layer being active on opening document. In the case of single-layer documents, no layer was active, which caused crashes under some circumstances. (BUG:480718) (merge request, Dmitry Kazakov)
Unstable branch (5.3.0-prealpha):

Features:

  • Toolbars, Shortcuts Add Docker Box action that shows a docker in a temporary box, which can be added to a toolbar or assigned to shortcut. (merge request, Freya Lupen)
  • Canvas Input Shortcuts Add new Tool Invocation action, "Activate with Other Color". This can be bound to a key+mousebutton, where holding those keys will cause, for instance, the Freehand Brush to paint with the background instead of foreground color. (merge request, ziplantil ..)

Bugfixes:

Nightly build regression bugfixes:

These changes are made available for testing in the following Nightly builds:

Like what we are doing? Help support us

Krita is a free and open source project. Please consider supporting the project with donations or by buying training videos or the artbook! With your support, we can keep the core team working on Krita full-time.

Donate Buy something
Categories: FLOSS Project Planets

Horizontal Digital Blog: Try this one weird trick with the Migrate API

Planet Drupal - Mon, 2024-05-06 18:00
One of the key concepts of the Drupal Migrate API is the so-called process pipeline, in which we pass a value that is transformed by a series of process plugins. From time to time we find ourselves in the middle of a process pipeline wishing we could easily reference the current value in the process pipeline. I even created an issue on Drupal.org asking for this feature. As it turns out, the feature already exists! That is, as long as you know this one weird trick...
Categories: FLOSS Project Planets

Talking Drupal: Talking Drupal #449 - Agile Methodologies

Planet Drupal - Mon, 2024-05-06 14:00

Today we are talking about Agile Methodologies, How to pick the best one, and why they matter with guest Chris Wells. We’ll also cover CKEditor Text Transformation / AutoCorrect as our module of the week.

For show notes visit: www.talkingDrupal.com/449

Topics
  • Drupal FL Camp talk
  • Fundamentals of Agile
  • How do you square long term planning
  • What is Redfin Solutions's preferred methodology
  • What is Crystal Agile Methodology
  • Do other methodologies have web specific versions
  • Would you agree that large companies can use different agile methodologies
  • Have you ever used Scrumban
  • Listener Question: Shivan xamount:: Story points are usually equated to fibonacci numbers. These are not supposed to correlate to hours, what do you think about that?
Resources Guests

Chris Wells - chrisfromredfin.dev chrisfromredfin

Hosts

Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Matthew Grasmick - grasmash

MOTW Correspondent

Martin Anderson-Clutz - mandclu

  • Brief description:
    • Have you ever wanted CKEditor to autocorrect symbols like the copyright mark, the “not equals” sign, and fractions, from their text equivalents? There’s a module for that
  • Module name/project name:
  • Brief history
    • How old: created in Mar 2024 by Gedvan Dias of Redfin Solutions
    • Versions available: 1.0.0-alpha1, which works with CKeditor 4 on Drupal 8, and 2.0.0-alpha1, which works with CKEditor 5 on Drupal 9 and 10
  • Maintainership
    • Actively maintained, was released just a few weeks ago
    • Not much documentation of its own, but the module leverages CKEditor’s Automatic text transformation, which has a fair bit of documentation on CKEditor.com
    • Number of open issues: only 1 open issues, which is the Project Update Bot’s automatically-created Drupal 11 compatibility issue
  • Usage stats:
    • 8 sites
  • Module features and usage
    • By default the module enables four categories of transformations: 'symbols', 'mathematical', 'typography', and 'quotes'
    • You can override the module’s plugin if you want a different set enabled, but the module also provides a hook you can use to alter the active sets or define custom transformations, similar to using emojis in Slack, for example
Categories: FLOSS Project Planets

Drupal Association blog: Drupal lead Dries Buytaert announces a completely new Drupal CMS 23 years after its creation

Planet Drupal - Mon, 2024-05-06 12:52

PORTLAND, Ore., 6 May 2024—Twenty-three years after creating Drupal as a university student and hundreds of thousands of websites later, Dries Buytaert announced today that a new version of Drupal will launch at the end of 2024. Drupal is an Open Source CMS that is foundational to a great digital experience platform. Its reliable, highly secure, and flexible tools build the versatile, structured content needed to create dynamic web experiences.

This new version of Drupal will incorporate the best of the 50,000+ modules created over the past decade into a curated, out-of-the-box experience for organizations wishing to build powerful websites quickly.

“We built this amazing platform to power the most robust digital experiences. And now we will make it more accessible to non-developers,” said Dries. “Drupal Starshot is an initiative that will deliver this new version of Drupal within eight months.”

“The Drupal Association is excited to support the Drupal Starshot initiative and to begin marketing the new version of Drupal as the first, best stop for those interested in understanding what Drupal can do,” said Owen Lansbury, Chair of the Drupal Association’s board of directors.

On 6 May, founder and project lead Dries Buytaert gave an inspiring keynote—also known as the Driesnote—introducing this completely new version of Drupal: Drupal Starshot.

Dries described how, much like the race to space in the 1960s, Drupal is also in a race. The web is moving forward, with or without Drupal. Drupal has a long history of being a leader in the Open Web, but it needs its “Moonshot” moment. Dries reiterated how the future of Drupal’s success will come from broadening its usability to a wider audience. The way to do this, Dries said, is to open Drupal’s powerful tools to non-developers.

What is Drupal Starshot, and how does it differ from the traditional version of Drupal? Drupal Starshot will leverage Drupal Core but have a different governance model to move fast, allowing for more innovation more quickly. 

After Drupal Starshot is introduced, when someone visits the Drupal.org download page, both traditional Drupal Core and Drupal Starshot will be available (under a different name, still to be determined). When Drupal Starshot is selected, it will automatically download the features that the user wants for their use case, making it easier for new users to try and test out Drupal, all from right in their browser. Drupal Core will still be the fundamental building block of Drupal Starshot and can still be used independently from Drupal Starshot for custom builds.

How will this new Drupal be different?

The Drupal that exists today, known as “Drupal Core,” will continue to exist and will be maintained by core maintainers. The Drupal Starshot initiative will introduce a new version of Drupal with a fully featured out-of-box experience.

Features that Drupal Starshot will include are:

  • Next generation page builder
  • Project Browser + Recipes
  • Automatic updates
  • Key contributed modules
  • Easy configuration
  • Default content
  • And possibly more!

Drupal community members who are interested in contributing to the development of Drupal Starshot can submit their interest via this interest form or join Dries at several Birds of a Feather sessions happening during DrupalCon Portland.

Watch the full Driesnote on the Drupal Association YouTube Channel. 

About DrupalCon

This year, DrupalCon North America is a four-day conference held in Portland, Oregon, from 6-9 May. Over 1,300 professionals and Drupal users collaborate on the project for a week. The Drupal Association is a non-profit organization that caters to the needs of Drupal and its worldwide community. It focuses on the growth of the Drupal community and supports the project’s vision to create a safe, secure, and Open Web for everyone. 

About Drupal and the Drupal Association

Drupal is a powerful open-source content management system for everyone, from small nonprofits to enterprises. It is used by millions of people and organizations worldwide, made possible by a community of 100,000-plus contributors and enabling more than 1.3 million users on Drupal.org. The Drupal Association is a non-profit organization dedicated to accelerating the Drupal software project, fostering the community, and supporting its growth.

Categories: FLOSS Project Planets

Open Source AI Definition – Weekly update May 6

Open Source Initiative - Mon, 2024-05-06 12:02
Definition validation: Seeking volunteers

The process has entered a new phase: We are now seeking volunteers to validate the Open Source AI Definition, using it to review existing AI systems. The objective of the phase is to confirm that the Definition works as intended and understand where it fails.  

  • A spreadsheet is given where you locate and link to the license, research paper, or other document that grants rights or provides information for each required component. 
  • Systems include, but are not limited to:
    • Arctic
    • BLOOM
    • Falcon
    • Grok
    • Llama 2
    • Mistral
    • OLMo
    • OpenCV
    • Phi-2
    • Pythia
    • T5
  • To volunteer by May 20th, please contact Mer on the forum
Summary of comments received on the Definition draft
  • Grammatical and wording corrections 
    • Some minor grammatical suggestions were made. These change and order the layout slightly differently, though the overall message remains. 
    • One user suggested to explain what Open Source is under the “preamble” and “Why we need open source AI”. Instead of speaking about why Open Source is important, the section should rather be an introduction to what it is and why it matters for AI.
    • Under “Preferred form to make modifications to machine-learning systems” and “data information”, clarification is needed regarding “the training data set used”. It is not clear whether this means that all training data must be open source for the whole model to be.
      • Stefano Maffulli added here that the intention is to know what dataset was used, not to necessarily have it made available, and that it indeed seems to need clarification
  • Technical points
    • Under “Preferred form to make modifications to machine-learning systems” the release of checkpoints is mentioned as an example of required components, under “model parameters”. An objection was raised, arguing that this poses an unnecessary burden: It’d be like requiring that for software to be Open Source, it should include past versions of the program.
      • Maffulli reiterated that this was merely an example but that this might need to be a submission to the FAQ page
    • “Preferred form to make modifications to machine-learning systems” and “data information”, a “skilled person” is mentioned in the context of requiring sufficient information about the training data used to create a model. Question regarding why skill has to do with acquiring data
      • Clarification was given by Maffulli, pointing out that this is in the context of getting information about the data so that a “skilled person” can use, study, share and modify the AI system.
      • A user suggested that this confusion can be solved by changing the context of the wording “a skilled person can recreate”. From “using the same or similar data” to “if able to gain access to the same or similar data”.
      • A user points out that “skilled person” as a legal term used in patent law might not be appropriate as it has different legal connotations and precedence in different countries.
  • Discussion on why specifically we focus on machine learning (ML) as an AI system
    • A question was raised regarding why we explicitly mention ML systems under “preferred form to make modification to an ML system” and subsequently the “checklist”, pointing out that not all AI systems are ML.
      • Maffulli replied that we address ML as they need special and urgent attention as rule-based AI systems can fit under the open source definition. This needs to be addressed in the FAQ
Town hall announcement 
  • The 9th town hall meeting was held on the 3d of May. Access the recording here if you missed it!
Categories: FLOSS Research

Drupalize.Me: DrupalCon Portland 2024: Issue Queue Initiatives

Planet Drupal - Mon, 2024-05-06 12:00
DrupalCon Portland 2024: Issue Queue Initiatives

This Wednesday, May 8, I'm speaking at DrupalCon Portland 2024 as part of the Drupal Project Initiatives Keynote. The keynote is kicking off Contribution Day on Wednesday first thing in the morning. I'll be highlighting initiatives and programs that are helping people contribute in a strategic way, and as a result, increasing throughput in the core issue queue. Throughput is the rate that a project’s issues are resolved and committed. And it’s one way to gauge the health of an open source project like Drupal.

Check out these resources to learn more about the initiative and programs I highlight in this presentation.

Amber Matz Mon, 05/06/2024 - 11:00
Categories: FLOSS Project Planets

Thomas Lange: Removing tens of thousands of web pages

Planet Debian - Mon, 2024-05-06 10:58

In January I've removed tens of thousands of web pages on www.debian.org. Have you noticed it?

In the past

From 1997 onwards, we had web pages for security announcements. We had to manually prepare a .data and a .wml file which then generated a web page for each security announcement (DSA or DLA). We have listed the 6 most recent messages in a short list that was created from these files. Most of the work that went into the Debian web pages was creating these files.

Our search engine often listed the pages with security announcements instead of a more relevant web page for a particular topic.

Preparation

At DebConf Kosovo (2022) I started with a proof of concept and wrote a script, that generates this list without using the .data/.wml files in the Git repository, but instead reading the primary sources of security information[1]. This new list now includes links to the security tracker and the email of the announcement.

Following web pages and scripts were also using these .data and .wml files:

  • OVAL files
  • RSS feeds for security announcements (and LTS)
  • Apache config file for mapping URLs from dsa-NNN to YEAR/dsa-NNN
  • A huge list of crossreferences between DSA and CVE numbers

Before I could remove all the security web pages, I had to adjust the scripts, that create the above information.

When I looked at the OVAL files and the apache logs of our web server, I saw that more than 99% of the web traffic was generated by these XML files (134TB of 135TB total in two weeks). They were not compressed and were around 50MB in size. With the help of Carsten Schönert we managed to modify the python scripts that generate this OVAL file without using the .data/.wml files and now we only provide bzip2 compressed XML files[2].

The RSS feeds are created by the new Perl script which reads the DSA/DLA list the security tracker and determines the URL of the email of all entries. This script also generates the list of the most recent DSA/DLA entries. Currently we show the last 350 entries which covers more than the last year and includes links to the announcement email and the security tracker.

The huge list of crossreferences is not needed any more, since the mapping of CVE to DSA is already included in the DSA list[3] of the security tracker.

The amount of translations of the DSA/DLA was very different. French translations were almost all done, but all other languages did translations for a couple of months or years only. E.g. in 2022, Italian had 2 translations, Russian 15, Danish 212, French and English each 279. But from 2023 on only French translations were made. By generating the list of DSA/DLA we lost the ability to translate these web pages, but since these announcements are made of simple, identical sentences it is easy to use an automatic translation service if needed.

Now the translation statistics of all web pages are more accurate. Instead of 12200 pages that need to be translated (including all these old DSA/DLA) there are now only 2500 pages to translate[4]. Languages that had a lot of old translations of DSA/DLA lost some percentage but languages that are doing translations of newer web pages won in the statistics of how many pages are translated. Examples:

Before German (de) 3501 28.5% Italian (it) 1005 8.2% Danish (da) 6336 51.7% After German (de) 1486 59.0% Italian (it) 909 36.1% Danish (da) 982 39.0% Cleanup of all the security web pages

Finally in January, I could remove all web pages of the security announcements in one git commit[5]. Using several git rm -rf commands this commit removed 54335 files, including around 9650 DSA/DLA data files, 44189 wml files, nearly 500 Makefiles.

Outcome

No more manual work is needed for the security team and we now have direct links from a DSA-NNN/DLA-NNN to the email in our mailing list archive. This was not possible before. The search results became more accurate.

But we still host a lot of other old content on the Debian web pages which may be removed in the future.

[1] https://www.debian.org/security/#infos

[2] https://www.debian.org/security/oval/

[3] https://salsa.debian.org/security-tracker-team/security-tracker/-/raw/master/data/DSA/list

[4] https://www.debian.org/devel/website/stats

[5] https://salsa.debian.org/webmaster-team/webwml/-/commit/2aa73ff15bfc4eb2afd85c

Categories: FLOSS Project Planets

Real Python: Python News: What's New From April 2024

Planet Python - Mon, 2024-05-06 10:00

In April 2024, Python’s core development team released versions 3.13.0a6 and 3.12.3 of the language! The former received several exciting features, improvements, and optimizations, while the latter got more than 300 commits for security improvements and bug fixes.

The 3.13.0a6 release is the last alpha release. In the first half of May, the code will be frozen and won’t accept new features. Note that 3.13.0a6 is a pre-release, so you shouldn’t use it for production environments. However, it provides a great way to try out some new and exciting language features.

There is also great news about PyCon US 2024, which opened its call for volunteers.

Let’s dive into the most exciting Python news from April 2024!

Python 3.13.0 Alpha 6 and 3.12.3 Arrive

This April, Python released its sixth alpha preview release, 3.13.0a6. This version is the last alpha release, as Python 3.13 will enter the beta phase on May 7. Once in beta, it won’t accept any new features.

Python 3.13 brings the following new features:

Meanwhile, the standard library comes with these new features:

  • The dbm module has a new dbm.sqlite3 backend for creating new files.
  • PEP 594 scheduled removals of many deprecated modules: aifc, audioop, chunk, cgi, cgitb, crypt, imghdr, mailcap, msilib, nis, nntplib, ossaudiodev, pipes, sndhdr, spwd, sunau, telnetlib, uu, xdrlib, lib2to3.
  • Many deprecated classes, functions, and methods (dead batteries) were removed.
  • New deprecations appeared, and many of them were scheduled for removal in Python 3.15 or 3.16.

For a detailed list of changes, additions, and removals, you can check out the Changelog document. The next pre-release of Python 3.13 will be 3.13.0b1, which is currently scheduled for May 7.

Read the full article at https://realpython.com/python-news-april-2024/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Mike Driscoll: How to Read and Write Parquet Files with Python

Planet Python - Mon, 2024-05-06 09:57

Apache Parquet files are a popular columnar storage format used by data scientists and anyone using the Hadoop ecosystem. It was developed to be very efficient in terms of compression and encoding. Check out their documentation if you want to know all the details about how Parquet files work.

You can read and write Parquet files with Python using the pyarrow package.

Let’s learn how that works now!

Installing pyarrow

The first step is to make sure you have everything you need. In addition to the Python programming language, you will also need pyarrow and the pandas package. You will use pandas because it is another Python package that uses columns as a data format and works well with Parquet files.

You can use pip to install both of these packages. Open up your terminal and run the following command:

python -m pip install pyarrow pandas

If you use Anaconda, you’ll want to install pyarrow using this command instead.

conda install -c conda-forge pyarrow

Anaconda should already include pandas, but if not, you can use the same command above by replacing pyarrow with pandas.

Now that you have pyarrow and pandas installed, you can use it to read and write Parquet files!

Writing Parquet Files with Python

Writing Parquet files with Python is pretty straightforward. The code to turn a pandas DataFrame into a Parquet file is about ten lines.

Open up your favorite Python IDE or text editor and create a new file. You can name it something like parquet_file_writer.pyor use some other descriptive name. Then enter the following code:

import pandas as pd import pyarrow as pa import pyarrow.parquet as pq def write_parquet(df: pd.DataFrame, filename: str) -> None: table = pa.Table.from_pandas(df) pq.write_table(table, filename) if __name__ == "__main__": data = {"Languages": ["Python", "Ruby", "C++"], "Users": [10000, 5000, 8000], "Dynamic": [True, True, False], } df = pd.DataFrame(data=data, index=list(range(1, 4))) write_parquet(df, "languages.parquet")

For this example, you have three imports:

  • One for pandas, so you can create a DataFrame
  • One for pyarrow, to create a special pyarrow.Table object
  • One for pyarrow.parquetto transform the table object into a Parquet file

The  write_parquet() function takes in a pandas DataFrame and the file name or path to save the Parquet file to. Then, you transform the DataFrame into a pyarrow Table object before converting that into a Parquet File using the write_table() method, which writes it to disk.

Now you are ready to read that file you just created!

Reading Parquet Files with Python

Reading the Parquet file you created earlier with Python is even easier. You’ll need about half as many lines of code!

You can put the following code into a new file called something like parquet_file_reader.pyif you want to:

import pyarrow.parquet as pq def read_parquet(filename: str) -> None: table = pq.read_table(filename) df = table.to_pandas() print(df) if __name__ == "__main__": read_parquet("languages.parquet")

In this example, you read the Parquet file into a pyarrow Table format and then convert it to a pandas DataFrame using the Table’s to_pandas() method.

When you print out the contents of the DataFrame, you will see the following:

Languages Users Dynamic 1 Python 10000 True 2 Ruby 5000 True 3 C++ 8000 False

You can see from the output above that the DataFrame contains all data you saved.

One of the strengths of using a Parquet file is that you can read just parts of the file instead of the whole thing. For example, you can read in just some of the columns rather then the whole file!

Here’s an example of how that works:

import pyarrow.parquet as pq def read_columns(filename: str, columns: list[str]) -> None: table = pq.read_table(filename, columns=columns) print(table) if __name__ == "__main__": read_columns("languages.parquet", columns=["Languages", "Users"])

To read in just the “Languages” and “Users” columns from the Parquet file, you pass in the a list that contains just those column names. Then when you call read_table() you pass in the columns you want to read.

Here’s the output when you run this code:

pyarrow.Table Languages: string Users: int64 ---- Languages: [["Python","Ruby","C++"]] Users: [[10000,5000,8000]]

This outputs the pyarrow Table format, which differs slightly from a pandas DataFrame. It tells you information about the different columns; for example, Languages are strings, and Users are of type int64.

If you prefer to work only with pandas DataFrames, the pyarrow package allows that too. As long as you know the Parquet file contains pandas DataFrames, you can use read_pandas() instead of read_table().

Here’s a code example:

import pyarrow.parquet as pq def read_columns_pandas(filename: str, columns: list[str]) -> None: table = pq.read_pandas(filename, columns=columns) df = table.to_pandas() print(df) if __name__ == "__main__": read_columns_pandas("languages.parquet", columns=["Languages", "Users"])

When you run this example, the output is a DataFrame that contains just the columns you asked for:

Languages Users 1 Python 10000 2 Ruby 5000 3 C++ 8000

One advantage of using the read_pandas() and to_pandas() methods is that they will maintain any additional index column data in the DataFrame, while the pyarrow Table may not.

Reading Parquet File Metadata

You can also get the metadata from a Parquet file using Python. Getting the metadata can be useful when you need to inspect an unfamiliar Parquet file to see what type(s) of data it contains.

Here’s a small code snippet that will read the Parquet file’s metadata and schema:

import pyarrow.parquet as pq def read_metadata(filename: str) -> None: parquet_file = pq.ParquetFile(filename) metadata = parquet_file.metadata print(metadata) print(f"Parquet file: {filename} Schema") print(parquet_file.schema) if __name__ == "__main__": read_metadata("languages.parquet")

There are two ways to get the Parquet file’s metadata:

  • Use pq.ParquetFile to read the file and then access the metadata property
  • Use pr.read_metadata(filename) instead

The benefit of the former method is that you can also access the schema property of the ParquetFile object.

When you run this code, you will see this output:

<pyarrow._parquet.FileMetaData object at 0x000002312C1355D0> created_by: parquet-cpp-arrow version 15.0.2 num_columns: 4 num_rows: 3 num_row_groups: 1 format_version: 2.6 serialized_size: 2682 Parquet file: languages.parquet Schema <pyarrow._parquet.ParquetSchema object at 0x000002312BBFDF00> required group field_id=-1 schema { optional binary field_id=-1 Languages (String); optional int64 field_id=-1 Users; optional boolean field_id=-1 Dynamic; optional int64 field_id=-1 __index_level_0__; }

Nice! You can read the output above to learn the number of rows and columns of data and the size of the data. The schema tells you what the field types are.

Wrapping Up

Parquet files are becoming more popular in big data and data science-related fields. Python’s pyarrow package makes working with Parquet files easy. You should spend some time experimenting with the code in this tutorial and using it for some of your own Parquet files.

When you want to learn more, check out the Parquet documentation.

The post How to Read and Write Parquet Files with Python appeared first on Mouse Vs Python.

Categories: FLOSS Project Planets

EuroPython Society: 🐍 Community Call for Venues - EuroPython 2025

Planet Python - Mon, 2024-05-06 09:23

Greetings to all community organizers across Europe! &#x1F30D;

We are thrilled to announce the opening of the Call for Venues for EuroPython 2025! &#x1F310;

EuroPython is the world&aposs oldest volunteer-led Python Conference. We are rooted in principles of diversity, inclusion, and accessibility; advocating for the community, and striving to create a welcoming space for all.

It is our mission to ensure accessibility for the wider community of Pythonistas when selecting conference locations. In addition to ticket prices, we carefully consider the ease of access and sustainability of future venues.

Similar to the process followed for the selection of Prague for EuroPython 2023, we would like your help in choosing the most suitable location for our next editions.

If you want to propose a location on behalf of your community, please send as an email to board@europython.eu with your proposal before May 14th. We will coordinate with you to collect all the necessary data required.

&#x1F4DD; Important Notes:

  • The EPS will also revisit community proposals from previous years.
  • Proposals submitted currently will be retained for future years.

Your city could be the next hub for collaboration, learning, and celebration within the Python ecosystem.

Join us in shaping an unforgettable experience for EuroPython 2025 participants!

✏️ Got any questions/suggestions/comments? Drop us a line at board@europython.eu and we will get back to you.

See you all soon,

EuroPython Society Board

Categories: FLOSS Project Planets

About QML Efficiency: Compilers, Language Server, and Type Annotations

Planet KDE - Mon, 2024-05-06 09:22

In our last post we had a look at how to set up QML Modules and how we can benefit from the QML Linter. Today we’re going to set up the QML Language Server to get an IDE-like experience in an editor of our choice. We’ll also help the the QML Compiler generate more efficient code.

Continue reading About QML Efficiency: Compilers, Language Server, and Type Annotations at basysKom GmbH.

Categories: FLOSS Project Planets

Django Weblog: Django bugfix releases issued: 5.0.5 and 4.2.12

Planet Python - Mon, 2024-05-06 07:57

Today we've issued 5.0.5 and 4.2.12 bugfix releases.

The release package and checksums are available from our downloads page, as well as from the Python Package Index. The PGP key ID used for this release is Sarah Boyce: 3955B19851EA96EF.

Categories: FLOSS Project Planets

LostCarPark Drupal Blog: Drupal Advent Calendar 2024 - Call for Ideas

Planet Drupal - Mon, 2024-05-06 07:01
Drupal Advent Calendar 2024 - Call for Ideas lostcarpark_admin Mon, 05/06/2024 - 12:01 Image Body

DrupalCon Portland starts today, so it seems a good time to start thinking about the 2024 Advent Calendar!

Advent Calendar? In May?

If there’s one thing I learned last year, it’s start early! We had a few hairy moments last year, and a couple of later nights than I would have liked, so this year I want to get the ball moving early.

Why a Drupal Advent Calendar?

For fun, mostly!

But also to promote the great Drupal projects and the people working on them.

It started on a whim in 2022, when I had the idea at the last possible moment, as I was falling asleep on the last day of November. As there wasn’t…

Categories: FLOSS Project Planets

CKEditor: Enhance Your Drupal Experience with the Free CKEditor 5 Plugin Pack

Planet Drupal - Mon, 2024-05-06 06:19
Enhance Drupal with the CKEditor 5 Plugin Pack: all premium and free tools available at no cost for diverse web projects.
Categories: FLOSS Project Planets

The Drop Times: Embracing the Community Spirit: DrupalCon Portland 2024

Planet Drupal - Mon, 2024-05-06 04:56

Today is an exciting day as DrupalCon Portland kicks off. The Drupal community eagerly awaits this event for its wealth of sessions, interactions, and updates. From the highly anticipated Driesnote, where Dries Buytaert shares his latest thoughts and plans, to various featured sessions that delve into specific topics, there's much to look forward to.

At this year’s conference, we're seeing updates on various initiatives and a host of workshops and trainings from different organizations. We also have the Healthcare Summit and the Return of the Nonprofit Summit, a great opportunity for Drupal users working in the nonprofit sector to connect and learn from one another.

Recognizing the need to continuously attract new talents to the Drupal community, DrupalCon 2024 has made significant efforts to reach out to students. This includes targeted advertising to local student communities and focusing on the career-enhancing opportunities at the DrupalCon job fair. Mentorship programs, resume help, and a special student discount ticket priced at only $50 are also included.

Another exciting addition this year is the community-designed DrupalCon T-shirt. The Drupal Association ran a contest for the official T-shirt design, receiving many creative entries. The winning design will be announced at the event and featured on the free attendee T-shirt.

Now, let’s look back at what we covered last week:

Alka Elizabeth, sub-editor, TDT, sat down with Thor Andre Gretland, the dynamic Head of Sales and Business Advisor at Frontkom. They discussed the exciting synergies between Gutenberg and Drupal. Thor shared his extensive knowledge about the groundbreaking Drupal Gutenberg project. During this discussion, it was revealed that the Frontkom team has four updates for the community, including major enhancements that will be part of the Drupal Gutenberg 4.0 release, set to be unveiled here at DrupalCon Portland 2024.

Additionally, Alka Elizabeth talked with Angie Byron, the Community Director at Aiven and a highly respected figure in the open-source community. Throughout their conversation, Angie shared her experiences and the pivotal decisions that have shaped her career. She discussed her challenges and transformations, such as introducing automated testing in Drupal, her leadership roles in various community projects, and her advocacy for diversity and inclusion within tech communities. Dive into the interview here. 

Last week, I had the opportunity to share insights directly from the Drupal Initiative Lead keynote speakers at DrupalCon Portland. Among the speakers were Cristina Chumillas, Janez Urevc, Ted Bowman, Fran Garcia-Linares, Jürgen Haas, and Mateu Aguiló Bosch, who were all set to provide valuable updates and insights on various aspects of Drupal and its ecosystem.

LagoonCon Portland 2024 is set to take place on  May 6, 2024. Following its successful debut in Pittsburgh last year, this free event hosted by amazee.io is designed for developers and tech leaders to dive into discussions about Lagoon. Alka Elizabeth, sub-editor at The Drop Times, has penned an article featuring detailed discussions with the speakers of Lagoon Portland,  Toby Bellwood, the Lagoon Product Lead at amazee.io, Christoph Weber, Solutions Architect at Pronovix, and Bryan Gruneberg, CEO and CTO at Workshop Orange. 

Norah Medlin brings over two decades of software development immersion to Stanford WebCamp, offering a strategic blueprint for maximizing project success. In her session on forging high-value partnerships and driving transformative change, she unveils essential insights for navigating the complexities of modern project management. 

Linux Foundation has launched the 2024 World of Open Source: Global Spotlight Survey, designed to examine the nuances of open-source technology in different regions and industries.

DrupalCon Barcelona 2024 is calling for submissions of case studies highlighting exceptional Drupal website projects developed between October 2023 and September 2024. The Drupal Camp Pune 2024 organisers seek talented individuals passionate about technology and community building to volunteer and add a unique touch to the array of planned activities.

DevOps professionals on the making can take the Pantheon WebOps certification exam onsite at the DrupalCon Portland venue. The exam registration is free of charge. 

Additionally, Carlos O. launched the IXP Fellowship initiative to help aspiring Drupal developers bridge the skills gap. The program sought community feedback through a survey to define competencies for inexperienced developers aiming to become junior professionals.

Other generic updates are here: Maotic, for the first time, has become a mentor organization for the coveted Google Summer of Code project for 2024. Learn about their winning projects here. Drupal is introducing an experimental navigation module in version 10.3 prior to introducing it officially in Drupal 11. Selwyn Polit's online reference book, "Drupal at your fingertips" is now officially listed in Drupal.org. Last year's Drupal Pitch-burgh contest-winning project, Drupal API Client, has released version 1.0. A new Drupal podcast has begun. Platform.sh DevRel Team has started a new podcast series called 'ChangeMode'. Marine Gandy hosts the show. 

We acknowledge that there are more stories to share. However, due to selection constraints, we must pause further exploration for now.

Stay tuned with The Drop Times. We are here to ensure you don't miss out on anything happening at the event. Our volunteers are on the ground to keep you updated with interviews, featured articles, coverage of sessions, and short video interviews with attendees and speakers.

To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.

Thank you,
Sincerely
Kazima Abbas
Sub-editor, The DropTimes.

Categories: FLOSS Project Planets

Zato Blog: What is an API gateway?

Planet Python - Mon, 2024-05-06 03:43
What is an API gateway? 2024-05-06, by Dariusz Suchojad

In this article, we are going to use Zato in its capacity as a multi-protocol Python API gateway - we will integrate a few popular technologies, accepting requests sent over protocols commonly used in frontend systems, enriching and passing them to backend systems and returning responses to the API clients using their preferred data formats. But first, let's define what an API gateway is.

Clearing up the terminology

Although we will be focusing on complex API integrations later on today, to understand the term API gateway we first need to give proper consideration to the very term gateway.

What comes to mind when we hear the word "gateway", and what is correct etymologically indeed, is an opening in an otherwise impermissible barrier. We use a gateway to access that which is in other circumstances inaccessible for various reasons. We use it to leave such a place too.

In fact, both "gate" and the verb "to go" stem from the same basic root and that, again, brings to mind a notion of passing through space specifically set aside for the purpose of granting access to what normally would be unavailable. And once more, when we depart from such an area, we use a gateway too.

From the perspective of its true intended purpose, a gateway letting everyone in and out as they are would amount to little more than a hole in a wall. In other words, a gateway without a gate is not the whole story.

Yes, there is undoubtedly an immense aesthetic gratification to be drawn from being close to marvels of architecture that virtually all medieval or Renaissance gates and gateways represent, but we know that, nowadays, they do not function to the fullest of their capacities as originally intended.

Rather, we can intuitively say that a gateway is in service as a means of entry and departure if it lets its operators achieve the following, though not necessarily all at the same time, depending on one's particular needs:

  • Telling arrivals where they are, including projection of might and self-confidence
  • Confirming that arrivals are who they say they are
  • Checking if their port of origin is friendly or not
  • Checking if they are allowed to enter that which is protected
  • Directing them to specific areas behind the gateway
  • Keeping a long term and short term log of arrivals
  • Answering potential questions right by the gate, if answers are known to gatekeepers
  • Cooperating with translators and coordinators that let arrivals make use of what is required during their stay

We can now recognize that a gateway operates on the border of what is internal and external and in itself, it is a relatively narrow, though possibly deep, piece of an architecture. It is narrow because it is only through the gateway that entry is possible but it may be deeper or not, depending on how much it should offer to arrivals.

We also keep in mind that there may very well be more than a single gateway in existence at a time, each potentially dedicated to different purposes, some overlapping, some not.

Finally, it is crucial to remember that gateways are structural, architectural elements - what a gateway should do and how it should do it is a decision left to architects.

With all of that in mind, it is easy to transfer our understanding of what a physical gateway is into what an API one should be.

  • API clients should be presented with clear information that they are entering a restricted area
  • Source IP addresses or their equivalents should be checked and requests rejected if an IP address or equivalent information is not among the allowed ones
  • Usernames, passwords, API keys and similar representations of what they are should be checked by the gateway
  • Permissions to access backend systems should be checked seeing as not every API client should have access to everything
  • Requests should be dispatched to relevant backend systems
  • Requests and responses should be logged in various formats, some meant to be read by programs and applications, some by human operators
  • If applicable, responses can be served from the gateway's cache, taking the burden off the shoulders of the backend systems
  • Requests and responses can be transformed or enriched which potentially means contacting multiple backend systems before an API caller receives a response

We can now define an API gateway as an element of a systems architecture that is certainly related to security, permissions and granting or rejecting access to backend systems, applications and data sources. On top of it, it may provide audit, data transformation and caching services. The definition will be always fluid to a degree, depending on an architect's vision, but this is what can be expected from it nevertheless.

Having defined what an API gateway is, let's create one in Zato and Python.

Clients and backend systems

In this article, we will integrate two frontend systems and one backend application. Frontend ones will use REST and WebSockets whereas the backend one will use AMQP. Zato will act as an API gateway between them all.

Not granting frontend API clients direct access to backend systems is usually a good idea because the dynamics involved in creation of systems on either side are typically very different. But they still need to communicate and hence the usage of Zato as an API gateway.

Python code

First, let's show the Python code that is needed to integrate the systems in our architecture:

# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class APIGateway(Service): """ Dispatches requests to backend systems, enriching them along the way. """ name = 'api.gateway' def handle(self): # Enrich incoming request with metadata .. self.request.payload['_receiver'] = self.name self.request.payload['_correlation_id'] = self.cid self.request.payload['_date_received'] = self.time.utcnow() # .. AMQP configuration .. outconn = 'My Backend' exchange = '/incoming' routing_key = 'api' # .. publish the message to an AMQP broker .. self.out.amqp.send(data, outconn, exchange, routing_key) # .. and return a response to our API client. self.response.payload = {'result': 'OK, data accepted'}

There are a couple of points of interest:

  • The gateway service enriches incoming requests with metadata but it could very well enrich it with business data too, e.g. it could communicate with yet another system to obtain required information and only then pass the request to the final backend system(s)

  • In its current form we send all the information to AMQP brokers only but we could just as well send it to other systems, possibly modifying the requests along the way

  • The code is very abstract and all of its current configuration could be moved to a config file, Redis or another data source to make it even more high-level

  • Security configuration and other details are not declared directly in the body of the gateway service but they need to exist somewhere - we will describe it in the next section

Configuration

In Zato, API clients access the platform's services using channels - let's create a channel for REST and WebSockets then.

First REST:

Now WebSockets:

We create a new outgoing AMQP connection in the same way:

Using the API gateway

At this point, the gateway is ready - you can invoke it from REST or WebSockets and any JSON data it receives will be processed by the gateway service, the AMQP broker will receive it, and API clients will have replies from the gateway as JSON responses.

Let's use curl to invoke the REST channel with JSON payload on input:

$ curl http://api:<password-here>@localhost:11223/api/v1/user ; echo curl --data-binary @request.json http://localhost:11223/api/v1/user ; echo {"result": "OK, data accepted"} $

Taken together, the channels and the service allowed us to achieve this:

  • Multiple API clients can access the backend AMQP systems, each client using its own preferred technology
  • Client credentials are checked on input, before the service starts to process requests (authentication)
  • It is possible to assign RBAC roles to clients, in this way ensuring they have access only to selected parts of the backend API (authorization)
  • Message logs keep track of data incoming and outgoing
  • Responses from channels can be cached which lessens the burden put on the shoulders of backend systems
  • Services accepting requests are free to modify, enrich and transform the data in any way required by business logic. E.g., in the code above we only add metadata but we could as well reach out to other applications before requests are sent to the intended recipients.

We can take it further. For instance, the gateway service is currently completely oblivious to the actual content of the requests.

But, since we just have a regular Python dict in self.request.payload, we can with no effort modify the service to dispatch requests to different backend systems, depending on what the request contains or possibly what other backend systems decide the destination should be.

Such additional logic is specific to each environment or project which is why it is not shown here, and this is also why we end the article at this point, but the central part of it all is already done, the rest is only a matter of customization and plugging in more channels for API clients or outgoing connections for backend systems.

Finally, it is perfectly fine to split access to systems among multiple gateways - each may handle requests from selected technologies on the one hand but on the other hand, each may use different caching or rate-limiting policies. If there is more than one, it may be easier to configure such details on a per-gateway basis.

Next steps More blog posts➤
Categories: FLOSS Project Planets

The Drop Times: Women in Drupal Luncheon at DrupalCon Portland 2024: A Convergence for Change

Planet Drupal - Mon, 2024-05-06 02:24
Delve into the experiences of women in technology at DrupalCon Portland 2024 with the "Women in Drupal Luncheon" on May 7. This crucial session features Sebastianna Skalisky, Laura Johnson, Jenna Harris, and Shanice Ortiz from Four Kitchens, who will discuss overcoming obstacles and fostering inclusion in the tech sector. Engage with these influential leaders as they share strategies for navigating a male-dominated industry, enhancing female leadership, and advocating for systemic change. Join for a compelling dialogue to empower women and expand their impact on technology.
Categories: FLOSS Project Planets

FSF Blogs: Sharing day two of LibrePlanet 2024: Cultivating Community

GNU Planet! - Sun, 2024-05-05 21:41
Categories: FLOSS Project Planets

Sharing day two of LibrePlanet 2024: Cultivating Community

FSF Blogs - Sun, 2024-05-05 21:41
Categories: FLOSS Project Planets

My work in KDE for April 2024

Planet KDE - Sun, 2024-05-05 20:00

Hello and sorry about the late post. I’ve been busy moving and other stuff that’s gotten in the way. I will also be idling the beginning of this month, so the next update may be shorter too.

Anyway, let’s get into the changes!

Kensa

I originally wanted to bring some of the “power-user” features from KSysGuard into the new System Monitor. I was rightfully turned down because they were hesitant of there being any use for most people and to prevent feature creep.

They suggested creating a seperate application instead. So Kensa, the detailed process viewer is born! It’s still mostly copy & pasted from old KSysGuard/libksysguard code, but updated for Qt6/KF6. And to make it clear, It’s very clearly modeled after the Window’s Process Explorer.

I have the general tab for viewing some basic information about the process. Said tab also includes an “Open With” so you can quickly open the executable in a hex viewer like Okteta.

The general tab

The memory maps tab shows what the process has mapped, mostly notably which shared libraries it’s currently using.

The memory maps tab

The open files tab makes it’s return as well, extremely useful.

The open files tab

And one of my own design, an environment variables tab. In the future I want to add a “Strings” tab for quickly viewing the executable strings and the ones currently in memory.

The environment tab

Note that Kensa is very early in development and not user-friendly. You currently have it give it a PID manually and lacks a process list.

Tokodon

Feature The window title now corresponds to the current page. This makes it easier to identify from a task bar, too. We know the title is duplicated inside the application as well (on desktop), but that’s Kirigami’s design decision. 24.05

Feature If your server doesn’t provide a human-readable error message, the network error is displayed instead. This is useful to see if the DNS lookup failed or some other network-related reason the server is inaccessible. 24.05

Feature Support for QtMultimedia has been added in situations where your system lacks or cannot use libmpv. This is preparatory work for a Windows version. 24.05

Feature In the same vein as the patch above, QtWebView is now optional and I included even more authentication fixes. Previously I enforced an in-app web view to facilitate authentication (compared to the external web browser method or auth code in previous versions.) This was only a stop-gap solution until I had more time to flesh out our authentication system, but now I feel much happier about it’s current state. 24.05

System Monitor

Bugfix Fix the column configuration dialog being shown too small on the Overview page. 6.0.4

Feature Add the About KDE page to the hamburger menu. 6.1

Bugfix Made sure cell tooltips shows up more reliably. 6.1

Feature Added a menu item to copy the current column’s text. This makes System Monitor just as usable as the old KSysGuard for me now, because I tend to copy the command line a lot. (And PIDs.) 6.1

Ruqola

Bugfix Use a better fitting icon for attaching files. The previous icon - when viewed at 16px - turned into something completely different. 2.1.2

PlasmaTube

Feature Added support for viewing a channel’s playlists. 24.05

Feature I also added a Craft blueprint for Android. Note that this is only preliminary and the Android version is nowhere near ready yet. I guess this could be used for a future Windows version too.

Feature I implemented more functionality in the PeerTube backend, so now it’s possible to log in and perform searching. Subscriptions work too, but I’m running into an issue where yt-dlp fails to pull certain videos. If you know anything about using yt-dlp with PeerTube, please let me know if there’s a workaround. 24.05

Feature Added a new feature to import/export OPML subscriptions. This only works for YouTube channels at the moment, PeerTube support TBA. 24.05

Gwenview

Feature I changed the old save bar to use the standard KMessageWidget widget. This isn’t just for looks, it fixes a lot of odd visual bugs and removes a ton of cruft in the process. 6.1.0

The new Gwenview save bar. If it doesn’t look “out of place”, then my patch did it’s job! NeoChat

Bugfix Fixed the share dialog not appearing properly, and improve the keyboard navigation inside of it. 24.05

Frameworks

Bugfix Remove some redundant QML_ELEMENT declarations which in turn reduces runtime warnings. 6.1.0

Bugfix Two KMessageWidget improvements, by fixing handling of color palette changes and making the icon label vertically centered. This is for that Gwenview patch. 6.1.0

Android

I once again sat down and fixed a ton of build and runtime issues for our Android applications, and started fixing some of the Qt 6.7 fallout. NeoChat and Tokodon build and run again, and spent some time ironing out their issues.

That’s all this month!

My work in KDE for March 2024

My Work in KDE

Home
Categories: FLOSS Project Planets

Pages