Feeds

PyCoder’s Weekly: Issue #621 (March 19, 2024)

Planet Python - Tue, 2024-03-19 15:30

#621 – MARCH 19, 2024
View in Browser »

Visualizing Data in Python With Seaborn

In this tutorial, you’ll learn how to use the Python seaborn library to produce statistical data analysis plots to allow you to better visualize your data. You’ll learn how to use both its traditional classic interface and more modern objects interface.
REAL PYTHON

Does Python Have Pointers?

Depending on how you’re using the term “pointer” changes the answer to the question. Read on to better understand the programming terminology and whether Python has pointers.
NED BATCHELDER

GPT Pilot Is an OSS Dev Tool That Builds Apps From Scratch by Talking to You

GPT Pilot is a collection of AI agents that automate developer workflows to try offloading 95% of coding tasks from you to the AI. Right now, it can build apps up to ~3000 lines of code see examples here →
PYTHAGORA TECHNOLOGIES INC. sponsor

The Python Memory Model

This article introduces you to how Python stores things in memory. Learn about the heap, the stack, and how the interpreter sees Python objects.
PEPIJN BAKKER

Python 3.13.0 Alpha 5 Released

CPYTHON DEV BLOG

New Malware Reporting Tool on PyPI

PYPI

Articles & Tutorials Build an LLM RAG Chatbot With LangChain

Large language models (LLMs) have taken the world by storm, demonstrating unprecedented capabilities in natural language tasks. In this step-by-step tutorial, you’ll leverage LLMs to build your own retrieval-augmented generation (RAG) chatbot using synthetic data with LangChain and Neo4j.
REAL PYTHON

Use Weird Tests to Capture Tacit Knowledge

Sometimes adding code in one place means configuration elsewhere also needs to be updated. One way of ensuring this is happening properly in a large project is to use unit tests. This post covers a few examples, complete with pytest code.
JUSTIN DUKE

Blocked by Slow Code Reviews? Here’s How to Stop Waiting

Code reviews are great - but they shouldn’t slow down your development. Sourcery can automatically review every one of your PR’s so your team can keep moving fast →
SOURCERY sponsor

Insecurity and Python Pickles

The pickle module allows you to serialize arbitrary Python objects. Serializing them back means executing code, which has potential security issues. Read on to discover what they are and what software may be impacted.
DAROC ALDEN

Python Basics Exercises: Installing Packages With pip

In this Python Basics Exercises video course, you’ll practice installing packages with pip. You’ll also practice creating virtual environments, making lists of requirements, and recreating a development environment.
REAL PYTHON course

How to Create a Dashboard in Python From PostgreSQL

Accessing a database in a terminal is not the best solution for everyone. Mljar lets you build a dashboard from scratch using only Python.
MLJAR • Shared by Piotr

Regex Character “$” Doesn’t Mean “End-of-String”

Regular expression syntax is only somewhat uniform across programming languages. Seth ran into a surprise with “$” and Python.
SETH LARSON

Feature Flags Are Ruining Your Codebase

An opinion piece on feature flags and the dangers of letting PMs control them. Includes suggestions on what to do instead.
ANTON ZAIDES

The 2038 Problem

Learn how “The 2038 problem” could impact software, hardware, and more - and what can be done to prepare.
CODE RELIANT

Add Magic Link Sign-in Using Django

This is a step-by-step guide to adding email sign-in (and verification) to Django using Gmail and others.
PHOTON DESIGNER

Email Testing With Python’s smtpd Module

This post dives deep into Python’s smtpd module and explores how it can used for local testing.
MUHAMMAD

Projects & Code zakuchess: Chess Challenge Game

GITHUB.COM/OLIVIERPHI

mountaineer: Web Framework for Python and React

GITHUB.COM/PIERCEFREEMAN

magika: Detect File Content Types With Deep Learning

GITHUB.COM/GOOGLE

hyperdiv: Build Reactive Web UIs in Python

GITHUB.COM/HYPERDIV

UFO: A UI-Focused Agent for Windows OS Interaction

GITHUB.COM/MICROSOFT

Events Weekly Real Python Office Hours Q&A (Virtual)

March 20, 2024
REALPYTHON.COM

PyData Bristol Meetup

March 21, 2024
MEETUP.COM

PyLadies Dublin

March 21, 2024
PYLADIES.COM

Python Barcamp Karlsruhe

March 23 to March 25, 2024
BARCAMPS.EU

PyDelhi User Group Meetup

March 23, 2024
MEETUP.COM

Happy Pythoning!
This was PyCoder’s Weekly Issue #621.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Categories: FLOSS Project Planets

Evolving Web: 9 Ways to Make the Most of the EvolveDrupal Summit

Planet Drupal - Tue, 2024-03-19 13:34

Hi there! I’m Jasmin Merchant, a Front-End Developer at Evolving Web. A few years ago I knew very little about the open web. That all changed when I started working here and was plunged into ongoing learning opportunities.

The EvolveDrupal Ottawa summit in 2023 was a big part of my learning curve. Hosted by Evolving Web, the event was a chance to gain new knowledge about development, accessibility, UX, digital strategy, and many other relevant topics.

What’s more, it was a great introduction to networking. This was my first tech summit and I had only moved to Canada one week before, so I was nervous to say the least. But EvolveDrupal was a valuable experience that really built my confidence.

I’ve shared my takeaways in this article to help others who are planning to attend EvolveDrupal (or any tech event). Use these 9 tips to attend with confidence and make the most of it!

Before the Summit 1. Create your own schedule

My day at EvolveDrupal Ottawa started with coffee (because obviously!). I hung out in the foyer where I could chat to people as they arrived. I was looking forward to talks on Drupal, design, AI, and much more. Crucially, I had gone through the session list earlier and planned out my day.

Having your own schedule means you can pace yourself, attend the talks you're most interested in, and maximize your networking opportunities. Remember that EvolveDrupal has a wide range of sessions and many of them run in parallel, so it isn’t possible to attend all of them.

I’d recommend making space in your schedule for a bit of down time to recharge. I found this was easy to do at EvolveDrupal—the event was very well organized with little breaks between sessions, helping me to relax and go with the flow.


 

2. Sign up for pre-summit introductions

Evolving Web often organizes a pre-summit dinner the night before EvolveDrupal. As an introvert, I found this was a great way to ease into networking. I got to meet some really interesting, talented people and talk to them on a personal level over a nice meal. It meant that I saw some familiar faces in the crowd full of attendees the next day.

Can’t attend the pre-summit dinner? Don’t worry, you can connect digitally with organizers, speakers and participants before the summit instead. If you’ve registered for the summit, you’ll be invited to the EvolveDrupal Slack channel. 

3. Build your visibility

Post on social media to tell your connections that you’ll be at EvolveDrupal. This can help you connect with more people at the event. Remember to tag Evolving Web on LinkedIn so we can help boost your post!

Email specific contacts and invite them to the summit, too. This can be a great way to grow and strengthen existing business relationships.

Want to maximize your visibility? Apply to be a speaker or sponsor. It’s a fantastic way to get in front of an audience—both in-person at the summit and also in Evolving Web’s digital communications.

 

“I had a blast yesterday talking at the EvolveDrupal conference in Ottawa. Thank you for giving me the opportunity. Based on the number of people in the audience that raised their phone to take pictures of the slides, I guess the content was on point!”

– Gauthier Garnier, Platform.sh (EvolveDrupal Ottawa 2023)

During the Summit  4. Make notes

At each session I attended at EvolveDrupal, I took notes about everything interesting and important that I was learning. Have a think about how you prefer to take notes—can you do it on your phone, or should you bring a tablet or notebook?

It’s best to jot down new knowledge and ideas at the event itself, before you forget. That said, some EvolveDrupal sessions are recorded and published a few weeks later, so you may have a chance to rewatch select talks. 

5. Embrace a bit of discomfort

While the pre-summit dinner put me at greater ease, I was still nervous about networking. But I learned that it’s ok to be outside of my comfort zone. As one of the speakers said: “get comfortable being uncomfortable.” Their talk was on AI, but I think the concept can be applied to a lot more!

I started by introducing myself to people I sat near during the sessions. This felt easier as there was already a clear topic to start the conversation. I met many more people at lunch and kept taking mental notes that I wrote down later. If there’s someone you want to follow up with, ask for their business card or record their name so you can add them on LinkedIn.

It’s also worth engaging with the speakers who interest you. As experts they can offer valuable information, and they’re usually more than happy to expand on their talk if you ask questions.


 

6. Stick around for evening events

EvolveDrupal Ottawa finished with a networking reception, but my day didn’t end there! I attended a post-summit dinner and drinks with people who wanted to continue connecting. As it was the end of the night, everyone was relaxed and chatting easily. It was a good chance to speak again with the people I’d met that day and get to know them better. Evolving Web often arranges a post-summit dinner, so leave your evening open if you can. 

After the Summit 7. Follow up with your connections 

I added most of the people I met at EvolveDrupal Ottawa on LinkedIn. It’s an easy way to stay connected because you can respond to their posts or share useful things with them. It’s also a good idea to send a personalized message to new contacts who you want to develop a deeper relationship with, such as business prospects and potential employers. Ask for a phone call or face-to-face meeting to discuss opportunities. 

8. Organize and apply what you learned 

Write up your notes soon after the summit. If you leave it too long, you may forget them or find they don’t make sense any more! Also, decide how you’re going to integrate your new knowledge into your workflows and projects. Act quickly while you’re still feeling energized from the event.

 

“It was a treasure trove of UX wisdom… Big thanks to the amazing speakers and organizers for making this experience unforgettable. Can't wait to apply what I've learned. Count me in for the next one!”

– Milan Nayak, UX Designer (EvolveDrupal Toronto 2023) 

9. Share your insights and experience 

My time at EvolveDrupal was eye-opening, so I wanted to share my experience in an article to help future participants. You can also share takeaways via an email, team meeting, or internal presentation. Finally, it’s always nice to thank the speakers on social media and tell them what you liked about their session.

I hope this article inspires you to attend EvolveDrupal with confidence and purpose. Sign up for updates to hear about upcoming events. Our next stops are in Atlanta (April 12) and Montreal (June 14)—we look forward to seeing you there!

+ more awesome articles by Evolving Web
Categories: FLOSS Project Planets

The Drop Times: Catch Up with the Drupal Community at DrupalSouth Sydney 2024!

Planet Drupal - Tue, 2024-03-19 10:29
Discover the highlights and insider views of DrupalSouth Sydney 2024, from key tech updates to diversity initiatives. Learn what makes this event a key spot for developers and digital enthusiasts, along with exclusive insights from the community's leading voices. Get a sneak peek into the enriching sessions, networking opportunities, and cultural experiences waiting in Sydney.
Categories: FLOSS Project Planets

Real Python: SQLite and SQLAlchemy in Python: Move Your Data Beyond Flat Files

Planet Python - Tue, 2024-03-19 10:00

All programs process data in one form or another, and many need to be able to save and retrieve that data from one invocation to the next. Python, SQLite, and SQLAlchemy give your programs database functionality, allowing you to store data in a single file without the need for a database server.

You can achieve similar results using flat files in any number of formats, including CSV, JSON, XML, and even custom formats. Flat files are often human-readable text files—though they can also be binary data—with a structure that can be parsed by a computer program. You’ll explore using SQL databases and flat files for data storage and manipulation and learn how to decide which approach is right for your program.

In this video course, you’ll learn how to use:

  • Flat files for data storage
  • SQL to improve access to persistent data
  • SQLite for data storage
  • SQLAlchemy to work with data as Python objects

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Robin Wilson: One reason for getting a ‘No HTTP triggers found’ error when using Azure Functions with Python V2 programming model

Planet Python - Tue, 2024-03-19 06:38

Summary: It might be because there is an exception raised when importing your function_app.py – for example, caused by one of your import statements raising an exception, or a parsing error caused by a syntax error.

I was deploying a FastAPI app to Azure Functions recently. Azure Function is the equivalent of AWS Lambda – it provides a way to run serverless functions.

Since I’d last used Azure Functions, Microsoft have introduced the Azure Functions Python V2 programming model which makes it easier and cleaner to do a number of common tasks, such as hooking up a FastAPI app to run on Functions.

However, it also led to an error that I hadn’t seen before, and that I couldn’t find documented very well online.

Specifically, I was getting an error at the end of my function deployment saying No HTTP triggers found. I was confused by this because I had followed the documented pattern for setting up a FastAPI app. For reference, my function_app.py file looked a bit like this:

import azure.functions as func from complex_fastapi_app import app app = func.AsgiFunctionApp(app=app, http_auth_level=func.AuthLevel.ANONYMOUS)

This was exactly as documented. But I kept getting this error – why?

I replaced the import of my complex_fastapi_app with a basic FastAPI app defined in function_app.py, this time copied directly from the documentation:

import azure.functions as func from fastapi import FastAPI, Request, Response fast_app = FastAPI() @fast_app.get("/return_http_no_body") async def return_http_no_body(): return Response(content="", media_type="text/plain") app = func.AsgiFunctionApp(app=fast_app, http_auth_level=func.AuthLevel.ANONYMOUS)

Everything worked fine now and I didn’t get the error.

After a lot of debugging, it turns out that if there is an exception raised when importing your function_app.py file then Functions won’t be able to establish what HTTP triggers you have, and will give this error.

In this case, I was getting an exception raised when I imported my complex_fastapi_app, and that stopped the whole file being processed. Unfortunately I couldn’t find anywhere that this error was actually being reported to me – I must admit that I find Azure logging/error reporting systems very opaque. I assume it would have been reported somewhere – if anyone reading this can point me to the right place then that’d be great!

I’m sure there are many other reasons that this error can occur, but this was one that I hadn’t found documented online – so hopefully this can be useful to someone.

Categories: FLOSS Project Planets

The Python Coding Blog: The Python Coding Book is Out in Paperback and EBook

Planet Python - Tue, 2024-03-19 04:24

The Python Coding Book is out—I published the First Edition in paperback and EBook, which is a revised version of the “Zeroth” Edition which you’ve been able to read here on this site for a while—just ask Google for a “python book” and it will recommend this one as one of it’s top entries!

Read on to see the table of contents, the back-cover blurb introducing the textbook, and testimonials from you, the readers!

Buy Paperback or EBook

The Python Coding Book • A relaxed and friendly programming textbook for beginners

Table of Contents

First Edition, 368 pages

  • 0 Preface • How to Learn to Code
  • Before You Start • Downloading Python and an IDE
  • 1 Getting Started: Your First Project
  • 2 Loops, Lists, and More Fundamentals
  • 3 Power-up Your Coding: Create Your Own Functions
  • 4 Data, Data Types, and Data Structures
  • Monty and The White Room: Understanding Programming
  • 5 Dealing With Errors and Bugs
  • 6 Functions Revisited
  • 7 Object-Oriented Programming
Blurb

Imagine a programming book that feels like a conversation with a friend who’s here to show you the ropes—that’s The Python Coding Book by Stephen Gruppetta. This isn’t a dry textbook. It’s a warm, engaging guide into the world of Python programming, designed with beginners in mind.

With an approach that emphasises clarity and the joy of learning, Stephen guides you through the core concepts of programming, breaking down the barriers that make coding seem inaccessible to many. This book is built on the premise that to truly grasp programming, you need to understand the ‘why’ just as much as the ‘how’. Through engaging explanations, thoughtful analogies, and practical projects, you’re not just learning to code—you’re learning to think and solve problems like a programmer.

Forget about overwhelming details and rapid leaps in complexity. The Python Coding Book introduces concepts at a pace that ensures comprehension, building a solid foundation that instils both knowledge and confidence.

Are you ready for a rewarding journey? The Python Coding Book is more than a book—it’s your first step towards mastering programming with Python. This book is an invitation to not only learn Python but to fall in love with coding.

Testimonials

“It’s the first time I’m understanding what everything does.”

“Your writing is succinct, easy to understand, and process oriented, which I really appreciate. I’m starting to realise that my first experiences with programming weren’t at all representative of my abilities to problem solve or structure my thinking. It has been a great confidence booster for me, and I’m sure other folks are also realising that they were never really the problem. It was the lack of resources or inaccessible information that was the issue. Thanks again for this wonderful resource.”

“The clarity of your writing has helped me understand Python at a deeper level.”

“Thank you for this great resource. I believe this book is the most comprehensive way to understand the material, going beyond the mere memorisation of code snippets or syntax. I recommend it to everyone I talk to who is looking for a Python resource for getting started and who wants to really understand what they are doing at a deeper level.”

Buy Paperback or EBook

The post The Python Coding Book is Out in Paperback and EBook appeared first on The Python Coding Book.

Categories: FLOSS Project Planets

Python Bytes: #375 Pointing at Countries

Planet Python - Tue, 2024-03-19 04:00
<strong>Topics covered in this episode:</strong><br> <ul> <li><a href="https://github.com/pycountry/pycountry">pycountry</a></li> <li><a href="https://nedbatchelder.com/blog/202403/does_python_have_pointers.html"><strong>Does Python have pointers?</strong></a></li> <li><a href="https://bruin-data.github.io/ingestr/">ingestr</a></li> <li><a href="https://davidism.com/starship-and-fish/"><strong>Make your terminal nice</strong></a></li> <li><strong>Extras</strong></li> <li><strong>Joke</strong></li> </ul><a href='https://www.youtube.com/watch?v=fFGXqNXnQR8' style='font-weight: bold;'data-umami-event="Livestream-Past" data-umami-event-episode="375">Watch on YouTube</a><br> <p><strong>About the show</strong></p> <p>Sponsored by ScoutAPM: <a href="https://pythonbytes.fm/scout"><strong>pythonbytes.fm/scout</strong></a></p> <p><strong>Connect with the hosts</strong></p> <ul> <li>Michael: <a href="https://fosstodon.org/@mkennedy"><strong>@mkennedy@fosstodon.org</strong></a></li> <li>Brian: <a href="https://fosstodon.org/@brianokken"><strong>@brianokken@fosstodon.org</strong></a></li> <li>Show: <a href="https://fosstodon.org/@pythonbytes"><strong>@pythonbytes@fosstodon.org</strong></a></li> </ul> <p>Join us on YouTube at <a href="https://pythonbytes.fm/stream/live"><strong>pythonbytes.fm/live</strong></a> to be part of the audience. Usually Tuesdays at 11am PT. Older video versions available there too.</p> <p><strong>Michael #1:</strong> <a href="https://github.com/pycountry/pycountry">pycountry</a></p> <ul> <li>A Python library to access ISO country, subdivision, language, currency and script definitions and their translations.</li> <li>pycountry provides the ISO databases for the standards: <ul> <li><a href="https://en.wikipedia.org/wiki/ISO_639-3">639-3</a> Languages</li> <li><a href="https://en.wikipedia.org/wiki/ISO_3166">3166</a> Codes for representation of names of countries and their subdivisions</li> <li><a href="https://en.wikipedia.org/wiki/ISO_3166-1">3166-1</a> Countries</li> <li><a href="https://en.wikipedia.org/wiki/ISO_3166-3">3166-3</a> Deleted countries</li> <li><a href="https://en.wikipedia.org/wiki/ISO_3166-2">3166-2</a> Subdivisions of countries</li> <li><a href="https://en.wikipedia.org/wiki/ISO_4217">4217</a> Currencies</li> <li><a href="https://en.wikipedia.org/wiki/ISO_15924">15924</a> Scripts</li> </ul></li> </ul> <p><strong>Brian #2:</strong> <a href="https://nedbatchelder.com/blog/202403/does_python_have_pointers.html"><strong>Does Python have pointers?</strong></a></p> <ul> <li>Ned Batchelder</li> <li>Turns out, this is really the description of “what’s a variable in Python?” that helps to make sense of the “variables as names” model in Python, especially for people coming from languages that use pointers a lot.</li> <li>You can use <code>id()</code> to find out what a variable points to</li> <li>You just can’t do the reverse of access it given an id.</li> <li>There’s no “dereference” operator.</li> <li>See also <a href="https://nedbatchelder.com/text/names1.html">Python Names and Values</a>, also by Ned <ul> <li>Should be required reading/viewing for all Python curriculum.</li> </ul></li> </ul> <p><strong>Michael #3:</strong> <a href="https://bruin-data.github.io/ingestr/">ingestr</a></p> <ul> <li>ingestr is a command-line application that allows ingesting or copying data from any source into any destination database.</li> <li>Works on both MongoDB and Postgres and <a href="https://bruin-data.github.io/ingestr/supported-sources/overview.html">many more</a>. </li> <li>incremental loading: <code>append</code>, <code>merge</code> or <code>delete+insert</code></li> </ul> <p><strong>Brian #4:</strong> <a href="https://davidism.com/starship-and-fish/"><strong>Make your terminal nice</strong></a></p> <ul> <li>David Lord</li> <li>David’s switched to <a href="https://fishshell.com">Fish</a> and <a href="https://starship.rs">Starship</a></li> <li>I tried switching to Fish several times, and I guess I’m good with zsh. <ul> <li>Although I admire the brave comic sans motto: “<strong>Finally, a command line shell for the 90s”</strong></li> </ul></li> <li>But I’m finally ready for Starship, and it takes <a href="https://starship.rs/guide/#%F0%9F%9A%80-installation">almost no time to set up</a></li> <li>Plus it’s fast. (Has it always been Rust?)</li> </ul> <p><strong>Extras</strong> </p> <p>Brian:</p> <ul> <li>Doing some groundwork for a SaaS project, using <a href="https://www.saaspegasus.com/?via=brian">SaaS Pegasus</a> <ul> <li>I just talked with Cory from <a href="https://www.saaspegasus.com/?via=brian">Pegasus</a> for an upcoming PythonTest episode</li> <li>I haven’t decided whether to save up SaaS episodes for one big series, or spread them out.</li> <li>But mostly I’m excited to get my project started.</li> </ul></li> </ul> <p>Michael:</p> <ul> <li>Excellent video about “<a href="https://youtu.be/a30vFpSaoZg?si=sdsOspRZB6Kg3Tpo">cloud exit</a>”</li> <li><a href="https://talkpython.fm/episodes/show/453/uv-the-next-evolution-in-python-packages">uv - The Next Evolution in Python Packages</a>?</li> <li><a href="https://pythoninsider.blogspot.com/2024/03/python-3130-alpha-5-is-now-available.html">Python 3.13 a5</a></li> <li><a href="https://tech.target.com/blog/open-source-fund">Target’s Open Source Fund</a> via Pat Decker</li> </ul> <p><strong>Joke:</strong> <a href="https://devhumor.com/media/anti-social-engineer">Anti-social engineer</a></p>
Categories: FLOSS Project Planets

Colin Watson: apt install everything?

Planet Debian - Tue, 2024-03-19 03:05

On Mastodon, the question came up of how Ubuntu would deal with something like the npm install everything situation. I replied:

Ubuntu is curated, so it probably wouldn’t get this far. If it did, then the worst case is that it would get in the way of CI allowing other packages to be removed (again from a curated system, so people are used to removal not being self-service); but the release team would have no hesitation in removing a package like this to fix that, and it certainly wouldn’t cause this amount of angst.

If you did this in a PPA, then I can’t think of any particular negative effects.

OK, if you added lots of build-dependencies (as well as run-time dependencies) then you might be able to take out a builder. But Launchpad builders already run arbitrary user-submitted code by design and are therefore very carefully sandboxed and treated as ephemeral, so this is hardly novel.

There’s a lot to be said for the arrangement of having a curated system for the stuff people actually care about plus an ecosystem of add-on repositories. PPAs cover a wide range of levels of developer activity, from throwaway experiments to quasi-official distribution methods; there are certainly problems that arise from it being difficult to tell the difference between those extremes and from there being no systematic confinement, but for this particular kind of problem they’re very nearly ideal. (Canonical has tried various other approaches to software distribution, and while they address some of the problems, they aren’t obviously better at helping people make reliable social judgements about code they don’t know.)

For a hypothetical package with a huge number of dependencies, to even try to upload it directly to Ubuntu you’d need to be an Ubuntu developer with upload rights (or to go via Debian, where you’d have to clear a similar hurdle). If you have those, then the first upload has to pass manual review by an archive administrator. If your package passes that, then it still has to build and get through proposed-migration CI before it reaches anything that humans typically care about.

On the other hand, if you were inclined to try this sort of experiment, you’d almost certainly try it in a PPA, and that would trouble nobody but yourself.

Categories: FLOSS Project Planets

Specbee: Hooks or Events? Choosing the Best Approach for your Drupal Project

Planet Drupal - Tue, 2024-03-19 01:57
For Drupal developers, it's crucial to understand two fundamental concepts: Events and Hooks. Why? Because they’re the most powerful ways to enable customization and extensibility in Drupal. An event is a system or module-generated occurrence that triggers specific actions, while a hook is a callback function that allows developers to interact with, modify, or extend the behavior of a Drupal core or any module. Ready to learn more about each of them and find out how they’re different in their roles and usage in Drupal development? Dive in! What are Events in Drupal Events are just like hooks that tell Drupal to call your function if something happens. We can say Events are Object Oriented Hook System. Drupal Events allow various system components to interact and communicate with one another independently or in a decoupled manner. Characteristics Events are part of Drupal's broader adoption of the Symfony framework. Events are dispatched by certain actions or triggers within the system. You can dispatch events while writing custom code in order to notify other components in the system about actions taken by your code. Developers can subscribe to these events and define custom actions to be executed when the event occurs. Example Drupal core event: KernelEvents::REQUEST Scenario: Implementing a custom module that listens to the REQUEST event to perform specific actions before the request is processed. // MyModuleEventSubscriber.php namespace Drupal\my_module\EventSubscriber; use Symfony\Component\EventDispatcher\EventSubscriberInterface; use Symfony\Component\HttpKernel\KernelEvents; use Symfony\Component\HttpKernel\Event\RequestEvent; class MyModuleEventSubscriber implements EventSubscriberInterface {   public static function getSubscribedEvents() {     $events[KernelEvents::REQUEST][] = ['onRequestEvent'];     return $events;   }   public function onRequestEvent(RequestEvent $event) {     // Custom logic to be executed on every request.   } }Discover Existing Events There are different methods for finding existing events: 1. Using the WebProfiler module: Download and enable WebProfiler and Devel module since WebProfiler depends on the Devel module Then navigate to Manage > Configuration > Devel Settings > WebProfiler and then select the checkbox to activate the “Events” toolbar item. Now while you visit any page on your site you should see the WebProfiler toolbar at the bottom of the page, and after clicking on the events toolbar icon you will get a list of all event subscribers and information that are called during that request. 2. Use Devel to view and event class: drush devel:event  Enter the number for which you want to get information.:   [0] kernel.controller   [1] kernel.exception   [2] kernel.request   [3] kernel.response   [4] kernel.terminate   [5] kernel.view  > 0  Enter the number to view the implementation.:   [0] Drupal\path_alias\EventSubscriber\PathAliasSubscriber::onKernelController   [1] Drupal\Core\EventSubscriber\EarlyRenderingControllerWrapperSubscriber::onController   [2] Drupal\webprofiler\DataCollector\RequestDataCollector::onKernelController  > 03. Search in your codebase for @Event:  In your editor such as Visual Studio or PHPStorm, search for text @Event within the file mask: *.php option. Subscribe to an Event As we know Drupal uses an event–driven architecture, where various components can communicate with each other by dispatching and subscribing to Events. Here is an example of subscribing to an Event in Drupal 9/10. 1. Define and Event subscriber service  # MyModule/my_module.services.yml services:   my_module.event_subscriber:     class: Drupal\my_module\EventSubscriber\MyModuleEventSubscriber     tags:       - { name: event_subscriber }2. Define an Event subscriber class // MyModule/src/EventSubscriber/MyModuleEventSubscriber.php namespace Drupal\my_module\EventSubscriber; use Symfony\Component\EventDispatcher\EventSubscriberInterface; use Symfony\Component\EventDispatcher\Event; /** * Class MyModuleEventSubscriber. */ class MyModuleEventSubscriber implements EventSubscriberInterface {   /**   * {@inheritdoc}   */   public static function getSubscribedEvents() {     // Specify the event(s) to subscribe to and the method to call when the event occurs.     $events = [       'node.insert' => 'onNodeInsert',       'user.login' => 'onUserLogin',     ];     return $events;   }   /**   * React to a node insert event.   */   public function onNodeInsert(Event $event) {     // Your logic here.     \Drupal::logger('my_module')->notice('Node inserted!');   }   /**   * React to a user login event.   */   public function onUserLogin(Event $event) {     // Your logic here.     \Drupal::logger('my_module')->notice('User logged in!');   } }In this example: MyModuleEventSubscriber is a class that implements the EventSubscriberInterface. The getSubscribedEvents method specifies which events the subscriber is interested in and which method to call when each event occurs. The onNodeInsert and onUserLogin methods contain the logic you want to execute when the corresponding events occur. Dispatch an Event In order to allow another developer to subscribe to the Events and react accordingly, you can dispatch an event within your modules or submodules. Before dispatching an Event we need to understand when to dispatch an event. You can dispatch an event if you want to extend your logic without updating your existing code. Events can be dispatched at any time like creating, updating, loading or deleting data managed by your module. Lets explain this with an example. Take a scenario where we want other developers to interact when a new entity (taking node here) is created after submitting your custom form. 1. Create a custom module (if don’t have): # Create the module directory mkdir modules/custom/custom_logger # Create the module file touch modules/custom/custom_logger/custom_logger.info.yml2. In custom_logger.info.yml, add the following content: name: 'Custom Logger' type: module description: 'Custom module for logging events.' core_version_requirement: ^8 || ^9 || ^10 package: Custom3. Create an Event: // modules/custom/custom_logger/src/Event/CustomLoggerEvent.php namespace Drupal\custom_logger\Event; use Symfony\Component\EventDispatcher\Event; /** * Defines the custom event for the custom_logger module. */ class CustomLoggerEvent extends Event {   /**   * The node that triggered the event.   *   * @var \Drupal\node\Entity\Node   */   protected $node;   /**   * CustomLoggerEvent constructor.   *   * @param \Drupal\node\Entity\Node $node   *   The node that triggered the event.   */   public function __construct($node) {     $this->node = $node;   }   /**   * Get the node object.   *   * @return \Drupal\node\Entity\Node   *   The node.   */   public function getNode() {     return $this->node;   } }4. Dispatch the Event: Creating any entity (taking node here) and dispatching a custom event with a created entity (node) as a parameter. // modules/custom/custom_logger/src/Form/MyCustomForm.php namespace Drupal\custom_logger\Form; use Drupal\Core\Form\FormBase; use Drupal\Core\Form\FormStateInterface; use Drupal\custom_logger\Event\CustomLoggerEvent; use Symfony\Component\DependencyInjection\ContainerInterface; use Symfony\Component\EventDispatcher\EventDispatcherInterface; /** * My custom form. */ class MyCustomForm extends FormBase {   /**   * The event dispatcher service.   *   * @var \Symfony\Component\EventDispatcher\EventDispatcherInterface   */   protected $eventDispatcher;   /**   * Constructs a new MyCustomForm object.   *   * @param \Symfony\Component\EventDispatcher\EventDispatcherInterface $event_dispatcher   *   The event dispatcher service.   */   public function __construct(EventDispatcherInterface $event_dispatcher) {     $this->eventDispatcher = $event_dispatcher;   }   /**   * {@inheritdoc}   */   public static function create(ContainerInterface $container) {     return new static(       $container->get('event_dispatcher')     );   }   /**   * {@inheritdoc}   */   public function getFormId() {     return 'my_custom_form';   }   /**   * {@inheritdoc}   */   public function buildForm(array $form, FormStateInterface $form_state) {     // Build your form elements here.     return $form;   }   /**   * {@inheritdoc}   */   public function submitForm(array &$form, FormStateInterface $form_state) {     // Process form submission and create a new node.     // ...     // Dispatch the custom event.     $node = $this->getCreatedNode(); // Implement this method based on your use case.     $event = new CustomLoggerEvent($node);     $this->eventDispatcher->dispatch('custom_logger.event', $event);     // Perform any additional actions after the event is dispatched.   } }5. React to Event: Now other modules or parts of the application can now subscribe to the custom_logger.event with the created node as a parameter. What are Hooks in Drupal Hooks allow modules to alter and extend the existing behavior of Drupal Core or any other module without modifying existing code. Read about update and post update hooks to update your Drupal site in this article. Characteristics Drupal's traditional way of allowing modules to interact with the system. Code can be improved or modified independently and incrementally. Hooks are predefined functions with specific names that Drupal core or modules call at various points during execution. Developers implement these functions in their modules to extend or alter default behavior. Very efficient and easy to implement. Types of Hooks Hooks that react to events: like when the user gets logged in and some action needs to be performed. This is very similar to Events. This is invoked when specific actions are performed. For example hook_user_cancel(). Hooks that answer questions: like “info hooks”. These are invoked when some component is gathering information about a particular topic. These hooks return arrays whose structure and values are determined in the hook definition. For example, see user module hook user_toolbar() that adds links to the common user account page to the Toolbar. Note: In  Drupal 8 or further versions this is generally handled by the plugin system. Therefore there are very few hooks present now than in Drupal 7. Hooks that alter existing data: Alter hooks are generally suffixed with alter. These are invoked to allow modules to alter the existing code. For example: hook_form_alter(). Example: Drupal core hook: hook_form_alter() Scenario: Modifying a form defined by another module. // my_module.module function my_module_form_alter(&$form, \Drupal\Core\Form\FormStateInterface $form_state, $form_id) {   // Custom logic to alter the form. }Discover existing Hooks Hooks can be defined by any of contrib or custom modules, even though there are some hooks invoked by Drupal core subsystems like Form API that are always present. This can be a little tricky sometime to find out what hooks are available and which ones to implement. There are different ways to discover existing hooks: Look for the hook definitions in *.api.php files contained either in Drupal core or any contributed modules. You can use your IDE to search for functions whose name starts with hook_. You can get a complete list of hooks here. Another way is to use drush command that will five you a list of all implementations of a specific hook. drush fn-hook help #Alias of drush devel:hookInvoke a New Hook To allow other developers to modify or extends our feature you should invoke a hook or alternatively dispatch an event. This can be done on any action like creating, deleting, updating or event when we receive or push some data through API. Hooks are invoked using the module_handler \Drupal::moduleHandler() services. Hooks can be invoked in different ways: Execute the hook in every module that implements it: ModuleHandler::invokeAll() Execute the hook per-module, usually by looping over a list of enabled modules: ModuleHandler::invoke() Call an alter allowing for alteration of existing data structures using ModuleHandler::alter(). Define a new hook To define a new hook you should do the following: 1. Choose a unique name for your hook 2. Document your hook Hooks are documented in a {MODULE_NAME}.api.php file: // custom_hooks.api.php /** * Define a custom hook for reacting to specific events. * * This hook is invoked when a certain event occurs in the system. * Modules can implement this hook to perform additional actions in response to the event. * * @param string $param1 *   An example parameter for the hook. * @param array $param2 *   Another example parameter for the hook. * * @ingroup custom_hooks_hooks */ function hook_custom_event($param1, array $param2) {   // Your custom hook logic here. }3. Invoking your hook in your module’s code: /** * Implements hook_ENTITY_TYPE_view(). */ function hooks_example_node_view(array &$build, EntityInterface $entity, EntityViewDisplayInterface $display, $view_mode) {   // Invoke a hook to alert other modules that the count was updated.   $module_handler = \Drupal::moduleHandler();   // In this example we're invoking hook_custom_event()             $module_handler->invokeAll('custom_event', [$entity]); }When to Use Events or Hooks Events: Prefer events when actions need to be decoupled or when integrating with Symfony components. Hooks: Use hooks for more straightforward modifications or when interacting with the Drupal core and contributed modules. Pros and Cons of using Events VS Hooks Events: Pros: Decoupling, better organization, and Symfony integration. Cons: Slightly steeper learning curve for those unfamiliar with Symfony. Hooks: Pros: Simplicity, well-established in Drupal, easier for Drupal – specific tasks. Cons: Tighter coupling, less organization in larger projects. Final Thoughts Understanding Events and Hooks is extremely crucial for effective Drupal development. While hooks are traditional Drupal mechanisms for extending functionality, events have been introduced since Drupal 8 as part of it’s event-driven architecture. Choosing the right mechanism should be based on the complexity and nature of the Drupal project. Got a Drupal project in mind that requires a 100% Drupal-focused expertise? We’d love to talk to you!
Categories: FLOSS Project Planets

Hynek Schlawack: You Can Build Portable Binaries of Python Applications

Planet Python - Mon, 2024-03-18 20:00

Contrary to popular belief, it’s possible to ship portable executables of Python applications without sending your users to Python packaging hell.

Categories: FLOSS Project Planets

Acquia Developer Portal Blog: DevOps: The Gravity of the Modern Web Cosmos

Planet Drupal - Mon, 2024-03-18 18:59

Some of the illustrations in this article are created by: Martin Anderson-Clutz and Thomas Scola.

🪐Imagine the modern web as an endless universe, a cosmos where websites and applications orbit around users like planets around stars, each vying for attention in the vast expanse of digital space. In this universe, for development teams, DevOps emerges as the force of gravity that holds everything together, an essential principle that ensures these digital worlds don't just float aimlessly but evolve, adapt, and thrive. 

Categories: FLOSS Project Planets

Joey Hess: policy on adding AI generated content to my software projects

Planet Debian - Mon, 2024-03-18 16:54

I am eager to incorporate your AI generated code into my software. Really!

I want to facilitate making the process as easy as possible. You're already using an AI to do most of the hard lifting, so why make the last step hard? To that end, I skip my usually extensive code review process for your AI generated code submissions. Anything goes as long as it compiles!

Please do remember to include "(AI generated)" in the description of your changes (at the top), so I know to skip my usual review process.

Also be sure to sign off to the standard Developer Certificate of Origin so I know you attest that you own the code that you generated. When making a git commit, you can do that by using the --signoff option.

I do make some small modifications to AI generated submissions. For example, maybe you used AI to write this code:

+ // Fast inverse square root + float fast_rsqrt( float number ) + { + float x2 = number * 0.5F; + float y = number; + long i = * ( long * ) &y; + i = 0x5f3659df - ( i >> 1 ); + y = * ( float * ) &i; + return (y * ( 1.5F - ( x2 * y * y ) )); + } ... - foo = rsqrt(bar) + foo = fast_rsqrt(bar)

Before AI, only a genious like John Carmack could write anything close to this, and now you've generated it with some simple prompts to an AI. So of course I will accept your patch. But as part of my QA process, I might modify it so the new code is not run all the time. Let's only run it on leap days to start with. As we know, leap day is February 30th, so I'll modify your patch like this:

- foo = rsqrt(bar) + time_t s = time(NULL); + if (localtime(&s)->tm_mday == 30 && localtime(&s)->tm_mon == 2) + foo = fast_rsqrt(bar); + else + foo = rsqrt(bar);

Despite my minor modifications, you did the work (with AI!) and so you deserve the credit, so I'll keep you listed as the author.

Congrats, you made the world better!

PS: Of course, the other reason I don't review AI generated code is that I simply don't have time and have to prioritize reviewing code written by falliable humans. Unfortunately, this does mean that if you submit AI generated code that is not clearly marked as such, and use my limited reviewing time, I won't have time to review other submissions from you in the future. I will still accept all your botshit submissions though!

PPS: Ignore the haters who claim that botshit makes AIs that get trained on it less effective. Studies like this one just aren't believable. I asked Bing to summarize it and it said not to worry about it!

Categories: FLOSS Project Planets

Data School: Jupyter &amp; IPython terminology explained 💡

Planet Python - Mon, 2024-03-18 14:58

Are you trying to understand the differences between Jupyter Notebook, JupyterLab, IPython, Colab, and other related terms? You&aposre in the right place!

I&aposll explain by walking through a brief history of the IPython and Jupyter projects:

IPython

IPython was first released in 2006 as an "interactive" version of the Python shell. Whereas the Python shell uses the >>> prompt, you can recognize IPython from its use of In [1] and Out [1] notation to indicate input/output and line numbers:

IPython includes many features not present in the default Python shell, such as object introspection, "magic" commands, system shell access, and more.

IPython Notebook

In 2011, the IPython Notebook was released. It was known as a "computational notebook" because it allowed you to weave together code, plots, and narrative text into a single document:

It was called the IPython Notebook (and not the Python Notebook) because it used IPython as the "kernel", which is the language-specific process that runs the code in a notebook.

Jupyter Notebook

In 2015, the IPython Notebook introduced support for programming languages other than Python.

Also in 2015, IPython split into two projects: IPython (for Python-specific components) and Jupyter (for language-agnostic components).

As part of that split, the IPython Notebook was renamed the Jupyter Notebook. The name "Jupyter" was inspired by the open languages of science: Julia, Python, and R:

To be clear, "Jupyter Notebook" was the name of both the coding environment and the files created by that environment. In other words, you would open "the Jupyter Notebook" to create "a Jupyter notebook".

Jupyter notebook files used the extension ".ipynb", which was the extension (and file format) originally created for IPython notebooks.

JupyterLab

At this point, the Jupyter Notebook was a lightweight coding environment, with far less features than a traditional IDE (integrated development environment).

In 2018, JupyterLab (one word) was released as a more full-featured alternative to the Jupyter Notebook:

Notebooks created within JupyterLab are still called "Jupyter notebooks", they still use the extension ".ipynb", and they&aposre compatible with notebooks created by the Jupyter Notebook.

JupyterLab was originally designed to replace the Jupyter Notebook environment. However, due to the continued popularity of the "classic" Notebook environment, JupyterLab and Jupyter Notebook continue to be developed as separate applications (as of 2024).

Summary
  • The Jupyter Notebook is a lightweight coding environment for creating and editing Jupyter notebooks.
  • JupyterLab is more full-featured IDE for creating and editing Jupyter notebooks.
  • IPython is the Python kernel for Jupyter Notebook and JupyterLab, and is also a standalone Python shell. IPython is the reason that magic commands and other enhancements are available within Jupyter Notebook and JupyterLab.
  • Jupyter notebooks are computational documents that can contain code, plots, and text. They use the extension ".ipynb" and are compatible with both the Jupyter Notebook and JupyterLab environments.

Here are a few related terms that I didn&apost mention above:

  • JupyterLab Desktop is a cross-platform desktop application that allows you to create and manage multiple JupyterLab sessions and Python environments.
  • JupyterLite is a JupyterLab distribution that runs entirely in the browser, without you having to launch a Jupyter server from a terminal.
  • Google Colab, Kaggle Code, and Deepnote are a few of the many web-based services that provide a Jupyter-like interface for creating notebooks that are compatible with Jupyter. (More specifically, they can import and export files that use the ".ipynb" format.)

Are there any other Jupyter-related terms you want me to explain? Please let me know if the comments! &#x1F447;

Categories: FLOSS Project Planets

Talking Drupal: Talking Drupal #442 - Mercury Editor

Planet Drupal - Mon, 2024-03-18 14:00

Today we are talking about Mercury Editor, What it does, and how it could change your editorial life with guest Justin Toupin. We’ll also cover Webform Protected Downloads as our module of the week.

For show notes visit: www.talkingDrupal.com/442

Topics
  • What is Mercury Editor
  • What is powering Mercury Editor
  • Do you see any risk building on top of Paragraphs
  • Does Mercury Editor cost anything
  • Can companies hire Aten to add features
  • What are some key features
  • What makes Mercury Editor unique
  • How stable is the content
  • What happens if Paragraphs stops being supported
  • How can the community help
Resources Guests

Justin Toupin - atendesigngroup.com justin2pin

Hosts

Nic Laflin - nLighteneddevelopment.com nicxvan John Picozzi - epam.com johnpicozzi Anna Mykhailova - kalamuna.com amykhailova

MOTW Correspondent

Martin Anderson-Clutz - mandclu

  • Brief description:
    • Have you ever wanted to have downloadable content on your website, only available to visitors who have filled out a webform? There’s a module for that.
  • Module name/project name:
  • Brief history
    • How old: created in Sep 2010 by berliner, but the most recent releases are by james.williams of Computer Minds
    • Versions available: 7.x-1.1 and 8.x-1.0-alpha2 versions available, the latter of which works with Drupal 9 and 10
  • Maintainership
    • Actively maintained, the latest release was a week ago
    • Security coverage
    • Introductory blog linked on the project page
    • Number of open issues: 18 open issues, none of which are bugs against the current branch
  • Usage stats:
    • 804 sites
  • Module features and usage
    • Having thought leadership content like white papers or reports gated behind a lead capture form is a common pattern for websites, and this module is designed to make that easy to set up
    • You use the module by adding a handler to your webform, similar to triggering an email send
    • In the configuration for your webform protected download handler you have options for how much verification you want for the download link, whether or not the link should expire after a period of time, and so on, in addition to uploading one or more files that can be downloaded by people who submit the webform
    • The module provides tokens for the download URLs, so you can easily include them in a submission confirmation message or email
Categories: FLOSS Project Planets

Open Source AI Definition – Weekly update Mar 18

Open Source Initiative - Mon, 2024-03-18 12:58
Comments on draft 0.0.6 from the forum
  • Point raised by participant that training data has been listed both as optional and a precondition. This might cause confusion as it is unclear whether we should have the right to access training data or know what training data was used for the model
  • To contribute, read the new draft here 
Moving on to next steps! Town hall this Friday Comments on definitions under “What is open source AI? Still strong debate about access to training data 
  • There is a fear that this will harm the ecosystem in the long run, as the original work of the model never can be “forked” to improve the model itself.
Categories: FLOSS Research

The Drop Times: MidCamp 2024 Innovates with Unconference Format and Training

Planet Drupal - Mon, 2024-03-18 12:44
MidCamp 2024, set to take place at DePaul University, promises innovative formats and a focus shift towards AI, marking a significant evolution in the annual Midwest Drupal Camp's history.
Categories: FLOSS Project Planets

The Drop Times: Tracking Drupal's Global Footprint

Planet Drupal - Mon, 2024-03-18 12:44

Dear Readers,

Drupal has seen widespread adoption worldwide, a testament to its flexibility, security, and scalability. Renowned for its modular architecture and strong community support, Drupal empowers developers, businesses, and governments to create and manage diverse digital experiences. From educational institutions and media outlets to non-profit organizations and governmental agencies, the platform's extensive capabilities allow for the customization and integration necessary to meet complex digital needs.

An endeavor to track Drupal's usage across various industry sectors represents a need of the moment for the Drupal community and is put forth by Paul Johnson. The project aims to showcase Drupal's diverse applicability and strengthen its shared knowledge base by gathering detailed resources and data. This concerted effort, aimed at illuminating Drupal's footprint, will offer insights into the platform's impact and success across different domains. It sets the stage for a deeper understanding of the community's achievements and challenges, guiding them toward more strategic, evidence-based decisions within the digital ecosystem.

The DropTimes [TDT] has already published a comprehensive study on Drupal's usage in prominent educational institutions worldwide, which can set the right example for this new project. Grzegorz Pietrzak's study on global city website trends analysis further underscores Drupal's influence across diverse sectors, amplifying the platform's prominence within various industry verticals. The full study is published on our website; read it here

As the community delves into the specifics of these endeavors and their implications, it is crucial to remain connected and informed. With that, welcome to the last week's most important content covered by The DropTimes.

Performance is the cornerstone of user experience and operational efficiency in web development. Learn about the genesis, capabilities, and transformative potential of Gander, the automated performance testing framework for Drupal, as elucidated by Nathaniel Catchpole and Janez Urevc in Elma John's latest article

We have a new addition to our Spotlights channel, Zoocha, a leading Drupal Development Agency in the UK. The article discusses the intricate world of Zoocha, bringing to light the company's journey, strategies, and outlook with contributions from Will Huggins, the CEO, and the Zoocha team.

Dries Buytaert, the founder of Drupal, visited Japan for the first time in nearly eight years. He presented the latest developments in Drupal and associated web technologies at the Drupal Meetup. Tokyo. Read the article by Kazima Abbas for a comprehensive overview of the Tokyo Meetup and its significance. 

Interestingly, the recent Drupal Meetup, organized by Valuebound in collaboration with the Drupal Association, marked a significant event for Drupal enthusiasts and professionals. The event, held at Valuebound's office in Bangalore, featured a special meet-and-greet session with Tim Doyle, the CEO of the Drupal Association. As an after-note, Tim wrote, that "Drupal is alive and well in India", owing to the enthusiastic Drupal Community in India.

The Drupal Community has many events to celebrate this week, but MidCamp 2024 and DrupalSouth Sydney 2024 tops the list. As the Media Partner for both events, The DropTimes is determined to provide its readers timely updates. A complete list of events for the week is available here.

The upcoming DrupalCamp Burkina Faso 2024 is close to reaching its funding target, needing just $2,000 more to facilitate the largest DrupalCamp event in West Africa.  DrupalCamp Asheville is now open for speaker submissions, inviting seasoned presenters and newcomers to share their expertise and insights. Also, the last chance to submit proposals for Stanford WebCamp 2024 ends on or before March 25, 2024. The DrupalSouth Splash Awards 2024 shortlist, sponsored by Ironstar, has been announced ahead of the event. 

Imre Gmelig Meijling, CEO of React Online Digital Agency in The Netherlands, has been introduced as one of the newest members elected to the Drupal Association Board. Alex Moreno has launched a comprehensive guide to enhance impactful contributions to Drupal and the Drupal Association. The contributor guide lists strategic initiatives, crucial issues, and essential modules, offering contributors an avenue to make significant impacts within the Drupal community.

The Drupal community has recently seen the introduction of a new module, Bill of Lading, created by Jeff Greenberg. This module simplifies site management by generating a comprehensive list of Drupal structures using a new Drush command, 'bol.' QuantCDN co-founder Kristen Pol has announced a major update to their Drupal static site generator, enhancing Drupal 9 and 10 integration with new features, including visibility into Quant metadata directly within the Drupal platform.

We acknowledge that there are more stories to share. However, due to constraints in selection, we must pause further exploration for now.

To get timely updates, follow us on LinkedIn, Twitter and Facebook. Also, join us on Drupal Slack at #thedroptimes.

Thank you,

Sincerely
Alka Elizabeth
Sub-editor, TheDropTimes.

Categories: FLOSS Project Planets

The Drop Times: Drupal Page Builders—Part 1: Paragraph-Based Solutions

Planet Drupal - Mon, 2024-03-18 12:44
Dive into the changing dynamics of Drupal's page-building features with André Angelantoni's new series on The DropTimes. Discover the progression of Drupal's page layout options as the author begins exploring Paragraph-Based Solutions. Gain historical insights, learn about the shift towards Layout Builder, and examine the Paragraphs module and related tools like Layout Paragraphs and Mercury Editor. Anticipate detailed discussions on contributed modules, alternative solutions, and potent distributions in the forthcoming parts of the series. Ideal for developers and content managers seeking to upgrade their Drupal 10+ projects.
Categories: FLOSS Project Planets

Simon Josefsson: Apt archive mirrors in Git-LFS

Planet Debian - Mon, 2024-03-18 12:15

My effort to improve transparency and confidence of public apt archives continues. I started to work on this in “Apt Archive Transparency” in which I mention the debdistget project in passing. Debdistget is responsible for mirroring index files for some public apt archives. I’ve realized that having a publicly auditable and preserved mirror of the apt repositories is central to being able to do apt transparency work, so the debdistget project has become more central to my project than I thought. Currently I track Trisquel, PureOS, Gnuinos and their upstreams Ubuntu, Debian and Devuan.

Debdistget download Release/Package/Sources files and store them in a git repository published on GitLab. Due to size constraints, it uses two repositories: one for the Release/InRelease files (which are small) and one that also include the Package/Sources files (which are large). See for example the repository for Trisquel release files and the Trisquel package/sources files. Repositories for all distributions can be found in debdistutils’ archives GitLab sub-group.

The reason for splitting into two repositories was that the git repository for the combined files become large, and that some of my use-cases only needed the release files. Currently the repositories with packages (which contain a couple of months worth of data now) are 9GB for Ubuntu, 2.5GB for Trisquel/Debian/PureOS, 970MB for Devuan and 450MB for Gnuinos. The repository size is correlated to the size of the archive (for the initial import) plus the frequency and size of updates. Ubuntu’s use of Apt Phased Updates (which triggers a higher churn of Packages file modifications) appears to be the primary reason for its larger size.

Working with large Git repositories is inefficient and the GitLab CI/CD jobs generate quite some network traffic downloading the git repository over and over again. The most heavy user is the debdistdiff project that download all distribution package repositories to do diff operations on the package lists between distributions. The daily job takes around 80 minutes to run, with the majority of time is spent on downloading the archives. Yes I know I could look into runner-side caching but I dislike complexity caused by caching.

Fortunately not all use-cases requires the package files. The debdistcanary project only needs the Release/InRelease files, in order to commit signatures to the Sigstore and Sigsum transparency logs. These jobs still run fairly quickly, but watching the repository size growth worries me. Currently these repositories are at Debian 440MB, PureOS 130MB, Ubuntu/Devuan 90MB, Trisquel 12MB, Gnuinos 2MB. Here I believe the main size correlation is update frequency, and Debian is large because I track the volatile unstable.

So I hit a scalability end with my first approach. A couple of months ago I “solved” this by discarding and resetting these archival repositories. The GitLab CI/CD jobs were fast again and all was well. However this meant discarding precious historic information. A couple of days ago I was reaching the limits of practicality again, and started to explore ways to fix this. I like having data stored in git (it allows easy integration with software integrity tools such as GnuPG and Sigstore, and the git log provides a kind of temporal ordering of data), so it felt like giving up on nice properties to use a traditional database with on-disk approach. So I started to learn about Git-LFS and understanding that it was able to handle multi-GB worth of data that looked promising.

Fairly quickly I scripted up a GitLab CI/CD job that incrementally update the Release/Package/Sources files in a git repository that uses Git-LFS to store all the files. The repository size is now at Ubuntu 650kb, Debian 300kb, Trisquel 50kb, Devuan 250kb, PureOS 172kb and Gnuinos 17kb. As can be expected, jobs are quick to clone the git archives: debdistdiff pipelines went from a run-time of 80 minutes down to 10 minutes which more reasonable correlate with the archive size and CPU run-time.

The LFS storage size for those repositories are at Ubuntu 15GB, Debian 8GB, Trisquel 1.7GB, Devuan 1.1GB, PureOS/Gnuinos 420MB. This is for a couple of days worth of data. It seems native Git is better at compressing/deduplicating data than Git-LFS is: the combined size for Ubuntu is already 15GB for a couple of days data compared to 8GB for a couple of months worth of data with pure Git. This may be a sub-optimal implementation of Git-LFS in GitLab but it does worry me that this new approach will be difficult to scale too. At some level the difference is understandable, Git-LFS probably store two different Packages files — around 90MB each for Trisquel — as two 90MB files, but native Git would store it as one compressed version of the 90MB file and one relatively small patch to turn the old files into the next file. So the Git-LFS approach surprisingly scale less well for overall storage-size. Still, the original repository is much smaller, and you usually don’t have to pull all LFS files anyway. So it is net win.

Throughout this work, I kept thinking about how my approach relates to Debian’s snapshot service. Ultimately what I would want is a combination of these two services. To have a good foundation to do transparency work I would want to have a collection of all Release/Packages/Sources files ever published, and ultimately also the source code and binaries. While it makes sense to start on the latest stable releases of distributions, this effort should scale backwards in time as well. For reproducing binaries from source code, I need to be able to securely find earlier versions of binary packages used for rebuilds. So I need to import all the Release/Packages/Sources packages from snapshot into my repositories. The latency to retrieve files from that server is slow so I haven’t been able to find an efficient/parallelized way to download the files. If I’m able to finish this, I would have confidence that my new Git-LFS based approach to store these files will scale over many years to come. This remains to be seen. Perhaps the repository has to be split up per release or per architecture or similar.

Another factor is storage costs. While the git repository size for a Git-LFS based repository with files from several years may be possible to sustain, the Git-LFS storage size surely won’t be. It seems GitLab charges the same for files in repositories and in Git-LFS, and it is around $500 per 100GB per year. It may be possible to setup a separate Git-LFS backend not hosted at GitLab to serve the LFS files. Does anyone know of a suitable server implementation for this? I had a quick look at the Git-LFS implementation list and it seems the closest reasonable approach would be to setup the Gitea-clone Forgejo as a self-hosted server. Perhaps a cloud storage approach a’la S3 is the way to go? The cost to host this on GitLab will be manageable for up to ~1TB ($5000/year) but scaling it to storing say 500TB of data would mean an yearly fee of $2.5M which seems like poor value for the money.

I realized that ultimately I would want a git repository locally with the entire content of all apt archives, including their binary and source packages, ever published. The storage requirements for a service like snapshot (~300TB of data?) is today not prohibitly expensive: 20TB disks are $500 a piece, so a storage enclosure with 36 disks would be around $18.000 for 720TB and using RAID1 means 360TB which is a good start. While I have heard about ~TB-sized Git-LFS repositories, would Git-LFS scale to 1PB? Perhaps the size of a git repository with multi-millions number of Git-LFS pointer files will become unmanageable? To get started on this approach, I decided to import a mirror of Debian’s bookworm for amd64 into a Git-LFS repository. That is around 175GB so reasonable cheap to host even on GitLab ($1000/year for 200GB). Having this repository publicly available will make it possible to write software that uses this approach (e.g., porting debdistreproduce), to find out if this is useful and if it could scale. Distributing the apt repository via Git-LFS would also enable other interesting ideas to protecting the data. Consider configuring apt to use a local file:// URL to this git repository, and verifying the git checkout using some method similar to Guix’s approach to trusting git content or Sigstore’s gitsign.

A naive push of the 175GB archive in a single git commit ran into pack
size limitations:

remote: fatal: pack exceeds maximum allowed size (4.88 GiB)

however breaking up the commit into smaller commits for parts of the
archive made it possible to push the entire archive. Here are the
commands to create this repository:

git init
git lfs install
git lfs track 'dists/**' 'pool/**'
git add .gitattributes
git commit -m"Add Git-LFS track attributes." .gitattributes
time debmirror --method=rsync --host ftp.se.debian.org --root :debian --arch=amd64 --source --dist=bookworm,bookworm-updates --section=main --verbose --diff=none --keyring /usr/share/keyrings/debian-archive-keyring.gpg --ignore .git .
git add dists project
git commit -m"Add." -a
git remote add origin git@gitlab.com:debdistutils/archives/debian/mirror.git
git push --set-upstream origin --all
for d in pool//; do
echo $d;
time git add $d;
git commit -m"Add $d." -a
git push
done

The resulting repository size is around 27MB with Git LFS object storage around 174GB. I think this approach would scale to handle all architectures for one release, but working with a single git repository for all releases for all architectures may lead to a too large git repository (>1GB). So maybe one repository per release? These repositories could also be split up on a subset of pool/ files, or there could be one repository per release per architecture or sources.

Finally, I have concerns about using SHA1 for identifying objects. It seems both Git and Debian’s snapshot service is currently using SHA1. For Git there is SHA-256 transition and it seems GitLab is working on support for SHA256-based repositories. For serious long-term deployment of these concepts, it would be nice to go for SHA256 identifiers directly. Git-LFS already uses SHA256 but Git internally uses SHA1 as does the Debian snapshot service.

What do you think? Happy Hacking!

Categories: FLOSS Project Planets

Pages