FLOSS Project Planets

Real Python: Advent of Code: Solving Your Puzzles With Python

Planet Python - Wed, 2021-12-01 09:00

Advent of Code is an online Advent calendar where you’ll find new programming puzzles offered each day from December 1st to the 25th. While you can solve the puzzles at any time, the excitement when new puzzles unlock is really something special. You can participate in Advent of Code in any programming language—including Python!

This tutorial will help you get started with solving puzzles and earning your first golden stars.

In this tutorial, you’ll learn:

  • What an online Advent calendar is
  • How solving puzzles can advance your programming skills
  • How you can participate in the Advent of Code
  • How you can organize your code and tests when solving Advent of Code puzzles
  • How test-driven development can be used when solving puzzles

Advent of Code puzzles are designed to be approachable by anyone with an interest in problem-solving. You don’t need a heavy computer science background to participate. Instead, Advent of Code is a great arena for learning new skills and testing out new features of Python.

Free Bonus: Click here to get a Python Cheat Sheet and learn the basics of Python 3, like working with data types, dictionaries, lists, and Python functions.

Puzzling in Programming?

Working on puzzles may seem like a waste of your available programming time. After all, it seems like you’re not really producing anything useful and you’re not advancing your current projects forward.

However, there are several advantages to taking some time off to practice with programming puzzles:

  • Programming puzzles are usually better specified and more contained than your regular job tasks. They offer you the chance to practice logical thinking on problems that are less complex than the ones you typically need to handle in your day job.

  • You can often challenge yourself with several similar puzzles. This allows you to build procedural memory, much like muscle memory, and get experience with structuring certain kinds of code.

  • Puzzles are often designed with an eye towards a solution. They allow you to learn about and apply algorithms that are tried and tested and are an important part of any programmer’s toolbox.

  • For some puzzle solutions, even the greatest supercomputers can be too slow if the algorithm is inefficient. You can analyze the performance of your solution and get experience to help you understand when a straightforward method is fast enough and when a more optimized procedure is necessary.

  • Most programming languages are well-suited for solving programming puzzles. This gives you a great opportunity to compare different programming languages for different tasks. Puzzles are also a great way for getting to know a new programming language or trying out some of the newest features of your favorite language.

On top of all of this, challenging yourself with a programming puzzle is often pretty fun! When you add it all up, setting aside some time for puzzles can be very rewarding.

Exploring Options for Solving Programming Puzzles Online

Luckily, there are many websites where you can find programming puzzles and try to solve them. There are often differences in the kinds of problems these websites present, how you submit your solutions, and what kind of feedback and community the sites can offer. You should therefore take some time to look around and find those that appeal the most to you.

In this tutorial, you’ll learn about Advent of Code, including what kind of puzzles you can find there and which tools and tricks you can employ to solve them. However, there are also other places where you can get started solving programming puzzles:

  • Exercism has learning tracks in many different programming languages. Each learning track offers small tutorials about different programming concepts, coding challenges, and mentors that give you feedback on your solutions.

  • Project Euler has been around for a long time. The site offers hundreds of puzzles, usually formulated as math problems. You can solve the problems in any programming language, and once you’ve solved a puzzle, you get access to a community thread where you can discuss your solution with others.

  • Code Wars offers tons of coding challenges, which they call katas. You can solve puzzles in many different programming languages with their built-in editor and automated tests. Afterward, you can compare your solutions to others’ and discuss strategies in the forums.

  • HackerRank has great features if you’re looking for a job. They offer certifications in many different skills, including problem-solving and Python programming, as well as a job board that lets you show off your puzzle-solving skills as part of your job applications.

There are many other sites available where you can practice your puzzle-solving skills. In the rest of this tutorial, you’ll focus on what Advent of Code has to offer.

Preparing for Advent of Code: 25 Fresh Puzzles for Christmas

It’s time for Advent of Code! It was started by Eric Wastl in 2015. Since then, a new advent calendar of twenty-five new programming puzzles has been published every December. The puzzles have gotten more and more popular over the years. More than 170,000 people have solved at least one of the puzzles from 2020.

Note: Traditionally, an Advent calendar is a calendar used to count the days of Advent while waiting for Christmas. Over the years, Advent calendars have become more commercial and have lost some of their Christian connection.

Most Advent calendars start on December 1st and end on December 24th, Christmas Eve, or December 25th, Christmas Day. Nowadays, there are all kinds of Advent calendars available, including LEGO calendars, tea calendars, and cosmetics calendars.

In traditional Advent calendars, you open one door every day to reveal what’s inside. Advent of Code mimics this by letting you open one puzzle each day from December 1st to December 25th. For each puzzle you solve, you’ll earn golden stars that are yours to keep.

In this section, you’ll get more familiar with Advent of Code and see a glimpse of your first puzzle. Later, you’ll look at the details of how you can solve these puzzles and practice solving a few of the puzzles yourself.

Advent of Code Puzzles

Advent of Code is an online Advent calendar where a new puzzle is published every day from December 1st to December 25th. Each puzzle becomes available at midnight, US Eastern Time. An Advent of Code puzzle has a few typical characteristics:

  • Each puzzle consists of two parts, but the second part isn’t revealed until you finish the first part.
  • You’ll earn one golden star (⭐) for each part that you finish. This means you can earn two stars per day and fifty stars if you solve all the puzzles for one year.
  • The puzzle is the same for everyone, but you need to solve it based on personalized input that you get from the Advent of Code site. This means that your answer to a puzzle will be different from someone else’s, even if you use the same code to calculate it.

You can participate in a global race to be the first to solve each puzzle. However, this is usually pretty crowded with highly skilled, competitive programmers. Advent of Code is probably going to be more fun if you use it as practice for yourself or if you challenge your friends and coworkers to a small, friendly competition.

Read the full article at https://realpython.com/python-advent-of-code/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Tag1 Consulting: Michael Schmid Talks about Contributing to Drupal -20 Years of Drupal miniseries

Planet Drupal - Wed, 2021-12-01 08:15

Michael Schmid is another long term Drupal community member and major contributor to multiple factors of the community. As a Drupal core contributor and Group CTO of Amazee Labs, Michael is no stranger to all of the things that have made Drupal successful in its 20 years.

Read more lynette@tag1co… Wed, 12/01/2021 - 05:15
Categories: FLOSS Project Planets

Dirk Eddelbuettel: digest 0.6.29 on CRAN: Package Maintenance

Planet Debian - Wed, 2021-12-01 07:31

Release 0.6.29 of the digest package arrived at CRAN earlier today, and will be uploaded Debian shortly.

digest creates hash digests of arbitrary R objects (using the md5, sha-1, sha-256, sha-512, crc32, xxhash32, xxhash64, murmur32, spookyhash, and blake3 algorithms) permitting easy comparison of R language objects. It is a mature and widely-used as many tasks may involve caching of objects for which it provides convenient general-purpose hash key generation.

This release only contains two smaller internal changes. We removed the YAML file (and badge) for the disgraced former continuous integration service we shall not name (yet that we all used to use). And we converted the vignette from using the minidown package to the (fairly new) simplermarkdown package which is so much more appropriate for our use of the minimal water.css style.

My CRANberries provides the usual summary of changes to the previous version. For questions or comments use the issue tracker off the GitHub repo.

If you like this or other open-source work I do, you can now sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Codementor: Best Free And Paid Air Quality API 2022

Planet Python - Wed, 2021-12-01 07:26
APIs for air quality may be used to detect air pollution, carbon monoxide, and the air quality index, among other things. You may get current, projected, and historical information. These APIs may be used to monitor the levels of key air pollutants from a variety of data sources.
Categories: FLOSS Project Planets

Floating Panel Merge Request Feedback: Will It Land?

Planet KDE - Wed, 2021-12-01 06:07
💸💸 Help me contribute to KDE and do these videos: 💸💸 Patreon: https://www.patreon.com/niccolove Youtube: https://www.youtube.com/channel/UCONH73CdRXUjlh3-DdLGCPw/join Paypal: https://paypal.me/niccolove Stay in the loop: https://t.me/veggeroblog My website is https://niccolo.venerandi.com and if you want to contact me, my telegram handle is [at] veggero.
Categories: FLOSS Project Planets

Tryton News: Newsletter December 2021

Planet Python - Wed, 2021-12-01 02:00

After release 6.2 we have already resumed development with some cleaning and performance improvements.

Changes for the User

The early planning for stock quantities now takes internal incoming requests into account.

You can now define a delay after which an email notification will be sent.

The reset password email now displays how long it is valid instead of the UTC expiry date and time.

The record names for addresses now contains all the street lines instead of just the first one.

Changes for the System Administrator

It is now possible to send test emails without a database.

Changes for the Developer

The PostgreSQL back-end now uses the JSONB type by default to store the value of Dict fields.

We activated support for window functions when using SQLite versions greater than 3.25.0.

We use RENAME and DROP COLUMN when using SQLite versions that support them.

It is now possible to customize the states of the shipments with which a sale can be grouped.

The button to generate SEPA messages has been prefixed by sepa avoiding any name clashes with other modules.

It is possible to mark the getter of a Function field as not depending on the context. By doing this it can be stored in the transaction cache in readonly transactions.

The cache of instances passed to Function field getters are now prefilled with data already calculated for the instance. The order in which the Function fields are called can be tuned to take advantage of this behavior when the calculations depend on other Function fields from the same record.

1 post - 1 participant

Read full topic

Categories: FLOSS Project Planets

Russ Allbery: Review: A World Without Email

Planet Debian - Wed, 2021-12-01 00:07

Review: A World Without Email, by Cal Newport

Publisher: Portfolio/Penguin Copyright: 2021 ISBN: 0-525-53657-4 Format: Kindle Pages: 264

A World Without Email is the latest book by computer science professor and productivity writer Cal Newport. After a detour to comment on the drawbacks of social media in Digital Minimalism, Newport is back to writing about focus and concentration in the vein of Deep Work. This time, though, the topic is workplace structure and collaborative process rather than personal decisions.

This book is a bit hard for me to review because I spoiled myself for the contents by listening to a lot of Newport's podcast, where he covers the same material. I therefore didn't enjoy it as much as I otherwise would have because the ideas were familiar. I recommend the book over the podcast, though; it's tighter, more coherent, and more comprehensive.

The core contention of this book is that knowledge work (roughly, jobs where one spends significant time working on a computer processing information) has stumbled into a superficially tempting but inefficient and psychologically harmful structure that Newport calls the hyperactive hive mind. This way of organizing work is a local maxima: it feels productive, it's flexible and very easy to deploy, and most minor changes away from it make overall productivity worse. However, the incentive structure is all wrong. It prioritizes quick responses and coordination overhead over deep thinking and difficult accomplishments.

The characteristic property of the hyperactive hive mind is free-flowing, unstructured communication between co-workers. If you need something from someone else, you ask them for it and they send it to you. The "email" in the title is not intended literally; Slack and related instant messaging apps are even more deeply entrenched in the hyperactive hive mind than email is. The key property of this workflow is that most collaborative work is done by contacting other people directly via ad hoc, unstructured messages.

Newport's argument is that this workflow has multiple serious problems, not the least of which is that it makes us miserable. If you have read his previous work, you will correctly expect this to tie into his concept of deep work. Ad hoc, unstructured communication creates a constant barrage of unimportant small tasks and interrupts, most of which require several asynchronous exchanges before your brain can stop tracking the task. This creates constant context-shifting, loss of focus and competence, and background stress from ever-growing email inboxes, unread message notifications, and the semi-frantic feeling that you're forgetting something you need to do.

This is not an original observation, of course. Many authors have suggested individual ways to improve this workflow: rules about how often to check one's email, filtering approaches, task managers, and other personal systems. Newport's argument is that none of these individual approaches can address the problem due to social effects. It's all well and good to say that you should unplug from distractions and ignore requests while you concentrate, but everyone else's workflow assumes that their co-workers are responsive to ad hoc requests. Ignoring this social contract makes the job of everyone still stuck in the hyperactive hive mind harder. They won't appreciate that, and your brain will not be able to relax knowing that you're not meeting your colleagues' expectations.

In Newport's analysis, the necessary solution is a comprehensive redesign of how we do knowledge work, akin to the redesign of factory work that came with the assembly line. It's a collective problem that requires a collective solution. In other industries, organizing work for efficiency and quality is central to the job of management, but in knowledge work (for good historical reasons) employees are mostly left to organize their work on their own. That self-organization has produced a system that doesn't require centralized coordination or decisions and provides a lot of superficial flexibility, but which may be significantly inferior to a system designed for how people think and work.

Even if you find this convincing (and I think Newport makes a good case), there are reasons to be suspicious of corporations trying to make people more productive. The assembly line made manufacturing much more efficient, but it also increased the misery of workers so much that Henry Ford had to offer substantial raises to retain workers. As one of Newport's knowledge workers, I'm not enthused about that happening to my job.

Newport recognizes this and tries to address it by drawing a distinction between the workflow (how information moves between workers) and the work itself (how individual workers solve problems in their area of expertise). He argues that companies need to redesign the former, but should leave the latter to each worker. It's a nice idea, and it will probably work in industries like tech with substantial labor bargaining power. I'm more cynical about other industries.

The second half of the book is Newport's specific principles and recommendations for designing better workflows that don't rely on unstructured email. Some of this will be familiar (and underwhelming) to anyone who works in tech; Newport recommends ticket systems and thinks agile, scrum, and kanban are pointed in the right direction. But there are some other good ideas in here, such as embracing specialization.

Newport argues (with some evidence) that the drastic reduction in secretarial jobs, on the grounds that workers with computers can do the same work themselves, was a mistake. Even with new automation, this approach increased the range of tasks required in every other job. Not only was this a drain on the time of other workers, it caused more context switching, which made everyone less efficient and undermined work quality. He argues for reversing that trend: where the work cannot be automated, hire more support workers and more specialized workers in general, stop expecting everyone to be their own generalist admin, and empower support workers to create better systems rather than using the hyperactive hive mind model to answer requests.

There's more here, ranging from specifics of how to develop a structured process for a type of work to the importance of enabling sustained concentration on a task. It's a less immediately actionable book than Newport's previous writing, but I welcome the partial shift in focus to more systemic issues. Newport continues to be relentlessly apolitical, but here it feels less like he's eliding important analysis and more like he thinks the interests of workers and good employers are both served by the approach he's advocating.

I will warn that Newport leans heavily on evolutionary psychology in his argument that the hyperactive hive mind is bad for us. I think he has some good arguments about the anxiety that comes with not responding to requests from others, but I'm not sure intrusive experiments on spectacularly-unusual remnant hunter-gatherer groups, who are treated like experimental animals, are the best way of making that case. I realize this isn't Newport's research, but I think he could have made his point with more directly relevant experiments.

He also continues his obsession with the superiority of in-person conversation over written communication, and while he has a few good arguments, he has a tendency to turn them into sweeping generalizations that are directly contradicted by, well, my entire life. It would be nice if he were more willing to acknowledge that it's possible to express deep emotional nuance and complex social signaling in writing; it simply requires a level of practice and familiarity (and shared vocabulary) that's often missing from the workplace.

I was muttering a lot near the start of this book, but thankfully those sections are short, and I think the rest of his argument sits on a stronger foundation.

I hope Newport continues moving in the direction of more systemic analysis. If you enjoyed Deep Work, you will probably find A World Without Email interesting. If you're new to Newport, this is not a bad place to start, particularly if you have influence on how communication is organized in your workplace. Those who work in tech will find some bits of this less interesting, but Newport approaches the topic from a different angle than most agile books and covers a broader range if ideas.

Recommended if you like reading this sort of thing.

Rating: 7 out of 10

Categories: FLOSS Project Planets

Paul Wise: FLOSS Activities November 2021

Planet Debian - Tue, 2021-11-30 21:52

This month I didn't have any particular focus. I just worked on issues in my info bubble.

Changes Issues Review Administration
  • Debian BTS: unarchive/reopen/triage bugs for reintroduced packages
  • Debian wiki: unblock IP addresses, approve accounts
  • Respond to queries from Debian users and contributors on the mailing lists and IRC

The SPTAG, visdom, gensim, purple-discord, plac, fail2ban, uvloop work was sponsored by my employer. All other work was done on a volunteer basis.

Categories: FLOSS Project Planets

Okular PDF digital signature improvements coming "soon" thanks to NLnet

Planet KDE - Tue, 2021-11-30 18:21

Starting on January I will be working on a project named "Improve Okular digital signature support" that has received a grant from the NLnet foundation as part of the NGI Assure fund.

This will allow me to work part time on Okular (in case it's not clear I work on Okular on a "when I have time-hobby" basis right now), the planned improvements are:

1. Support for signing unsigned signatures. I know it sounds confusing, think about it like something like the old "sign here" boxes on printed paper forms.

2. Support digital signatures in the Okular Windows version

3. Make signature text support all character sets

4. Write Okular-mobile user interface to show signature status

5. Support digital signatures in the Okular Android version

But I'm hoping to squeeze some other signature related improvements in, if you have a particular favourite please leave a comment.

Thanks to NLnet for trusting me on this, and also thanks to my current employer (KDAB) for allowing me to work less hours for a few months so I can take on this project.

Categories: FLOSS Project Planets

More about those zero-dot users

Planet KDE - Tue, 2021-11-30 14:57

Yesterday’s article about KDE’s target users generated some interesting discussions about the zero-dot users. One of the most insightful comments I read was that nobody can really target zero-dot users because they operate based on memorization and habit, learning a series of cause-effect relationships: “I click/touch this picture/button, then something useful happens”–even with their smartphones! So even if GNOME and ElementaryOS might be simpler, that doesn’t really matter because it’s not much harder to memorize a random-seeming sequence of clicks or taps in a poor user interface than it is in a good one.

I think there’s a lot of truth to this perspective. We have all known zero-dot users who became quite proficient at specific tasks; maybe they learned how to to everything they needed in MS Office, Outlook, or even Photoshop.

The key detail is that these folks rely on the visual appearance and structure of the software remaining the same. When the software’s user interface changes–even for the better–they lose critical visual cues and reference points and they can’t find anything anymore.

On the desktop side, these people are the target audience for Long Term Support (LTS) distros, where the UI never changes for years at a time. This is exactly what they want because they prefer a bad yet unchanging UI to one that incrementally evolves to be better.

So I think if we want to reach these people, it will probably be done less by improving Plasma or KDE apps, but rather by being more attentive to our existing Plasma LTS offering and broadening it to encompass apps and frameworks as well. That way these other KDE products that are used alongside or underneath Plasma can benefit from more bugfixes without the UI changes of non-LTS upgrades. And we should increase the support period to 5 years or more. It’s 10 years for Red Hat Enterprise Linux! This is what’s needed to have a real LTS product and bring the zero-dot users into the fold.

However I’m not sure we have these resources right now. No KDE developer I know uses the Plasma LTS release. Working on old crappy code isn’t any fun. Backporting fixes is a thankless task. I think we would probably have to pay someone to be the full-time LTS developer-and-backporter if we wanted to have an LTS product worth of its name. It will most likely need to be on the back burner for a while. Hence, focusing on the one-or-more-dots users for the time being.

Categories: FLOSS Project Planets

Security public service announcements: Drupal 8 is now end-of-life - PSA-2021-11-30

Planet Drupal - Tue, 2021-11-30 14:31
Date: 2021-November-30Description: 

As of November 17, 2021, the Drupal core version 8 series has reached end-of-life. This means that all releases of Drupal 8 core (with 8.y.x version numbers) and Drupal contributed project releases that are compatible with only Drupal 8 will be marked unsupported as they no longer have security team support.

Drupal 8.0.0 was first released on November 9, 2015. The last version was released on November 17, 2021.

All Drupal 8 site owners must upgrade to Drupal 9 to receive security updates and bug fixes. The Drupal Association will also disable testing with unsupported versions of Drupal.

Security issues that only affect Drupal 8 (and not Drupal 9 or Drupal 7) will be made public and sites are at risk of having these issues exploited if they do not upgrade.

What about Drupal 7 and Drupal 9?

Contributed projects like themes and modules will still receive security advisories if they are compatible with either Drupal 7 or 9 and have opted in to security coverage.

Drupal 7's end-of-life is currently scheduled for November of 2022, and it will receive security updates until then. Drupal 9's end-of-life is scheduled for November of 2023. For more information on release schedules, see the core release cycle overview.

Categories: FLOSS Project Planets

PyCoder’s Weekly: Issue #501 (Nov. 30, 2021)

Planet Python - Tue, 2021-11-30 14:30

#501 – NOVEMBER 30, 2021
View in Browser »

Late-Bound Argument Defaults for Python

Python supports default values for arguments to functions, but those defaults are evaluated at function-definition time. A proposal to add defaults that are evaluated when the function is called has been discussed at some length on the python-ideas mailing list.

Data Visualization Interfaces in Python With Dash

In this course, you’ll learn how to build a dashboard using Python and Dash. Dash is a framework for building data visualization interfaces. It helps data scientists build fully interactive web applications quickly.

Application Performance Monitoring Built for Developers, by Developers

Scout is a lightweight, production-grade application monitoring service built for modern development teams. Proactive alerting, real-time insight and always-on support means you can rest easy knowing Scout will find performance issues before your customers do. Embed our agent, and we’ll do the rest →
SCOUT APM sponsor

Rats in Doom: A Novel VR Setup for Rodents

A novel VR setup developed for rodents that lets rats play Doom. Uses Python for the controller interface.

CPython Dev in Residence: Weekly Report, November 22–28

A short review of try/except, exception handling in asyncio today, and the future of asyncio error handling.

Discussions Python 3.10 Pattern Matching: Wildcard in String?


Does NumPy Array Really Take Less Memory Than Python List?


Python Jobs Senior Python Developer (Anywhere)

Six Feet Up

Senior Backend Engineer (Anywhere)


Senior Backend Software Engineer (Anywhere)


Senior Python Engineer (Anywhere)


Senior Software Engineer (United States)


Full Stack Software Engineer (Washington D.C., USA)


Senior Software Engineer (Washington D.C., USA)


Python Backend Engineer (Hasselt, Belgium)


More Python Jobs >>>

Articles & Tutorials Programmers Should Stop Celebrating Incompetence

The creator of Ruby on Rails on programmer competence: “The world has never had a greater need for programmers than it does today. It’s never had a greater need for competent programmers than it does today. Let’s skip the overly self-deprecating nonsense that nobody knows what they’re doing, and trying to learn things in depth is not for us.”

How to Ditch Codecov for Python Projects

“Codecov’s unreliability breaking CI on my open source projects has been a constant source of frustration for me for years. I have found a way to enforce coverage over a whole GitHub Actions build matrix that doesn’t rely on third-party services.”

Get Your Next Python Job Through Hired

Hired is home to thousands of companies ranging from startups to Fortune 500s that are actively hiring developers, data scientists, mobile engineers, and more. Its really easy: once you create a profile, hiring managers can send interview requests based on your skills. Sign up today →
HIRED sponsor

Ready to Publish Your Python Packages?

Are you interested in sharing your Python project with the broader world? Would you like to make it easily installable using pip? How do you create Python packages that share your code in a scalable and maintainable way?

How to Write a Great Stack Overflow Question

“The single best thing you can do when asking for coding help online is provide a short, complete example script that others can copy, paste, and run without any modification to reproduce your problem.”
KEVIN MARKHAM • Shared by Kevin Markham

Experimental APIs in Python 3.10 and the Future of Trust Stores

“In Python 3.10.0 there were a few new APIs added to the ssl module related to certificate chains that weren’t listed in the Python 3.10 release notes due to being experimental.”

The Problem With Python’s map and filter (And How We Might Fix It)

The author explores the pros and cons of Python’s default map() and filter() implementation and potential improvements that could be made.
ABHINAV OMPRAKASH • Shared by Abhinav

Shortcut Provides Speedy Task Management, Reporting, and Collaboration for Software Teams

Shortcut is project management built for developers. Whether you’re a startup that iterates quickly by providing every engineer with a free pallet of Red Bull, or a large org that has strict ship dates to hit, give us a try for free.
SHORTCUT sponsor

The Rinds of the Cheese Shop Menu

A silly analysis of the weird and wonderful contents of PyPI: What are the longest package names? Numeric vs non-numeric version names?
BEN NUTTALL • Shared by Ben Nuttall

Financial Market Data Analysis With Pandas


Introduction to Python’s operator Module

FLORIAN DAHLITZ • Shared by Florian Dahlitz

Projects & Code pyc64: Commodore-64 Simulator in Pure Python


tiptop: Command-Line System Monitoring


myhdl: Python as a Hardware Description Language


RexMex: Recommender System Evaluation Library

GITHUB.COM/ASTRAZENECA • Shared by Benedek Rozemberczki

kolo: See Everything Happening in Your Running Django App

GITHUB.COM/KOLOFORDJANGO • Shared by Philipp Acsany

SQLLineage: SQL Lineage Analysis Tool

GITHUB.COM/REATA • Shared by Ryan Hu

FastAPI-Azure-Auth: Azure AD Auth for FastAPI Apps

INTILITY.GITHUB.IO • Shared by Jonas Krüger Svensson

Events Weekly Real Python Office Hours Q&A (Virtual)

December 1, 2021

FlaskCon 2021 Online

December 1 to December 5, 2021

PyCon Tanzania

December 1 to December 6, 2021

PyCode Conference

December 2 to December 4, 2021

Pyjamas Conf 2021

December 4 to December 6, 2021

PyCon Indonesia 2021

December 4 to December 6, 2021

Happy Pythoning!
This was PyCoder’s Weekly Issue #501.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Categories: FLOSS Project Planets

Ching-Hwa Yu: How to Create a Python Plugin System with Stevedore

Planet Python - Tue, 2021-11-30 13:00
Photo by Kieran Wood on Unsplash

One of the questions that I see and hear often is how to extend applications using a Python plugin system. For a test engineer, this is usually related to hardware abstraction. For others, they may want to separate the core functionality from extensions. Through this method, deployment can be simplified where only the required pieces are installed through their individual package(s). Whatever the reason may be, several libraries are available to tackle this challenge. In this article, a simple abstraction layer will be built using the Stevedore library and we will cover the the main concepts of building plugins.

The code in this article is available at https://github.com/chinghwayu/plugin_tutorial.

NOTE: While the code illustrates relays and a hardware abstraction layer, the same concept can be used for services such as a database with a software abstraction layer.

Plugin Architecture

Regardless of the plugin framework used, the architecture behind the implementation of plugins is mostly the same.

  • Define abstract base class
  • Define plugin class inheriting from base class
Abstract Base Class

For plugins of similar type in a Python plugin system, there usually is some common functionality. Plugins based on a protocol or specification typically have common basic functionality. For example, instruments that adhere to IEEE 488.2 standard must respond to queries for its identification.

Although the abstract part is sometimes omitted, this is an important step as it establishes the basic structure of each plugin by defining which methods must be implemented. In Python, this is accomplished with the abc module.

The example shown defines a basic relay instrument driver with three methods – connect, disconnect, and reconnect. In the __init__ method, the connected attribute is established for status as this is common between all plugins.

For the other abstract methods, only docstrings are needed. There is no need to include pass or other code.

from abc import ABCMeta, abstractmethod class RelayBase(metaclass=ABCMeta): """Base class for relay plugins""" def __init__(self) -> None: """Define base attributes.""" self.connected = False @abstractmethod def disconnect(self) -> None: """Disconnects relay.""" @abstractmethod def connect(self) -> None: """Connects relay.""" @abstractmethod def reconnect(self, seconds: int) -> None: """Disconnects for specified time and reconnects. Args: seconds (int): Amount of time to sleep between disconnect and connect. """ Plugin Class

When defining the actual implementation, each of the plugins will inherit from the plugin base class. In the example, two plugins are defined that inherit from RelayBase.

In the __init__ method, a super() call is made to create the connected attribute which the other methods will update.

class RelayOne(RelayBase): def __init__(self): super().__init__() def disconnect(self): self.connected = False print("Disconnected One") def connect(self): self.connected = True print("Connected One") def reconnect(self, seconds: int = 5): self.seconds = seconds self.disconnect() print(f"One paused for {seconds} seconds...") self.connect() class RelayTwo(RelayBase): def __init__(self): super().__init__() def disconnect(self): self.connected = False print("Disconnected Two") def connect(self): self.connected = True print("Connected Two") def reconnect(self, seconds: int = 5): self.seconds = seconds self.disconnect() print(f"Two paused for {seconds} seconds...") self.connect() Loading Plugins

At this point, you may think that this is all that’s needed to establish a Python plugin system. For simple plugins, this may be true. However, consider the following:

  • How do we decide which plugin to call at runtime?
  • How can we easily switch between the implementations?
  • How can we package these plugins for distribution?
  • How can we test the interfaces to ensure the correct implementation is being called?

For simple applications or scripts, we may simply call the specific implementation directly and hardcode this in. But consider the case where you may be sharing code with a teammate in another location and they don’t have the same relay. We don’t want to maintain another version of the same code with the alternate implementation hardcoded. What we need is way to easily switch between implementations without altering any code. The only change that should be made is in a configuration file for which implementation to use at runtime.

Plugin Entry Point

There are several ways to discover and load plugins. With Stevedore, this uses entry points to establish keys that can be queried as pointers to the location of specific code. This is the same mechanism found often in packages that enable the launching of Python code as command line scripts once installed into an environment. While this uses a built in console_scripts type of entry point, custom entry point types can be created for the purpose of managing plugins.

For a full discussion on extending applications with plugins, please watch the PyCon 2013 talk on this subject from the author of Stevedore, Doug Hellman. The presentation compares the approach taken with Stevedore compared to other existing approaches at the time.

When building packages for distribution, we can add entry points so that when it’s installed into a Python environment, the environment immediately knows where the plugins are located from the established keys. Adding entry points is simple. In the setup.py for a package, assign a dictionary to the entry_points parameter of setup.

from setuptools import setup setup( entry_points={ "plugin_tutorial": [ "relay1 = relay:RelayOne", "relay2 = relay:RelayTwo", ], }, )

The plugin_tutorial key is used as the plugin namespace. The names for each plugin is defined as relay1 and relay2. The location of the the plugin is defined as the module name and class within the module separated by colon, relay:RelayOne and relay:RelayTwo.

For cases when we are using plugins but don’t need to install as package, we can register them into the entry point cache of Stevedore. This is useful for development purposes and when implementing unit tests.

The example below checks a namespace for the specified entry point. If the entry point doesn’t exist, it will be added.

View this gist on GitHubRegisters a stevedore plugin dynamically without needing to install as a package Managing Plugins

With Stevedore, there are several ways to manage plugins in a Python plugin system.

  • Drivers – Single Name, Single Entry Point
  • Hooks – Single Name, Many Entry Points
  • Extensions – Many Names, Many Entry Points

For this article, the driver approach will be discussed since that is the most common use case. Take a look at the Stevedore documentation that discuss Patterns for Loading for the other methods.

For a driver, we need to call the DriverManager. In the parameter list, only namespace and name are required which are directly related to the entry points. Optional parameters are available and the one used in this example is invoke_on_load. While the relay example only establishes a class attribute, for an actual instrument driver, we usually need to perform some kind of initialization. This can be executed at the time when the plugin is loaded.

Calling DriverManager will return a manager object. The actual driver object can be accessed through the driver property. From this property, we can also create abstracted methods to call the driver methods.

from stevedore import driver class Relay: def __init__(self, name="", **kwargs) -> None: self._relay_mgr = driver.DriverManager( namespace="plugin_tutorial", name=name, invoke_on_load=True, invoke_kwds=kwargs, ) @property def driver(self): return self._relay_mgr.driver def disconnect(self) -> None: self.driver.disconnect() def connect(self) -> None: self.driver.connect() def reconnect(self, seconds: int = 5) -> None: self.driver.reconnect(seconds)

The **kwargs parameter is not used but included to show implementation of how to pass parameters to drivers which may have different initialization parameters.

The @property decorator for the driver method is syntactic sugar to provide a shortcut to the driver object. If this wasn’t provided, we would need to call the driver’s disconnect method as:

r = Relay(name="relay1") r._relay_mgr.driver.disconnect() Putting It Together

For a plugin installed into a Python environment, the entry points have established when the package was installed. Through the abstraction interface, we can decide which plugin to load at runtime. Shown is a unit test that calls both plugins and each of its methods.

In order to run this, we will need to first install as a package.

$ pip install -e /path/to/plugin_tutorial ... Installing collected packages: plugin-tutorial Running setup.py develop for plugin-tutorial Successfully installed plugin-tutorial $ pip list | grep plugin_tutorial plugin-tutorial 1.0.0 /path/to/plugin_tutorial from relay import Relay def test_installed_plugin(): r1 = Relay(name="relay1") assert isinstance(r1, Relay) assert r1.driver.connected == False r1.disconnect() assert r1.driver.connected == False r1.connect() assert r1.driver.connected == True r1.reconnect(7) assert r1.driver.seconds == 7 r2 = Relay(name="relay2") assert isinstance(r2, Relay) assert r2.driver.connected == False r2.disconnect() assert r2.driver.connected == False r2.connect() assert r2.driver.connected == True r2.reconnect(9) assert r2.driver.seconds == 9

To call plugins not installed through the package installation process, we’ll need to first register and then call the plugins. This is useful for writing unit tests that include a dummy plugin. Shown is the same unit test with the plugin registered.

from relay import Relay from register_plugin import register_plugin def test_register_plugin(): namespace = "plugin_tutorial" register_plugin( name="relay1", namespace=namespace, entry_point="relay:RelayOne", ) register_plugin( name="relay2", namespace=namespace, entry_point="relay:RelayTwo", ) r1 = Relay(name="relay1") assert isinstance(r1, Relay) assert r1.driver.connected == False r1.disconnect() assert r1.driver.connected == False r1.connect() assert r1.driver.connected == True r1.reconnect(7) assert r1.driver.seconds == 7 r2 = Relay(name="relay2") assert isinstance(r2, Relay) assert r2.driver.connected == False r2.disconnect() assert r2.driver.connected == False r2.connect() assert r2.driver.connected == True r2.reconnect(9) assert r2.driver.seconds == 9 Resources

For additional information:

The post How to Create a Python Plugin System with Stevedore appeared first on Ching-Hwa Yu.

Categories: FLOSS Project Planets

Python for Beginners: Shortest Path Length from a Vertex to other Vertices in a Graph

Planet Python - Tue, 2021-11-30 12:56

Graphs are used to represent geographical maps, computer networks, etc. In this article, we will discuss how to calculate the shortest distance between vertices in an unweighted graph. To calculate the shortest path length from a vertex to other vertices, we will use breadth first search algorithm.

How to calculate the Shortest Path Length from a Vertex to other vertices?

In an unweighted graph, all the edges have equal weight. It means that we just have to count the number of edges between each vertex to calculate the shortest path length between them. 

For example, consider the following graph.

Graph in Python

Let us calculate the shortest distance between each vertex in the above graph.

There is only one edge E2 between vertex A and vertex B. So, the shortest path length between them is 1. 

We can reach C from A in two ways. The first one is using the edges E4-> E5->E6 and the second path is using the edges E2-> E6. Here, we will choose the shortest path, i.e. E2-> E6. Hence the shortest path length between vertex A and vertex C is 2.

There is only one edge E1 between vertex A and vertex D. So, the shortest path length between them is 1. 

There is only one edge E3 between vertex A and vertex E. So, the shortest path length between them is 1. 

We can reach F from A in two ways. The first one is using the edges E2-> E5 and the second path is using the edges E4. Here, we will choose the shortest path, i.e. E4. Hence the shortest path length between vertex A and vertex F is 1.

Algorithm to calculate the Shortest Path Length from a Vertex to other vertices

By now, you must have understood that we have to count the number of edges between the vertices to calculate the distance between the vertices. For this, we will modify the  breadth first search algorithm as follows.

  • We will declare a python dictionary that will contain the vertices as their keys and distance from the source vertex as the associated values. 
  • Initially, we will assign the distance of each vertex from the source as infinite , denoted by a large number. Whenever we will find a vertex during traversal, we will calculate the current distance of the vertex from the source. If the current distance appears to be less than the distance mentioned in the dictionary containing the distance between source and other vertices, we will update the distance in the dictionary. 
  • After full breadth first traversal, we will have the dictionary containing the least distance from the source to each vertex. 

We can formulate the algorithm for calculating the shortest path length between vertices of an unweighted graph as follows.

  1. Create an empty Queue Q. 
  2. Create a list visited_vertices to keep track of visited vertices.
  3. Create a dictionary distance_dict to keep track of distance of vertices from the source vertex. Initialize the distances to 99999999. 
  4. Insert source vertex into Q and visited_vertices.
  5. If Q is empty, return. Else goto 6.
  6. Take out a vertex v from Q.
  7. Update the distances of unvisited neighbors of v in distance_dict. 
  8. Insert the unvisited neighbors to  Q and visited_vertices.
  9. Go to 5.

As we have discussed the example and formulated an algorithm to find the shortest path length between source vertex and other vertices in a graph, let us implement the algorithm in python.

from queue import Queue graph = {'A': ['B', 'D', 'E', 'F'], 'D': ['A'], 'B': ['A', 'F', 'C'], 'F': ['B', 'A'], 'C': ['B'], 'E': ['A']} print("Given Graph is:") print(graph) def calculate_distance(input_graph, source): Q = Queue() distance_dict = {k: 999999999 for k in input_graph.keys()} visited_vertices = list() Q.put(source) visited_vertices.append(source) while not Q.empty(): vertex = Q.get() if vertex == source: distance_dict[vertex] = 0 for u in input_graph[vertex]: if u not in visited_vertices: # update the distance if distance_dict[u] > distance_dict[vertex] + 1: distance_dict[u] = distance_dict[vertex] + 1 Q.put(u) visited_vertices.append(u) return distance_dict distances = calculate_distance(graph, "A") for vertex in distances: print("Shortest Path Length to {} from {} is {}.".format(vertex, "A", distances[vertex]))


Given Graph is: {'A': ['B', 'D', 'E', 'F'], 'D': ['A'], 'B': ['A', 'F', 'C'], 'F': ['B', 'A'], 'C': ['B'], 'E': ['A']} Shortest Path Length to A from A is 0. Shortest Path Length to D from A is 1. Shortest Path Length to B from A is 1. Shortest Path Length to C from A is 2. Shortest Path Length to E from A is 1. Conclusion

In this article, we have discussed and implemented the algorithm to calculate the shortest path length between vertices in an unweighted graph. Here we have used the breadth first graph traversal algorithm. To read about binary tree traversal algorithms, you can read  Inorder tree traversal algorithm or level order tree traversal algorithm.

The post Shortest Path Length from a Vertex to other Vertices in a Graph appeared first on PythonForBeginners.com.

Categories: FLOSS Project Planets

PyCharm: Last day to buy PyCharm and Support Python!

Planet Python - Tue, 2021-11-30 09:01

You still have 1 day to purchase your PyCharm Pro license at 30% off and have the full amount of your purchase donated directly to the PSF!

Buy PyCharm and Support Python

PyCharm joins the Python Software Foundation on their end-of-the-year fundraiser one more time! Since November 9 you can purchase a new PyCharm Pro individual license at 30% off, and have the full amount of your purchase donated directly to the PSF.

The campaign ends tomorrow, December 1. If you still want to support Python, the time is now!

About the Partnership

The Python Software Foundation is the principal organization behind the Python programming language. As a non-profit organization, the PSF depends on sponsorships and donations to support its work. The contributions are used to maintain ongoing sprints, meetups, community events, Python documentation, fiscal sponsorships, software development, and community projects. None of this is possible without your support.

The end-of-the-year fundraiser is an important source of resources for the PSF and helps sustain programs that support the larger Python community. The Python Software Foundation and JetBrains have partnered before, conducting four consecutive Python Developers Surveys, and this year JetBrains PyCharm will once again support the Python Software Foundation in its fundraiser campaign.

Categories: FLOSS Project Planets

Real Python: Data Visualization Interfaces in Python With Dash

Planet Python - Tue, 2021-11-30 09:00

In the past, creating analytical web applications was a task for seasoned developers that required knowledge of multiple programming languages and frameworks. That’s no longer the case. Nowadays, you can make data visualization interfaces using pure Python. One popular tool for this is Dash.

Dash gives data scientists the ability to showcase their results in interactive web applications. You don’t need to be an expert in web development. In an afternoon, you can build and deploy a Dash app to share with others.

In this course, you’ll learn how to:

  • Create a Dash application
  • Use Dash core components and HTML components
  • Customize the style of your Dash application
  • Use callbacks to build interactive applications
  • Deploy your application on Heroku

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Russell Coker: Links November 2021

Planet Debian - Tue, 2021-11-30 08:31

The Guardian has an amusing article by Sophie Elmhirst about Libertarians buying a cruise ship to make a “seasteading” project off the coast of Panama [1]. It turns out that you need permits etc to do this and maintaining a ship is expensive. Also you wouldn’t want to mine cryptocurrency in a ship cabin as most cabins are small and don’t have enough airconditioning to remain pleasant if you dump 1kW or more into the air.

NPR has an interesting article about the reaction of the NRA to the Columbine shootings [2]. Seems that some NRA person isn’t a total asshole and is sharing their private information, maybe they are dying and are worried about going to hell.

David Brin wrote an insightful blog post about the singleton hypothesis where he covers some of the evidence of autocratic societies failing [3]. I think he makes a convincing point about a single centralised government for human society not being viable. But something like the EU on a world wide scale could work well.

Ken Shirriff wrote an interesting blog post about reverse engineering the Yamaha DX7 synthesiser [4].

The New York Times has an interesting article about a Baboon troop that became less aggressive after the alpha males all died at once from tuberculosis [5]. They established a new more peaceful culture that has outlived the beta males who avoided tuberculosis.

The Guardian has an interesting article about how sequencing the genomes of the entire population can save healthcare costs while improving the health of the population [6]. This is somthing wealthy countries should offer for free to the world population. At a bit under $1000 per test that’s only about $7 trillion to test everyone, and of course the price should drop significantly if there were billions of tests being done.

The Strategy Bridge has an interesting article about SciFi books that have useful portrayals of military strategy [7]. The co-author is Major General Mick Ryan of the Australian Army which is noteworthy as Major General is the second highest rank in use by the Australian Army at this time.

Vice has an interesting article about the co-evolution of penises and vaginas and how a lot of that evolution is based on avoiding impregnation from rape [8].

Cory Doctorow wrote an insightful Medium article about the way that governments could force interoperability through purchasing power [9].

Cory Doctorow wrote an insightful article for Locus Magazine about imagining life after capitalism and how capitalism might be replaced [10]. We need a Star Trek future!

Arstechnica has an informative article about new developmenet in the rowhammer category of security attacks on DRAM [11]. It seems that DDR4 with ECC is the best current mitigation technique and that DDR3 with ECC is harder to attack than non-ECC RAM. So the thing to do is use ECC on all workstations and avoid doing security critical things on laptops because they can’t juse ECC RAM.

Related posts:

  1. Links April 2021 Dr Justin Lehmiller’s blog post comparing his official (academic style)...
  2. Links October 2021 Bloomburg has an insightful article about Juniper, the NSA, and...
  3. Links September 2021 Matthew Garrett wrote an interesting and insightful blog post about...
Categories: FLOSS Project Planets