FLOSS Project Planets

ItsMyCode: ImportError: No module named requests

Planet Python - Wed, 2021-11-24 06:56

ItsMyCode |

In Python, if you try to import Requests without installing the module using pip, you will get ImportError: no module named requests error. 

In this tutorial, let’s look at installing the Requests module correctly in different operating systems and solve no module named requests error.  

ImportError: No module named requests

Requests are not a built-in module (it doesn’t come with the default python installation) in Python, you need to install it explicitly using the pip installer and then use it.

If you are getting an error installing pip checkout pip: command not found to resolve the issue.

Install Requests in OSX/Linux 

The recommended way to install the requests module is using pip or pip3 for Python3 if you have pip installed already.

Using Python 2

$ sudo pip install requests 

Using Python 3

$ sudo pip3 install requests 

Alternatively, if you have easy_install in your system, you can install requests using the below command.

Using easy install

$ sudo easy_install -U requests

For CentOs

$ yum install python-requests

For Ubuntu

To install requests module on Debian/Ubuntu for Python2:

$ sudo apt-get install python-requests

And for Python3, the command is:

$ sudo apt-get install python3-requests Install Requests in Windows

In the case of windows, you can use pip or pip3 based on the Python version you have to install the requests module.

$ pip3 install requests

If you have not added the pip to the environment variable path, you can run the below command in Python 3, which will install the requests module. 

$ py -m pip install requests

The post ImportError: No module named requests appeared first on ItsMyCode.

Categories: FLOSS Project Planets

ItsMyCode: ImportError: No module named Pandas

Planet Python - Wed, 2021-11-24 06:53

ItsMyCode |

In Python, if you try to import pandas without installing the module using pip, you will get ImportError: no module named pandas error. 

In this tutorial, let’s look at installing the pandas module correctly in different operating systems and solve no module named pandas error.  

ImportError: No module named pandas

Pandas are not a built-in module (it doesn’t come with the default python installation) in Python, you need to install it explicitly using the pip installer and then use it.

If you are getting an error installing pip checkout pip: command not found to resolve the issue.

Pandas are distributed through pip as a wheel, which means you need to install wheel first and then pandas:

Install pandas in OSX/Linux 

The recommended way to install the pandas module is using pip or pip3 for Python3 if you have installed pip already.

Using Python 2

$ sudo pip install wheel $ sudo pip install pandas

Using Python 3

$ sudo pip3 install wheel $ sudo pip3 install pandas

Alternatively, if you have easy_install in your system, you can install pandas using the below command.

Using easy install

$ sudo easy_install -U wheel $ sudo easy_install -U pandas

For CentOs

$ yum install python-wheel $ yum install python-pandas

For Ubuntu

To install pandas module on Debian/Ubuntu :

$ sudo apt-get install python-wheel $ sudo apt-get install python-pandas Install Pandas in Windows

In the case of windows, you can use pip or pip3 based on the Python version, you have to install the pandas module.

$ pip3 install wheel $ pip3 install pandas

If you have not added the pip to the environment variable path, you can run the below command in Python 3, which will install the pandas module. 

$ py -m pip install wheel $ py -m pip install pandas

The post ImportError: No module named Pandas appeared first on ItsMyCode.

Categories: FLOSS Project Planets

Stack Abuse: Graphs in Python: Breadth-First Search (BFS) Algorithm

Planet Python - Wed, 2021-11-24 06:30

Graphs are one of the most useful data structures. They can be used to model practically everything - object relations and networks being the most common ones. An image can be represented as a grid-like graph of pixels, and sentences can be represented as graphs of words. Graphs are used in various fields, from cartography to social psychology even, and of course they are widely used in Computer Science.

Due to their widespread use, graph search and traversal play an important computational role. The two fundamental algorithms used for graph search and traversal are Depth-First Search (DFS) and Breadth-First Search (BFS).

If you'd like to read more about Depth-First Search, read our Graphs in Python: Depth-First Search (DFS) Algorithm!

In this article we will go over the theory behind the algorithm and the Python implementation of Breadth-First Search and Traversal. First, we'll be focusing on node search, before delving into graph traversal using the BFS algorithm, as the two main tasks you can employ it for.

Note: We're assuming an adjacency-list implemented graph in the guide.

Breadth-First Search - Theory

Breadth-First Search (BFS) traverses the graph systematically, level by level, forming a BFS tree along the way.

If we start our search from node v (the root node of our graph or tree data structure), the BFS algorithm will first visit all the neighbours of node v (it's child nodes, on level one), in the order that is given in the adjacency list. Next, it takes the child nodes of those neighbours (level two) into consideration , and so on.

This algorithm can be used for both graph traversal and search. When searching for a node that satisfies a certain condition (target node), the path with the shortest distance from the starting node to the target node. The distance is defined as the number of branches traversed.

Breadth-First Search can be used to solve many problems such as finding the shortest path between two nodes, determining the levels of each node, and even solving puzzle games and mazes.

While it's not the most efficient algorithm for solving large mazes and puzzles - and it's outshined by algorithms such as Dijkstra's Algorithm and A* - it still plays an important role in the bunch and depending on the problem at hand - DFS and BFS can outperform their heuristic cousins.

If you'd like to read more about Dijkstra's Algorithm or A* - read our Graphs in Python: Dijkstra's Algorithm and Graphs in Python: A* Search Algorithm!

Breadth-First Search - Algorithm

When implementing BFS, we usually use a FIFO structure like a Queue to store nodes that will be visited next.

Note: To use a Queue in Python, we need to import the corresponding Queue class from the queue module.

We need to pay attention to not fall into infinity loops by revisiting the same nodes over and over, which can easily happen with graphs that have cycles. Having that in mind, we'll be keeping track of the nodes that have been visited. That information doesn't have to be explicitly saved, we can simply keep track of the parent nodes, so we don't accidentally go back to one after it's been visited.

To sum up the logic, the BFS Algorithm steps look like this:

  1. Add the root/start node to the Queue.
  2. For every node, set that they don't have a defined parent node.
  3. Until the Queue is empty:
    • Extract the node from the beginning of the Queue.
    • Perform output processing.
    • For every neighbour of the current node that doesn't have a defined parent (is not visited), add it to the Queue, and set the current node as their parent.

Output processing is performed depending on the purpose behind the graph search. When searching for a target node, output processing is usually testing if the current node is equal to the target node. This is the step on which you can get creative!

Breadth-First Search Implementation - Target Node Search

Let's first start out with search - and search for a target node. Besides the target node, we'll need a start node as well. The expected output is a path that leads us from the start node to the target node.

With those in mind, and taking the steps of the algorithm into account, we can implement it:

from queue import Queue def BFS(adj_list, start_node, target_node): # Set of visited nodes to prevent loops visited = set() queue = Queue() # Add the start_node to the queue and visited list queue.put(start_node) visited.add(start_node) # start_node has not parents parent = dict() parent[start_node] = None # Perform step 3 path_found = False while not queue.empty(): current_node = queue.get() if current_node == target_node: path_found = True break for next_node in adj_list[current_node]: if next_node not in visited: queue.put(next_node) parent[next_node] = current_node visited.add(next_node) # Path reconstruction path = [] if path_found: path.append(target_node) while parent[target_node] is not None: path.append(parent[target_node]) target_node = parent[target_node] path.reverse() return path

When we're reconstructing the path (if it is found), we're going backwards from the target node, through it's parents, retracing all the way to the start node. Additionally, we might want to reverse the path for our own intuition of going from the start_node towards the target_node.

On the other hand, if there is no path, the algorithm will return an empty list.

Let's construct a simple graph, as an adjacency list:

graph = { 1 : [2, 3, 4, 5], 2 : [1, 3], 3 : [1, 2, 4, 6], 4 : [1, 3, 5, 6], 5 : [1, 4, 6], 6 : [3, 4, 5] }

Now, say we'd like to search for node 6 starting at node 1:

path = BFS(graph, 1, 6) print(path)

Running this code results in:

[1, 3, 6]

Now, let's take a look at a visual representation of the graph itself:

The shortest path between 1 and 6 is indeed [1, 3, 6]. Though, you could also traverse [1, 4, 6] which appears to be shorter (due to a diagonal path), and [1, 5, 6]. These alternative paths are, fundementally, the same distance as [1, 3, 6] - however, consider how BFS compares nodes. It "scans" from left to right and 3 is the first node on the left-hand side of the adjacency list that leads to 6, so this path is taken instead of the others.

Note: There are cases in which a path between two nodes cannot be found. This scenario is typical for disconnected graphs, where there are at least two nodes that are not connected by a path.

Here's how a disconected graph looks like:

If we were to try and perform a search for a path between nodes 0 and 3 in this graph, that search would be unsuccessful, and an empty path would be returned.

Breadth-First Implementation - Graph Traversal

Breadth-First Traversal, is a special case of Breadth-First Search that traverses the whole graph, instead of searching for a target node. The algorithm stays the same as we've defined it before, the difference being that we don't check for a target node and we don't need to find a path that leads to it.

This simplifies the implementation significantly - let's just print out each node being traversed to gain an intuition of how it passes through the nodes:

def traversal(adj_list, start_node): visited = set() queue = Queue() queue.put(start_node) visited.add(start_node) while not queue.empty(): current_node = queue.get() print(current_node, end = " ") for next_node in adj_list[current_node]: if next_node not in visited: queue.put(next_node) visited.add(next_node)

Now, let's define a simple graph:

graph = { 1 : [2, 3], 2 : [1, 3, 5], 3 : [1, 2, 4], 4 : [3], 5 : [2] } traversal(graph, 1)

Finally, let's run the code:

1 2 3 5 4 Step by step

Let's dive into this example a bit deeper and see how the algorithm works step by step. Here's a picture of the graph so we can easily follow the steps:

As we start the traversal from the start node 1, it is put into the visited set and into the queue as well. While we still have nodes in the queue, we extract the first one, print it, and check all of it's neighbours.

When going through the neigbours, we check if each of them is visited, and if not we add them to the queue and mark them as visited:

Steps Queue Visited Add start node `1` [`1`] {`1`} Visit `1`, add `2` & `3` to Queue [`2`, `3`] {`1`} Visit `2`, add `5 `to Queue [`3`, `5`] {`1`, `2`} Visit `3`, add `4` to Queue [`5`, `4`] {`1`, `2,` `3`} Visit `5`, no unvisited neighbours [`4`] {`1`, `2,` `3`, `5`} Visit `4`, no unvisited neighbours [ ] {`1`, `2,` `3`, `5`, `4`} Time complexity

During Breadth-First Traversal, every node is visited exactly once, and every branch is also viewed once in case of a directed graph, that is, twice if the graph is undirected. Therefore, the time complexity of the BFS algorithm is O(|V| + |E|), where V is a set of the graph's nodes, and E is a set consisting of all of it's branches (edges).


In this guide, we've explained the theory behind the Breadth-First Search algorithm and defined it's steps.

We've depicted the Python implementation of both Breadth-First Search and Breadth-First Traversal, and tested them on example graphs to see how they work step by step. Finally, we've explained the time complexity of this algorithm.

Categories: FLOSS Project Planets

Droptica: A Quick Way for Editing and Customizing a Drupal Paragraph

Planet Drupal - Wed, 2021-11-24 05:16

For laying out content on a website, Drupal standardly uses a single body field that takes advantage of the WYSIWYG capabilities and is available immediately after installation of the CKEditor. This solution is easy to use but not flexible enough to be suitable in every case. For more advanced layouts, we can use the functionality prepared by the community in the Paragraphs module.

Drupal paragraph - characteristics

Paragraphs in Drupal are a useful solution with many possibilities. Thanks to them, we aren’t dependent on one WYSIWYG field where we place all the images and videos one under another. Paragraphs can be considered as some kind of sections on a given page. Each of these sections can have different configuration options, affecting the paragraph's appearance and its performance. This may be, for example, a color scheme, the size of images or a completely different layout of fields. With just a few paragraphs added to a page, editing them becomes time-consuming because Drupal standard UI doesn't allow for responsive editing. Is there any way to improve the UX? The answer to this question is Droopler, a Drupal distribution for building websites that uses the Geysir and Field Group modules.

Editing paragraphs in Drupal

As I pointed out above, editing paragraphs in pure Drupal is a time-consuming task. Let us analyze an example:


On the given page, we see several paragraphs that have many configuration options. Here is the configuration form for one of them:


As we can see in this example, the configuration options aren’t grouped, although the structure of the paragraph allows for separating at least two sections. The Field Group module - described further on in this article - is responsible for this functionality.

We finished the presentation of the configuration options. Below you can see how the paragraph looks like on the frontend:


The standard editing path in Drupal firstly requires going to edit entities and then editing the selected paragraph. If a paragraph has references to other paragraphs in it, and our goal is to edit one of the referenced paragraphs, then the whole process starts to get complicated. Usually, the paragraphs aren’t very well described, and it’s easy to enter the edition of the wrong paragraph. After editing, you need to save the changes and then switch to the entity view to see if you are satisfied with the modifications. Note that each time the page must be reloaded, as we don’t see the changes "live". This makes the editing process time-consuming and not very pleasant. How does it look like in Droopler?

Editing a Drupal paragraph in Droopler

Droopler is an installation profile that allows you to create websites in Drupal in a very simple and fast way. It’s great for both building business web pages and microservices, and from version 2.2 also for online stores. Starting from this version, Droopler has an integration with the Commerce module.

The frontend of this installation profile is based on Bootstrap 4, so it isn’t complicated to make any modifications to the website’s appearance. Droopler also uses Geysir and Field Group modules, which will help us improve and speed up paragraph editing.

In Droopler, there is no reason why we would want to go into editing an entire entity if we only want to make a change to a paragraph. We use the Geysir module to make changes to a paragraph, which we can launch by clicking on the Paragraph overlay button at the top right of the screen.


After entering the paragraph edit mode, point the cursor to the paragraph you are interested in and click on the edit button.


When clicked, you’ll see an overlay menu containing the same settings as in the standard view. However, this time they are grouped using the Field Group module.


Once you've made your changes, all you have to do is save them, and the page asynchronously reloads only the paragraph that was edited. This means that we just got rid of the need to reload the whole website, and we can see the changes almost immediately!

It’s worth noting that the paragraph presented in Droopler uses the Field Group module, which allows you to divide the fields available in an entity into tabs. They are configured in the edit options of the entity view form.


Among other options, we can choose here between horizontal and vertical tabs, close or open our selected tab when entering the edition, or add custom classes to sections that allow us to manipulate only selected elements using CSS or JavaScript.

Editing a Drupal paragraph - summary

Paragraphs offer many possibilities, but configuring, editing and maintaining them on a standard Drupal instance isn’t the most convenient. The approach presented in Droopler will certainly make the process of creating, editing and managing paragraphs easier and faster. We recommend taking a closer look at how Droopler manages them. If your website also uses these components, we recommend using a similar method or the Droopler installation profile.

Categories: FLOSS Project Planets

Vardot: Top 10 Free Drupal Themes

Planet Drupal - Wed, 2021-11-24 04:57
Image Top 10 Free Drupal Themes Image Rashed Azzam Position Wednesday, November 24, 2021 - 11:57 Teaser image Top 10 Free Drupal Themes Join the conversation + Comments Solutions by need Enterprise CMS Knowledge Management Drupal Managed Services On-Site SEO Related services Web Development UI/UX Design Digital Marketing Digital Strategy Product Varbase Vardoc Uber Publisher Open Social Marketing Automation Vote up! 0 claps
Categories: FLOSS Project Planets

Specbee: Why Page Speed Matters and How to Get Your Website to Load Faster!

Planet Drupal - Wed, 2021-11-24 04:23
Why Page Speed Matters and How to Get Your Website to Load Faster! Shri Ganesh Hegde 24 Nov, 2021

If a page load time goes from one second to ten seconds, the probability of a visitor bouncing, increases by 123% - Google
A user visiting your website has the patience level of a 5-year-old. Nothing against 5-year-olds (Specbee has a strict nothing against 5-year-olds policy), but as a site owner, you can’t expect anyone to wait more than 5 seconds for a page to load. Not when there’s infinite internet to explore. Even the best content will lose the audience if your pages don’t load quickly. But that’s just the human nature aspect of it. Your site speed plays a part in so much more…

A little bit of History

Google has always been on a mission to make the web faster. This goes back to 2009, when Google announced plans to “make the web faster”. The first step to those plans was to publish guidelines on website speed to persuade website owners to take a hard look at their load times.

Acting further on this, Google announced in 2010 that site speed would become one of the factors for their desktop SERP algorithms! If you were a site owner back then who was reading the not-so-subtle signs on the wall, this was a great opportunity for those already making faster loading sites in that they suddenly had a huge SEO advantage.

A few years later, by 2015, when Google announced that the number of searches performed on mobile devices exceeded those performed on desktop, it was time for site owners to adapt to a mobile first age. Google developed Accelerated Mobile Pages (AMP) and three years later, page speed for mobile was introduced as one of the ranking factors as well.

And, as if you couldn’t guess, things continue to progress (they always do) and in the summer of 2021, Google rolled out the latest addition, Page experience update to throw another critical factor affecting the current algorithm!

Data, and what you need to know!

While page speed is a complex factor, people often get confused between site speed and page speed. Though they seem similar, site speed is an average of various sample pages of the website while page speed defines how long a page takes to load. Both of these play a vital role in the page experience of a website. In addition, Core Web Vitals are an important set of metrics that the site owner should look at to provide a seamless experience for users.

While we still spend a lot of time on keywords, desktop and mobile experience, backlink, content marketing, etc, we now must also pay attention to the on-page experience. And Core Web Vitals are the new set of metrics to let you know if you provide a quality page experience.

Let’s dig in a bit more on core signals.

Largest Contentful Paint [LCP]

Referring to the page loading performance, this Google experience metric indicates the time taken for the largest chunk of information on the page to load. For example, if the page has a video (often the largest piece of information on a page), the time taken for that video to completely load so that the user can play it, refers to LCP.

This is an important metric - Google does use LCP as a ranking factor for the pages in SERP.

First Input Delay [FIP]

The First Input Delay measures the responsiveness of the page/website. This Google metric indicates the response time to a user’s first interaction on the page/site.

Example: Your page has video and after the page is loaded the user clicks on “play”.  The time taken to play the video after the click is FID.

Cumulative Layout Shift [CLS]

CLS is a Google metrics which indicates the frequency of unexpected layout shifts/changes, affecting the website’s overall appearance.
I’m sure you have seen a website where you saw something interesting and you went on to click it, but at the last moment the screen jumps and a new button loads, which you end up clicking? That a big indication of poor coding or, in some cases, a hidden motive to make users click on ads or other links!

Core Web Vital Report

The details and the reporting of Core Web Vitals can be handled through Search Console which offers detailed information of any errors and tips for improvement.

What can be done to improve speed?

When it comes to Google metrics and how website owners try to do their best to optimize their websites, it’s a bit of a race in the dark. While many website owners have an idea of how things work and what needs optimization, some owners blindly make drastic changes just hoping something works.

Without throwing things against the wall and seeing what sticks, here’s a few things that could improve the speed of your site/page.

Light, fast and furious!

Modern websites are often referred to as mini buses or limousines. They’re huge, filled with animations/graphics and have all sorts of code-heavy integrations - which makes them heavy & slow! When making a beautiful website, owners often deprioritize functionality.

Image Credits: Pingdom

To win the SEO race, it’s important that the site is light and is not loaded with unnecessary animations, fancy backgrounds, huge images, additional plugins and more. In addition to the reduced extravagance, excessive weight can also be shed by optimizing images, reducing the third-party scripts (make use of GTM), Implementing AMP, compressing the code (CSS & JS) and more.

In a CMS like Drupal, there are quite a few easy to use modules that help improve page speed.

Image Credits: Almanac

Now that your website is clean and decluttered, it’s time to give it more power! And the best way to do that is by using a proper hosting environment. Your website on a properly tuned hosting environment can work wonders when it comes to the performance. Additionally, you can also use a CDN to load your website resources faster.

Tools are your best friends!

Now that you know how important page speed is, the next thing to remember is the equal importance of monitoring. There are various tools that give you a great amount of information about your website’s speed performance (amongst a lot of other important metrics) that will also suggest the next optimizations to make. Here’s Specbee’s picks for the top 5 tools to measure your website performance:

  • Chrome Lighthouse
  • Web.dev
  • Google Page Insights
  • GTMetrix
  • Website Speed Test

The simple truth is that a majority of website visitors prefer using a mobile device to visit a website, search for something on Google, or even do some online shopping. With this trend continuing for years to come, page speed/site speed is always going to be an important factor to provide a seamless user experience! For someone new to website building, page speed optimization could be complex. But understanding how essential speed is already offers you a little bit of guiding light for that race in the dark. And for someone who already understands these metrics, you already know the importance of page speed. It might be time for you to revisit your website to keep progressing on your head start. And if someone asks what you’re doing, as people who dress up as Maverick from Top Gun for Halloween continuously say, “I feel the need for speed!”

SEO Drupal Planet Web Development Subscribe to our Newsletter Now Subscribe Leave this field blank

Leave us a Comment

  Recent Blogs Image Why Page Speed Matters and How to Get Your Website to Load Faster! Image Drupal 9 Custom Module Development – A Beginners Guide Image Easy and Effective Content Moderation in Drupal 9 (With an Example!) Want to increase your organic visibility with Drupal? TALK TO OUR DRUPAL SEO EXPERTS Featured Success Stories

A Drupal powered multi-site, multi-lingual platform to enable a unified user experience at SEMI.


Discover how our technology enabled UX Magazine to cater to their massive audience and launch outreach programs.


Discover how a Drupal powered internal portal encouraged the sellers at Flipkart to obtain the latest insights with respect to a particular domain.


Categories: FLOSS Project Planets

Python Software Foundation: Loren Crary has joined the PSF as its Director of Resource Development

Planet Python - Wed, 2021-11-24 04:00

The Python Software Foundation (PSF) is excited to welcome Loren Crary as our new Director of Resource Development!

Financial sustainability is critical to the PSF and the entire Python ecosystem as Python continues to grow in popularity. Financial sustainability will ensure we can continue providing initiatives such as our international Grants Program, supporting fiscal projects, producing PyCon US, maintaining community infrastructure such as pypi.org and python.org, supporting roles such as Python's new Developer-in-Residence and the new Packaging Project manager, and delivering on strategic goals the PSF Board sets in place. 

With that in mind, we knew we needed an expert to spearhead fundraising efforts and we're very luckily to have been able to hire Loren as that expert! Loren will be focusing on all aspects of fundraising including sponsorships, fundraisers, grants, donor stewardship, as well as supporting the PSF Board with their fundraising efforts. 

Loren spent most of the last decade leading revenue strategy for Educate!, a nonprofit social enterprise, as it scaled by 5x in both budget and impact. (In that work she got to spend a lot of time in Rwanda, Kenya, and especially Uganda—and she is excited to connect with the active Python communities in that part of the world!) She is a lawyer by training, with a JD from Stanford Law School, where she focussed on public interest law. 

Loren is jumping into the software space, learning Python for the first time, and seeking all opportunities to learn about and from the dynamic and welcoming Python community.

Welcome, Loren! We are super excited about the impact your work will have on Python and its community. 

Categories: FLOSS Project Planets

Where in the Stack?

Planet KDE - Tue, 2021-11-23 18:00

For one of my projects – not KDE-related – I have a parser, written in YACC / bison. To build the project, the bison grammar needs to be compiled (by the bison command) to C, and then the C can be compiled to the final executable. On my workstation, the bison step would fail when the build was run one-process-at-a-time in KDE konsole. Workarounds were really weird: build with a -j flag to build with more processes at once, or pipe the build-output to cat, or run the build in xterm instead of konsole. So where is the bug? In konsole, in bison, or in something underneath? It’s definitely something to do with the terminal emulator: here’s a screenshot of bison compiling a sample file successfully in xterm, and crashing in konsole, roxterm and alacritty. It crashes in cool-retro-term as well, but leaves the terminal itself in a messed-up-state. It also crashes on the FreeBSD text console.

Five terminal windows with bison crashes

The output from bison, once piped to od -c – and then it does not crash – is intriguing. Here’s a part of the output:

0000120 ** ** , u s e ‘ ** ** % d e f i 0000140 n e a p i . p u r e ’ ** ** [ 0000160 033 [ 3 5 m 033 ] 8 ; i d = 5 b 1 f 0000200 0 f d 7 0 0 0 5 d 1 7 5 2 e 1 f 0000220 4 2 3 b 0 0 0 0 0 0 0 0 ; h t t 0000240 p s : / / w w w . g n u . o r g

This is colorized output, which uses the terminal-control sequences for xterm – there is a big long list of them – to manipulate colors and styles. Most terminal emulators understand these control sequences.

Huh, there’s a URL there. It doesn’t display in the terminal, though.

URL Highlighting in Konsole Konsole displaying URLs

Konsole has a neat feature where if you display a URL, it can be highlighted, and you can click on it from konsole to open the URL. That looks like the screenshot here; if it looks like a URL, it gets underlined when you mouse over it, and you can ctrl-click to open.

There is terminal-control sequence to hide a URL within terminal output. This uses the ESC ] 8 ; sequence (it probably has a name, but I haven’t been able to find it). By default, this is ignored, as a security measure: clicking on text in your text terminal should not invoke web browsers on invisible URLs. Konsole can turn the feature on, though: go to Settings -> Edit Current Profile -> Mouse -> Miscellaneous tab, then tick the box Allow escape sequences for links. Read the WARNING that is there, because it’s on-point.

You can use the printf command to construct this kind of terminal-control sequences. From the shell, something like this:

$ printf '\e]8;;https://kde.org/\e\\GNOME\e]8;;\e\\\n' GNOME

Notice how the URL isn’t shown (it is the terminal emulator that handles that), but the text is. Notice, too, how the URL doesn’t match the text: that is what the security warning is all about.

This is actually pretty cool: bison prints a warning, and links from the warning message (which mentions a specific command-line flag) to the documentation which explains what the flag is for and provides additional context. It makes the warning message a bit more useful – if, and only if, that security-sensitive setting is on. (Which makes this cool feature from bison a lot less useful in most settings .. maybe GNOME terminal has a similar setting, but switched on by default).

OK, but looking at the bison output, specifically, there’s more than just a URL there: there is a id= part as well. Let’s add that to the printf command:

$ printf '\e]8;id=x;https://kde.org/\e\\GNOME\e]8;;\e\\\n' GNOME

Up to, and including, KDE Gear 21.08.3 (the latest release as of this writing), konsole eats the terminal-control sequence, displays only the text, but now – after adding that id=x part – the URL is no longer recognized, and you can’t click on it.

I fixed this in konsole merge request #535. Konsole assumed that that id-part was empty, when clearly it isn’t (or at least, bison thinks it is reasonable to output something there).

Konsole bug, squashed

Bug, squashed! But .. bison still crashes.

Bison Warnings

I rebuilt bison with debugging enabled. Using gdb I found that the crash was invoked from warnings_print_categories(). The code looks like this:

const char *warning = argmatch_warning_argument (&w); char ref[200]; snprintf (ref, sizeof ref, "%s#W%s", diagnostics_url, warning); begin_hyperlink (out, ref); ostream_printf (errstream, "-W%s%s", s == severity_error ? "error=" : "", warning); end_hyperlink (out);

Bison is Free Software, copyrighted by the Free Software Foundation. The code above is licensed under the GPL Version 3.

I can see what it’s doing here: composing a URL and sending that to begin_hyperlink(), then printing the name of the warning itself, then calling end_hyperlink(). Unpacking each of those functions takes me to GNU libtextstyle which is part of GNU gettext. Bison, it turns out, isn’t doing anything strange, but something is going wrong lower down in the stack.

Libtextstyle Code

Since messing around with all of bison at once was a bit inconvenient, and with the code from warnings_print_categories() as inspiration, I came up with this bit of code (and some boilerplate):

styled_ostream_set_hyperlink (errstream, argc > 1 ? argv[1] : NULL, argc > 2 ? argv[2] : NULL); ostream_printf (errstream, "hello world"); styled_ostream_set_hyperlink (errstream, NULL, NULL); ostream_flush (errstream, FLUSH_THIS_STREAM);

This allows me to test calls into libtextstyle with different hyperlink formats, URLs, etc. In the meantime I built the library with debugging support. Here is a typical crash:

#0 0x0000000000000000 in ?? () #1 0x000000080038230a in delay_output_sp (sp=0x0, ms=<optimized out>) at /usr/src/src/contrib/ncurses/ncurses/tinfo/lib_tputs.c:104 #2 0x0000000800382b81 in delay_output (ms=0) at /usr/src/src/contrib/ncurses/ncurses/tinfo/lib_tputs.c:116 #3 tputs_sp (sp=<optimized out>, sp@entry=0x7fffffffbf88, string=0x800a08083 "", string@entry=0x800a08080 "57b", affcnt=affcnt@entry=1, outc=<optimized out>) at /usr/src/src/contrib/ncurses/ncurses/tinfo/lib_tputs.c:422 #4 0x0000000800382cfb in tputs (string=0x800a08080 "57b", affcnt=1, outc=0x800278c40 <out_char>) at /usr/src/src/contrib/ncurses/ncurses/tinfo/lib_tputs.c:444 #5 0x0000000800278bb0 in out_hyperlink_change (stream=0x800a0d000, new_hyperlink=0x800a790c0, async_safe=false) at term-ostream.oo.c:1586 #6 0x000000080027979c in out_attr_change (stream=0x800a0d000, new_attr=...) at term-ostream.oo.c:1737 #7 0x0000000800278f3b in output_buffer (stream=0x800a0d000, goal_attr=...) at term-ostream.oo.c:1906 #8 0x000000080027647a in term_ostream__flush (stream=0x800a0d000, scope=FLUSH_THIS_STREAM) at term-ostream.oo.c:2052 #9 0x0000000800276f0b in term_ostream_flush (first_arg=0x800a0d000, scope=FLUSH_THIS_STREAM) at term-ostream.c:2729

In this particular case, I have set the ID to “57b”. Why? Well, I originally used the same ID as bison (see the octal dump, above). That crashes, so I had a reproducer but I wanted to cut down the size of what was being printed.

This test-code crashes in konsole. Crashes in alacritty. Does not crash in xterm or urxvt.

In any case, you can see that libtextstyle, in the function out_hyperlink_change(), is calling the ncurses function tputs() with a user-supplied string. The code is here in git as of this writing, but it looks like so:

tputs ("\033]8;id=", 1, out_ch); tputs (new_hyperlink->real_id, 1, out_ch); tputs (";", 1, out_ch); tputs (new_hyperlink->ref, 1, out_ch); tputs ("\033\\", 1, out_ch);

GNU libtextstyle is Free Software, copyrighted by the Free Software Foundation. The code above is licensed under the GPL Version 3.

Here tputs() is being called to send part of an escape code, and then again with the ID (“57b”) and then a semicolon, etc. Actual documentation of tputs() is rather hard to come by, but manpages are fairly consistent in their wording of what parameters there are for the function. Here’s a quote from the OpenBSD manpage for tputs:

The tputs routine applies padding information to the string str and outputs it. The str must be a terminfo string variable or the return value from tparm, tgetstr, or tgoto. affcnt is the number of lines affected, or 1 if not applicable.

The link ID is not a terminfo string variable, nor is it the return value of any of those three functions. Neither is the partial escape sequence, or the single semicolon, or the other two strings. I’m going to conclude that libtextstyle is Doing It Wrong. For nearly all cases, there is no problem with this! What tputs() does is call out_ch() repeatedly, so in the end the string gets written.

Anyway, let’s pipe the output to od -c again to see what’s up:

$ ./example2 2>&1 | od -c 0000000 e x a m p l e . c : 033 ] 8 ; i 0000020 d = b ; h t t p : / / e x a m p 0000040 l e . c o m / 033 \ h e l l o w 0000060 o r l d 033 ] 8 ; ; 033 \

The ID is supposed to be “57b” (see the stacktrace). Where has it gone in output? Only the “b” is left.

A little experimentation shows me that whenever the ID starts with digits, they disappear, and the example program crashes (in konsole, in alacritty, … but not in xterm or urxvt). If it starts with anything else, it’s fine.

At this point I wrote a patch for libtextstyle to write the strings directly, instead of via tputs(). That seemed like the right thing to do, given the wording of the manpage – and it solved my original problem of bison crashing. I’ve got the patch lined up to apply to FreeBSD ports, but I don’t know how to go about submitting it to GNU gettext or GNU libtextstyle.

Fix submitted to the FreeBSD ports tree as PR 260016. I don’t know when (or if) that will land.

libtextstyle bug, squashed

Bug, squashed! But .. why was it crashing in some terminal emulators, why is the ID being mangled?

ncurses code

I wrote another example program. It Does It Wrong, intentionally, by calling tputs() with a user-provided string, like so:

tputs("8675309jenny", 1, out_ch);

On FreeBSD, in all the terminals I tried, this outputs “jenny” without her number. It crashes in the usual suspects. So what’s going on here? The backtrace gives me a line number, let’s take a look at the implementation function, in the FreeBSD git repo. The link goes to the top of the function, but scrolling down a little finds this gem:

#if BSD_TPUTS /* * This ugly kluge deals with the fact that some ancient BSD programs * (like nethack) actually do the likes of tputs("50") to get delays. */

ncurses is Free Software, copyrighted by the Free Software Foundation and Thomas E. Dickey. The code above is licensed under the MIT license.

Well, I am on a BSD, so presumably that define is set, and it does something special with leading digits? Yes indeed: leading digits are consumed and treated as if they were in a $<> block (a not-particularly-well-documented feature where tputs() interprets some parts of the string it is outputting as delay-specifiers, from back when you had a real vt100 on a 2400 bps serial line).

So there we have it: inside the ncurses library, Jenny’s number is consumed and turned into a delay. This is going to call delay_output(), intending to delay for 8675309ms. That’s a little over 140 minutes, enough to let Tommy Tutone play the song thirty times! There’s plenty of decent covers, but it’s still going to get pretty repetetive.

This reinforces my belief that libtextstyle is doing it wrong, in a context where the ncurses library has this kludge.

So we now know why the numbers are being eaten, but not at all why the application then crashes in some terminals and not in others.

Let’s turn back to gdb to step though the code inside ncurses.

(gdb) next 99 NCURSES_SP_OUTC my_outch = GetOutCh(); (gdb) next 102 nullcount = (ms * _nc_baudrate(ospeed)) / (BAUDBYTE * 1000); (gdb) print my_outch $1 = (NCURSES_OUTC_sp) 0x0 (gdb) next 103 for (_nc_nulls_sent += nullcount; nullcount > 0; nullcount--) (gdb) print nullcount $2 = 186509

So there’s two things of note here: my_outch is a function pointer for printing delays; something hasn’t been set up properly, or whatever: the function pointer is NULL and we’re going to have a bad time when calling it; we’ll call it 186509 times to output those two hours of delays. (In retrospect, seeing the 0x0 address in the stack trace should have tipped me off much earlier, but I was chasing this bug initially as a konsole issue, so went diving from the top of the applications stack).

Stepping through the same code inside xterm gives me slightly different results:

(gdb) print my_outch $1 = (NCURSES_OUTC_sp) 0x0 (gdb) next 103 for (_nc_nulls_sent += nullcount; nullcount > 0; nullcount--) (gdb) print nullcount $2 = -963

Same bad function pointer, but we’ll call it -963 times. That is convenient, since the for loop is never entered and the bad pointer isn’t called. Looking back at the calculation of nullcount here, I checked the value of ospeed when running under konsole: 9600, that’s slow for a terminal but not unreasonable. Under xterm, it says -27136.

Um .. so xterm is avoiding the problem by having an uninitialized (or otherwise garbage) ospeed value, and it’s nothing special about the terminal at all.

Well! Good thing we have the ol’ stty command, which we can use to manipulate speeds. We can use stty speed 9600 for regular-speed terminals, or stty speed 38400 for fast-fast terminals. Heck, I even have a 56k modem, which was fast-fast-fast! That’s bits per second, kids, from back when you could beatbox the sound of a modem syncing and doing carrier detection.

When, in konsole, I run stty speed 9600, and then the sample application, it crashes. Crank up the speed, stty speed 38400 and it’s fine. Checking in xterm, the default speed that xterm has set is 38400. Turning it down a notch: example application crashes. Turning it back to 38400: it’s fine.

Say, -27136, that’s the 16-bit two’s complement value of 38400 (which overflows a 16-bit signed integer). Taking a look at the definition of the ospeed variable, which lives in /usr/local/termcap.h, shows me this:


And yes, on my platform short is probably a 16-bit integer.

Let’s summarize:

  • bison uses libtextstyle,
  • libtextstyle calls some ncurses functions and, it can be argued is Doing It Wrong,
  • but ncurses then uses a NULL function pointer, and that’s not libtextstyle’s fault,
  • except when the terminal baud rate overflows a signed 16 bit integer.
ncurses bug, understood

Bug, understood but not squashed. How are things elsewhere?

Looking at Linux

You know, debuginfo is pretty darn useful, and I’m glad openSUSE’s gdb downloads it automatically. It makes running my example programs much less complicated (I know how to wrangle all the bits on FreeBSD, but not on Linux).

None of the example programs crashed. Running with gdb and setting a breakpoint on delay_output_sp() showed me the following:

  • no_pad_char is set, so the problematic code path isn’t touched at all,
  • ospeed has a value that is not directly the baud rate on the tty (9600 baud yields an ospeed of 13, 38400 baud yields 15, 115200 baud yields 4098).

Looking more closely at the ospeed values, this looks like a mapping of baud rates to internal “enum” values. There are constants defined in termios.h. These have names like B38400, the internal value for 38400 baud. On Linux, the internal value is 017, which (it’s octal!) means fifteen.

There is a function in the ncurses library that maps internal values (e.g. B38400) to actual baud rates (38400 in the case of B38400). On Linux, that maps internal value 15 to 38400. And then in the calculation of how long to delay, it uses 38400 bits-per-second to figure out how many padding characters to send. It doesn’t hit that code path, but it if did, it would be calculating with the right speed.

Back to FreeBSD

On the FreeBSD side, the internal values are the same as the baud rates, e.g. B38400 is defined as 38400 rather than 017. This gives us an interesting tour of ncurses internals and C types, because we have a table of speeds, with entries that get hammered to NCURSES_OSPEED type (short .. signed short).

struct speed { int given_speed; /* values for 'ospeed' */ int actual_speed; /* the actual speed */ }; #define DATA(number) { (NCURSES_OSPEED)B##number, number } static struct speed const speeds[] = { DATA(38400), };

There’s even a comment in the source about older FreeBSD versions that had old-style versions of the defines, and how those do fit into a short. In any case, we end up with initializing an element of the speeds array with B38400, cast to short, then promoted to integer. Because 38400 overflows the short, it’s a negative value, which is promoted to a negative integer: -27136.

When looking up internal values – 15 on Linux, 38400 on FreeBSD – ncurses goes through that table, matching the given speed against the speed value obtained from the terminal (as an internal value, in a short, passed as an int). This happens in the function _nc_baudrate(). So we’re going to be comparing -27136 with values in the table, right? Not so fast: the authors know about overflow, and so there’s specific code in the library:

if (OSpeed < 0) OSpeed = (unsigned short) OSpeed;

So here we take the passed-in value (-27136), mash it to an unsigned short, so in 16 unsigned bits that comes out to 38400, then promote that to int again and assign it back to the variable Ospeed. So we are now looking for a speed value of 38400.

The table contains -27136, and when the comparison is done, it is done with (signed) integers, not shorts, and since those two values differ by 65536 – I am assuming that ints are longer than shorts and more than 16 bits – all the comparisons fail. For any internal value that exceeds 32767, the table-lookup will always fail.

In PR 256731 there is an commit, from an upstream commit, that avoids the NULL-pointer and the crash. Issues with terminal speed remain, so don’t wait around for Jenny.

ncurses bug, mostly-squashed

Bug, mostly-squashed by an update to FreeBSD.

  • Search in the bug database before starting a days-long-debugging-marathon,
  • ncurses and libtextstyle have some neat features that might make sense to use also in other text-based applications of mine.
Categories: FLOSS Project Planets

Event Organizers: What Do You Need to Create a Drupal Event Website? Fill Out Our Survey

Planet Drupal - Tue, 2021-11-23 15:33

To help ensure we are meeting the needs of other Drupal community events, we are have created this survey to help us prioritize the feature sets for our development roadmap. Whether you are a new or veteran Drupal event organizer please complete this short survey to tell us how a Drupal Event Platform could help you. Fill Out Our Survey

The Drupal Event Platform (DEP) is a committee within the Drupal Event Organizers Working Group. Our goal is to create a Drupal application MVP that helps event organizers build and maintain the entire event production process, from collecting session submissions to assigning presentations to the schedule.

COMPLETE OUR SURVEY HEREFile attachments:  dep-survey.jpg
Categories: FLOSS Project Planets

PyCoder’s Weekly: Issue #500 (Nov. 23, 2021)

Planet Python - Tue, 2021-11-23 14:30

#500 – NOVEMBER 23, 2021
View in Browser »

Python News: What’s New From October 2021?

The Python community gave a warm welcome to Python 3.10 in October 2021. In this article, you’ll catch up on some big events that happened this past month, including Python being named TIOBE’s most popular programming language.

Ignore All Web Performance Benchmarks, Including This One

”[…] measuring web application performance is incredibly complex […] In this article you can see the results of my effort in understanding and fixing this benchmark, re-running it, and finally arriving at a shocking revelation.”
MIGUEL GRINBERG • Shared by Caleb Hattingh

Cheat Sheet: 9 Python Security Best Practices

Avoid PyYAML vulnerabilities by using yaml.SafeLoader() instead of yaml.Loader(). Learn 8 more tips in the Python security cheat sheet →
SNYK.IO sponsor

Building a Neural Network and Making Predictions With Python AI

In this step-by-step course, you’ll build a neural network from scratch as an introduction to the world of artificial intelligence (AI) in Python. You’ll learn how to train your neural network and make accurate predictions based on a given dataset.

Static Duck Typing in Python With Protocols

Protocols in Python provide a nice way to use duck typing while still having some static type guarantees. You can define contracts for your functions without caring too much about the actual types of your inputs.

Minimax Algorithm and Alpha-Beta Pruning

The minimax algorithm and alpha-beta pruning can be used to implement AIs to play games. This tutorial will teach you about the minimax algorithm and alpha-beta pruning, from a beginner’s perspective.
RODRIGO GIRÃO SERRÃO • Shared by Rodrigo Girão Serrão

PyPI User Feedback: PyPI Team Features

The PyPI team is seeking feedback from teams using PyPI. If you use PyPI as a team, consider responding to the (short) poll form at the link.

Django 4.0 Release Candidate 1 Released


CPython Dev In-Residence Weekly Report, November 15–21


Discussions Python Is Getting Faster, and So Is Django


You Can Insert Emoji Using "\N{NAME_OF_EMOJI}"


Python Jobs Senior Backend Engineer (Anywhere)


Senior Backend Software Engineer (Anywhere)


Senior Python Engineer (Anywhere)


Senior Software Engineer - Backend (United States)


Full Stack Software Engineer (Washington D.C., USA)


Senior Software Engineer (Washington D.C., USA)


Python Backend Engineer (Hasselt, Belgium)


Full Stack Developer (Anywhere)

Level 12

More Python Jobs >>>

Articles & Tutorials Building a Content Aggregator and Working With RSS in Python

Have you wanted to work with RSS feeds in Python? Maybe you’re looking for a new project to build for your portfolio that uses Django, unit tests, and custom commands. This week on the show, it’s Real Python author Ricky White talking about his recent step-by-step project titled, “Build a Content Aggregator in Python.”

Combine Coverage for Different Python Versions

Is it enough to run code coverage for a single Python version? Probably, but not necessarily. While this is or was a super important topic when you still needed to support Python 2, also now this is still valid as Python 3 moves forward quickly and requires to have different code paths for different versions.
JÜRGEN GMACH • Shared by Jürgen Gmach

Connect, Integrate, & Automate Your Data  - From Python, or Any Other Application or Tool

CData makes it easier to unlock the value of data — simplifying connectivity between applications and data sources. Our SQL-based connectors streamline data access making it easy to access real-time data from on-premise and cloud databases, SaaS, APIs, NoSQL and more. Visit cdata.com to learn more →

Python Community Interview With Eric Wastl

An interview with Eric Wastl, a Senior Architect at TCGPlayer and the founder of Advent of Code (AoC). Join us as we discuss the inception of AoC, how to avoid common pitfalls when solving code challenges in AoC, and what Eric gets up to in the “3.8 seconds” of free time he has per year.

Building Lists With Python’s .append()

In this step-by-step course, you’ll learn how Python’s .append() works and how to use it for adding items to your list in place. You’ll also learn how to code your own stacks and queues using .append() and .pop().

Tic Tac Toe Game in Python

Introduction into game development with Python by writing a simple Tic Tac Toe game. Best development practices and designed patterns are included into this step-by-step tutorial.
MISHA BEHERSKY • Shared by Misha Behersky

Using PostgreSQL Views in Django

This article explains how views differ from materialized views and shows how using Postgres views makes querying aggregated data easier and (with materialized views) faster.
JOSH ALLETTO • Shared by Manuel Weiss

Replicating Minecraft World Generation in Python

“I used to play Minecraft as a kid and I always wondered how does it generate never-ending worlds. In this article, I am going to try to replicate this in Python.”

Python String Formatting Comparison

This article compares the three main string formatting methods in Python and suggests which methods to use in each situation.

Get Your Next Python Job through Hired

Hired is home to thousands of companies from startups to Fortune 500s actively hiring developers, data scientists, mobile engineers & more. Create a profile & let hiring managers reach out to you. It’s completely free for job-seekers, so don’t wait!
HIRED sponsor

Speed Up Your Conda Installs With Mamba

“Conda installs are very slow, but you can speed them with a much-faster Conda reimplementation called Mamba.”

My Journey to Type Checking 7521 Lines of Python

A small recap on how the author added type annotations to the Logilab’s RQL package.
PATRICK NSUKAMI • Shared by Patrick Nsukami

Peculiar Self-References

Here is a tiny Python example that creates a self-referential list…

Movie Name Generation Using GPT-2


How I Accidentally Built a Podcast API Business With Python/Django

WENBIN FANG • Shared by Wenbin Fang

Beyond the Basics: Data Visualization in Python

GITHUB.COM/STEFMOLIN • Shared by Stefanie Molin

Projects & Code prometeo: Experimental Python-to-C Transpiler and DSL


lark-parser: Friendly, Production-Ready, and Comprehensive Parsing Toolkit


Spack: A Flexible Package Manager


friendly-traceback: Now Available in Spanish


objprint: Print Python Objects in Human Readable Format


Comment Webpage Screenshot on Pull Requests (GitHub Action)

GITHUB.COM/SAADMK11 • Shared by Maksudul Haque

Events Weekly Real Python Office Hours Q&A (Virtual)

November 23, 2021

Xtreme Python

November 24 to November 25, 2021

PythOnRio Meetup

November 27, 2021

Introduction to the Python Programming Language (In Persian)

November 30, 2021

Python Sheffield

November 30, 2021

FlaskCon 2021 Online

December 1 to December 5, 2021

PyCon Tanzania

December 1 to December 6, 2021

PyCode Conference

December 2 to December 4, 2021

Happy Pythoning!
This was PyCoder’s Weekly Issue #500.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Categories: FLOSS Project Planets

ItsMyCode: Python Ternary Operator

Planet Python - Tue, 2021-11-23 14:00

ItsMyCode |

In Python, Ternary operator, also called conditional expressions, are operators that evaluate something based on a binary condition. Ternary operators provide a shorthand way to write conditional statements, which makes the code more concise.

In this tutorial, let us look at what is the ternary operator and how we can use it in our code with some examples.

Syntax of Ternary Operator

The ternary operator is available from Python 2.5 onwards, and the syntax is:

[value_if_true] if [expression] else [value_if_false] 

Simpler way to represent ternary operator Syntax

<expression 1> if <condition> else <expression 2> 

Note: The conditionals are an expression, not a statement. It means that you can’t use assignment statements or pass other statements within a conditional expression.

Introduction to Python Ternary Operator

Let us take a simple example to check if the student’s result is pass or fail based on the marks obtained. 

Using a traditional approach

We are familiar with “if-else” conditions lets first write a program that prompts to enter student marks and returns either pass or fail based on the condition specified.

marks = input('Enter the marks: ') if int(marks) >= 35: print("The result is Pass") else: print("The result is Fail")


Enter the marks: 55 The result is Pass

Now instead of using a typical if-else condition, let us try using ternary operators. 

Example of Ternary Operator in Python

The ternary operator evaluates the condition first. If the result is true, it returns the value_if_true. Otherwise, it returns value_if_false

The ternary operator is equivalent to the if-else condition. 

If you are from a programming background like C#, Java, etc., the ternary syntax looks like below.

if condition: value_if_true else: value_if_true condition ? value_if_true : value_if_false

However, In Python, the syntax of the ternary operator is slightly different. The following example demonstrates how we can use the ternary operator in Python.

# Python program to demonstrate ternary operator marks = input('Enter the marks: ') print("The result is Pass" if int(marks)>=35 else "The result is Fail")


Enter the marks: 34 The result is Fail

The post Python Ternary Operator appeared first on ItsMyCode.

Categories: FLOSS Project Planets

Enrico Zini: Really lossy compression of JPEG

Planet Debian - Tue, 2021-11-23 13:58

Suppose you have a tool that archives images, or scientific data, and it has a test suite. It would be good to collect sample files for the test suite, but they are often so big one can't really bloat the repository with them.

But does the test suite need everything that is in those files? Not necesarily. For example, if one's testing code that reads EXIF metadata, one doesn't care about what is in the image.

That technique works extemely well. I can take GRIB files that are several megabytes in size, zero out their data payload, and get nice 1Kb samples for the test suite.

I've started to collect and organise the little hacks I use for this into a tool I called mktestsample:

$ mktestsample -v samples1/* 2021-11-23 20:16:32 INFO common samples1/cosmo_2d+0.grib: size went from 335168b to 120b 2021-11-23 20:16:32 INFO common samples1/grib2_ifs.arkimet: size went from 4993448b to 39393b 2021-11-23 20:16:32 INFO common samples1/polenta.jpg: size went from 3191475b to 94517b 2021-11-23 20:16:32 INFO common samples1/test-ifs.grib: size went from 1986469b to 4860b

Those are massive savings, but I'm not satisfied about those almost 94Kb of JPEG:

$ ls -la samples1/polenta.jpg -rw-r--r-- 1 enrico enrico 94517 Nov 23 20:16 samples1/polenta.jpg $ gzip samples1/polenta.jpg $ ls -la samples1/polenta.jpg.gz -rw-r--r-- 1 enrico enrico 745 Nov 23 20:16 samples1/polenta.jpg.gz

I believe I did all I could: completely blank out image data, set quality to zero, maximize subsampling, and tweak quantization to throw everything away.

Still, the result is a 94Kb file that can be gzipped down to 745 bytes. Is there something I'm missing?

I suppose JPEG is better at storing an image than at storing the lack of an image. I cannot really complain :)

I can still commit compressed samples of large images to a git repository, taking very little data indeed. That's really nice!

Categories: FLOSS Project Planets

Chocolate Lily: What do the usage data on Drupal.org actually mean?

Planet Drupal - Tue, 2021-11-23 11:15

Many Drupal insiders recognize that data on Drupal usage that's collected and displayed on Drupal.org have their limitations. Since 2018 there's been a proposed Drupal core telemetry initiative to expand and improve the data collected. Meantime, though, the usage stats are widely referred to and cited. So it's worth spending a bit of time with them. What do they actually capture?

Categories: FLOSS Project Planets

Redfin Solutions: Redfin’s Delta Site System: Helping Clients Create Customizable Websites at Scale

Planet Drupal - Tue, 2021-11-23 10:41
Redfin's Delta Site System (DSS) helps clients who have a need to quickly launch and maintain new Drupal websites. DSS gives our clients the ability to define a “template” website, so that new sites can use the same themes (look and feel), modules (enhanced functionality), and configuration (content types and taxonomies), while also allowing for customization at the individual website level.
Categories: FLOSS Project Planets

Python for Beginners: Depth First Traversal in Python

Planet Python - Tue, 2021-11-23 10:03

Graphs are non linear data structures used to represent relationships between different objects. In this article, we will discuss depth first traversal algorithm to print the vertices in a graph. We will also implement the depth first traversal algorithm in python to illustrate the working of the algorithm.

What is depth first traversal?

Depth first traversal is a graph traversal algorithm in which we start from a vertex of a graph and print its value. Then we move to one of the neighbors of the present vertex and print its values. If there are no neighbors of the current vertex that have to be printed, we move to the previous vertex to see if all of their neighbors are printed. If not, we select one of the neighbors and print its value. We repeat this process until all the vertices of the graph are printed.  It should be noted that we have to print each vertex only once. 

Let us try this process on the following graph.


Here, Let us start from the vertex A. First, we will print A. After that we will select one of the vertices from B, D,E and F as all are neighbors of A. 

Let us select B. After printing B, we will select one of the vertices from A, C and F as these are neighbors of B. Here, A is already printed so it won’t be selected.

Let us select C. After printing C, we have no neighbors of C to print. So, we will move to the previous vertex of C i.e. vertex B to check if all the neighbors of B are already printed. Here, F has not been printed yet. So, we will select F.

After printing F, we have to select one of the vertices from A and B as these are neighbors of F. But, both these vertices are already printed. So, we will move to previous vertex of F i.e. B. 

At B, we can see that all the neighbors of B have already been printed. So, we will move to previous vertex of B i.e. A. 

At A, neighbors D and E have not been printed yet, So we will select one of the vertices.

Let us select D. After printing D, we can see that D does not have any neighbors to print. So, we will move to its previous vertex i.e A. 

Now, only one neighbor of A has not been printed i.e. E. So, we will print E.

At this point, all of the vertices of the graph have been printed in the order A, B, C, F, D, E. Hence, we will stop this process.

You can observe that there may be many depth first traversal of a single graph based on the neighbor that we choose. 

Algorithm for depth first traversal

The algorithm for depth first traversal of a graph is implemented using a stack data structure. Here, we will assume that we have a connected graph. In other words, we can reach each vertex of the graph from the starting vertex.

We will maintain a stack to store the vertices that have not been printed and a list to store the visited vertices. After that we will process the graph using the following algorithm.

  1. Create an empty stack S to store the vertices that have not been printed.
  2. Create an empty list L to store the visited vertices.
  3. Insert the source vertex into S. Also, insert the source vertex to L.
  4. If  the stack S is empty, Go to 9 else go to 5.
  5. Take out a vertex v from S.
  6. Print the value in v. 
  7. Insert all the unvisited neighbors of v into the stack S and list L.
  8. Go to 4.
  9. Stop.

This Algorithm can be demonstrated using the following source code for the graph given in the figure above.

graph = {'A': ['B', 'D', 'E', 'F'], 'D': ['A'], 'B': ['A', 'F', 'C'], 'F': ['B', 'A'], 'C': ['B'], 'E': ['A']} print("Given Graph is:") print(graph) def DFS_Algorithm(input_graph, source): stack = list() visited_list = list() print("At {}, adding vertex {} to stack and visited list".format(source, source)) stack.append(source) visited_list.append(source) while stack: vertex = stack.pop() print("At vertex :", vertex) print("Printing vertex:", vertex) for u in input_graph[vertex]: if u not in visited_list: print("At {}, adding vertex {} to stack and visited list".format(vertex, u)) stack.append(u) visited_list.append(u) print("Vertices in visited list are:", visited_list) print("Vertices in stack are:", stack) print("Explanation of DFS traversal of graph with source A is:") DFS_Algorithm(graph, "A")


Given Graph is: {'A': ['B', 'D', 'E', 'F'], 'D': ['A'], 'B': ['A', 'F', 'C'], 'F': ['B', 'A'], 'C': ['B'], 'E': ['A']} Explanation of DFS traversal of graph with source A is: At A, adding vertex A to stack and visited list At vertex : A Printing vertex: A At A, adding vertex B to stack and visited list At A, adding vertex D to stack and visited list At A, adding vertex E to stack and visited list At A, adding vertex F to stack and visited list Vertices in visited list are: ['A', 'B', 'D', 'E', 'F'] Vertices in stack are: ['B', 'D', 'E', 'F'] At vertex : F Printing vertex: F Vertices in visited list are: ['A', 'B', 'D', 'E', 'F'] Vertices in stack are: ['B', 'D', 'E'] At vertex : E Printing vertex: E Vertices in visited list are: ['A', 'B', 'D', 'E', 'F'] Vertices in stack are: ['B', 'D'] At vertex : D Printing vertex: D Vertices in visited list are: ['A', 'B', 'D', 'E', 'F'] Vertices in stack are: ['B'] At vertex : B Printing vertex: B At B, adding vertex C to stack and visited list Vertices in visited list are: ['A', 'B', 'D', 'E', 'F', 'C'] Vertices in stack are: ['C'] At vertex : C Printing vertex: C Vertices in visited list are: ['A', 'B', 'D', 'E', 'F', 'C'] Vertices in stack are: []

In the output, you can observe that all the vertices are finally present in the visited list and they are printed in the order A, F, E, D, B,C. You can see that the order of vertices is not similar to what we have discussed in the previous section. This is due to the fact that we have many options to choose when we are selecting neighbors of a vertex and the order will depend on the vertex we select.

Implementation of Depth first traversal in Python

As we have discussed the general idea for depth first traversal of a graph and observed how the algorithm works using the python program, we can implement the depth first traversal algorithm as follows.

graph = {'A': ['B', 'D', 'E', 'F'], 'D': ['A'], 'B': ['A', 'F', 'C'], 'F': ['B', 'A'], 'C': ['B'], 'E': ['A']} print("Given Graph is:") print(graph) def dfs_traversal(input_graph, source): stack = list() visited_list = list() stack.append(source) visited_list.append(source) while stack: vertex = stack.pop() print(vertex, end=" ") for u in input_graph[vertex]: if u not in visited_list: stack.append(u) visited_list.append(u) print("DFS traversal of graph with source A is:") dfs_traversal(graph, "A")


Given Graph is: {'A': ['B', 'D', 'E', 'F'], 'D': ['A'], 'B': ['A', 'F', 'C'], 'F': ['B', 'A'], 'C': ['B'], 'E': ['A']} DFS traversal of graph with source A is: A F E D B C Conclusion

In this article, we have discussed the depth first traversal algorithm for a fully connected graph and we have also implemented it in Python. We have used a list as a stack in our implementation but stacks can also be implemented using a linked list in Python. TO read more about other algorithms, you can read this article on In-order tree traversal in Python.

The post Depth First Traversal in Python appeared first on PythonForBeginners.com.

Categories: FLOSS Project Planets

Jacob Rockowitz: Part 2: Good Drupal Leadership: What is a good plan for implementing a Drupal website/application for organizations?

Planet Drupal - Tue, 2021-11-23 09:14

Good Drupal leadership begins with a good plan. A good plan requires a vision specific to its particular business. Drupal, like any software, is a tool that empowers an organization to build its vision and address its business requirements. The fact that a Drupal website/application consists of Drupal core, with contributed modules and themes, that are glued together using custom code, creates a unique set of challenges to ensure each aspect is done right.

Drupal is both a Swiss Army knife with a massive variety of tools and a set of Lego®-like building blocks used to compose a custom and ideally optimized solution that addresses each and every business requirement with precision. Drupal is highly flexible, and presents a variety of options. Having a plan is essential - without one, an organization could find itself in a pickle with an unstable and insecure website/application.

Planning challenges

Recognizing that Drupal provides flexibility helps organizations work towards a plan for building and maintaining a stable website.


Drupal is a modular framework that allows developers to add and remove features as needed. There are several ways to solve common feature requests and dozens of ways to enhance existing features and functionality. Drupal's flexibility is one of its strengths that in turn requires effort to ensure stability and maintainability.


Drupal core provides a stable code base to build from, but as contributed modules are added to a Drupal stack, each module has different levels of support and stability. Realistically, regressions and issues will inevitably occur when updating Drupal core and modules. If a regression is caught before it is pushed to production, I think it’s reasonable to reconsider labeling it a regression or just part of the process of updating...Read More

Categories: FLOSS Project Planets

Real Python: Building a Neural Network &amp; Making Predictions With Python AI

Planet Python - Tue, 2021-11-23 09:00

If you’re just starting out in the artificial intelligence (AI) world, then Python is a great language to learn since most of the tools are built using it. Deep learning is a technique used to make predictions using data, and it heavily relies on neural networks. This course will show you how to build a neural network from scratch.

In a production setting, you would use a deep learning framework like TensorFlow or PyTorch instead of building your own neural network. That said, having some knowledge of how neural networks work is helpful because you can use it to better architect your deep learning models.

In this course, you’ll learn:

  • What artificial intelligence is
  • How both machine learning and deep learning play a role in AI
  • How a neural network functions internally
  • How to build a neural network from scratch using Python

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

KDE News: Overview Effect, Better Notifications... New Recording Setup!

Planet KDE - Tue, 2021-11-23 05:50
💸💸 Help me contribute to KDE and do these videos: 💸💸 Patreon: https://www.patreon.com/niccolove Youtube: https://www.youtube.com/channel/UCONH73CdRXUjlh3-DdLGCPw/join Paypal: https://paypal.me/niccolove Stay in the loop: https://t.me/veggeroblog My website is https://niccolo.venerandi.com and if you want to contact me, my telegram handle is [at] veggero.
Categories: FLOSS Project Planets

Specbee: Drupal 9 Custom Module Development – A Beginners Guide

Planet Drupal - Tue, 2021-11-23 05:31
Drupal 9 Custom Module Development – A Beginners Guide Mohammed Farhaz 23 Nov, 2021

The power of Drupal lies with its modules. With thousands of core and contributed Drupal modules to choose from, Drupal developers can easily implement ones that meet their needs. But what if the requirements are very specific and you don’t have a module available for it? How can you customize a Drupal website for a client who has unique needs that just cannot be accomplished with core or contributed modules? Enter Custom modules.

Custom modules in Drupal 9 give you the flexibility to create new features and/or tweak existing features to accommodate a business’ unique and growing aspirations. Drupal custom modules propels innovation and enhancements to help enterprises expand their horizons. This article should help you get started with creating your very own Drupal 9 module

Drupal 8 has now reached EOL. Now is the time to migrate to Drupal 9. However, if you're still trying your hands out at creating a custom module in Drupal 8, these steps will work for a Drupal 8 setup too. 

Getting started with Module development in Drupal 9

Let’s now get started with creating a custom module in Drupal 9 in a few easy steps:

Step 1: Name the Drupal 9 Module

First, we need to create a custom module under ‘web/modules/custom’ folder. We will name the folder as welcome_module.

Some things that you should keep in mind before giving a machine name to your module are:

  • It should not start with uppercase letters.
  • There should not be any spaces between the words.
Step 2: Get noticed with the info.yml file

We have to create a yaml file with the machine name of our module for Drupal to be able recognize the module. I’ll create a yaml file like this welcome_module.info.yml.
Here is our welcome_module.info.yml file created under "welcome" directory we created in Step 1.

name: Welcome Module type: module description: 'First Custom Drupal 9 Module.' package: Custom version: 1.0 core_version_requirement: ^8 || ^9

name: Welcome Module (The name displayed on the modules list in Drupal) 
type: module - (Declaring that this is a module or theme) 
description: Custom Example Drupal 9 Module (Description of the module) 
package: Custom - (Mentioning that this is a custom module) 
version: 1.0 - (Module version) 
core_version_requirement: ^8 || ^9 - (Drupal version)

Step 3: Creating the routing file with routing.yml

Next step is to add a welcome_module.routing.yml file under the "welcome" directory:

welcome_module.welcome: path: '/welcome/page' defaults: _controller: '\Drupal\welcome_module\Controller \WelcomeController::welcome' _title: 'Welcome to My Module in Drupal 9' requirements: _permission: 'access content'

The first line is the route name [welcome_module.my_welcome].
Under path, we specify the URL path we want this route to register. This is the URL to the route.
Under defaults, we have two things: the _controller which references a method on the WelcomeController class and the default page title (_title).
Under requirements, we specify the permission of the accessing. User needs to have to be able to view the page.

Step 4: Adding a Controller 

Create a folder "modules/custom/welcome_module/src/Controller". In this folder, create a file named "WelcomeController.php" with the following content:

<?php namespace Drupal\welcome_module\Controller; class WelcomeController {   public function welcome() {     return array(       '#markup' =----> 'Welcome to our Website.'     );   } }

Now Login to your Drupal site and enable your module. To check if it functions properly, visit the path you specified in your routing file, that is /welcome/page. If you get the ‘Page Not Found’ error, then clear the cache by navigating to admin->configuration->performance. Check again and it should function properly this time.

Controller page

That’s it! You’ve successfully created your very first custom module in Drupal 9.
Finally, when you go to /welcome/page URL, you'll see Title “Welcome to My Module in Drupal 9" and markup “Welcome to our Website." text printed from your module.

The great advantage of CMSs like Drupal is their ability to create websites without the need for developers to delve into coding. However, when you get to build your own custom modules in Drupal 9 and customize them according to your needs, the real power of Drupal is unleashed. Need expert help creating simple to complex custom modules? We'd love to help!

Drupal Planet Drupal 9 Drupal 9 Module Drupal Development Subscribe to our Newsletter Now Subscribe Leave this field blank

Leave us a Comment

  Recent Blogs Image Drupal 9 Custom Module Development – A Beginners Guide Image Easy and Effective Content Moderation in Drupal 9 (With an Example!) Image How to Create a Custom Drupal 9 Theme in 9 Simple Steps Want to extract the maximum out of Drupal? TALK TO US Featured Success Stories

A Drupal powered multi-site, multi-lingual platform to enable a unified user experience at SEMI.


Discover how our technology enabled UX Magazine to cater to their massive audience and launch outreach programs.


Discover how a Drupal powered internal portal encouraged the sellers at Flipkart to obtain the latest insights with respect to a particular domain.


Categories: FLOSS Project Planets

Talk Python to Me: #342: Python in Architecture (as in actual buildings)

Planet Python - Tue, 2021-11-23 03:00
At PyCon 2017, Jake Vanderplas gave a great keynote where he said, "Python is a mosaic." He described how Python is stronger and growing because it's being adopted and used by people with diverse technical backgrounds. In this episode, we're adding to that mosaic by diving into how Python is being used in the architecture, engineering, and construction industry. Our guest, Gui Talarico, has worked as an architect who help automate that world by bringing Python to solve problems others were just doing by point-and-click tooling. I think you'll enjoy this look into that world. We also touch on his project pyairtable near the end as well.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Pyninsula Python in Architecture Talk</b>: <a href="https://www.youtube.com/watch?v=H6BKT4rLncM" target="_blank" rel="noopener">youtube.com</a><br/> <b>Using technology to scale building design processes at WeWork talk</b>: <a href="https://www.youtube.com/watch?v=Qbdt8kMj2Eo&t=90s" target="_blank" rel="noopener">youtube.com</a><br/> <b>Revit software</b>: <a href="https://www.autodesk.com/products/revit/overview" target="_blank" rel="noopener">autodesk.com</a><br/> <b>Creating a command in pyRevit</b>: <a href="https://www.notion.so/Create-Your-First-Command-2509b43e28bd498fba937f5c1be7f485" target="_blank" rel="noopener">notion.so</a><br/> <b>IronPython</b>: <a href="https://ironpython.net/" target="_blank" rel="noopener">ironpython.net</a><br/> <b>Python.NET</b>: <a href="https://github.com/pythonnet/pythonnet" target="_blank" rel="noopener">github.com</a><br/> <b>revitpythonwrapper</b>: <a href="https://revitpythonwrapper.readthedocs.io/en/latest/" target="_blank" rel="noopener">readthedocs.io</a><br/> <b>aec.works site</b>: <a href="https://aec.works/companies?sort=updated" target="_blank" rel="noopener">aec.works</a><br/> <b>Speckle</b>: <a href="https://speckle.systems/" target="_blank" rel="noopener">speckle.systems</a><br/> <b>Ladybug Tools</b>: <a href="https://www.ladybug.tools/" target="_blank" rel="noopener">ladybug.tools</a><br/> <b>Airtable</b>: <a href="https://www.airtable.com/" target="_blank" rel="noopener">airtable.com</a><br/> <b>PyAirtable</b>: <a href="https://pyairtable.readthedocs.io/" target="_blank" rel="noopener">pyairtable.readthedocs.io</a><br/> <b>PyAirtable ORM</b>: <a href="https://pyairtable.readthedocs.io/en/latest/orm.html" target="_blank" rel="noopener">pyairtable.readthedocs.io</a><br/> <b>Revitron</b>: <a href="https://github.com/revitron/revitron" target="_blank" rel="noopener">github.com</a><br/> <b>WeWork</b>: <a href="https://www.wework.com/" target="_blank" rel="noopener">wework.com</a><br/> <b>Article: Using Airtable as a Content Backend</b>: <a href="https://medium.com/@gtalarico/using-airtable-as-a-content-backend-e373cd0d9974" target="_blank" rel="noopener">medium.com</a><br/> <b>Python is a Mosaic Talk</b>: <a href="https://www.youtube.com/watch?v=ZyjCqQEUa8o" target="_blank" rel="noopener">youtube.com</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=Fmr2GP5aojE" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/342/python-in-architecture-as-in-actual-buildings" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>---------- Stay in touch with us ----------</b><br/> <b>Subscribe on YouTube (for live streams)</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Twitter</b>: <a href="https://twitter.com/talkpython" target="_blank" rel="noopener">@talkpython</a><br/> <b>Follow Michael on Twitter</b>: <a href="https://twitter.com/mkennedy" target="_blank" rel="noopener">@mkennedy</a><br/></div><br/> <strong>Sponsors</strong><br/> <a href='https://shortcut.com/talkpython'>Shortcut</a><br> <a href='https://talkpython.fm/linode'>Linode</a><br> <a href='https://talkpython.fm/assemblyai'>AssemblyAI</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>
Categories: FLOSS Project Planets