Feeds

Edward Betts: Find link needs a rewrite, the visual editor broke it

Planet Debian - Wed, 2022-06-15 18:22

Find link is a tool that I wrote for adding links between articles in Wikipedia. Given an article title, find link will find other articles that include the entered article title but no link to the article. There is the option to edit the found articles and add the missing link.

For example, you might want to find missing links to the gig economy article.

I originally wrote the tool in 2008 when the MediaWiki software didn't have a rich-text editor. Wikipedia articles were edited by writing wiki markup in MediaWiki syntax. Since then MediaWiki has evolved and now has rich-text editing via the visual editor. Users don't need to know how to write wiki markup to modify an article.

Within MediaWiki there is a user preference to disable the visual editor and stick with editing via the original wiki markup.

Find link edits articles by taking the article text, adding the missing link, and sending the user to the changes view of the modified article on Wikipedia, if they're happy with the change they hit save. This only works with the original editor, it doesn't work with the visual editor.

English Wikipedia has had the visual editor enabled by default since 2016. For somebody to use find link they need to disable the visual editor in their Wikipedia preferences first.

Fixing this bug means quite a significant change to how the tool works.

My plan is to rewrite find link to save edits directly without needing to send the user to Wikipedia article edit change view page to make the edits. Users will authenticate with their Wikipedia account via OAuth and give permission for find link to edit articles on their behalf.

Some of my other tools use OAuth for editing OpenStreetMap and Wikidata, so I'm confident about using it to edit Wikipedia.

The source code for find link is on GitHub.

I'll post updates here as I make progress on the rewrite.

Categories: FLOSS Project Planets

GNU Taler news: GNU Taler Scalability

GNU Planet! - Wed, 2022-06-15 14:34
Anonymity loves company. Hence, to provide the best possible anonymity to GNU Taler users, the scalability of individual installations of a Taler payment service matters. While our design scales nicely on paper, NGI Fed4Fire+ enabled us to evaluate the transaction rates that could be achieved with the actual implementation. Experiments were conducted by Marco Boss for his Bachelor's thesis at the Bern University of Applied Sciences to assess bottlenecks and suggest avenues for further improvement.
Categories: FLOSS Project Planets

Drupal blog: Drupal 9.4.0 is available

Planet Drupal - Wed, 2022-06-15 13:56
What’s new in Drupal 9.4.0?

The fourth feature release of Drupal 9 brings a whole new frontend look with the Olivero theme by default and a refreshed backend interface with the Claro theme. There is also a new starterkit theme generator, better image loading performance and easier permission management.

Drupal now uses the Olivero frontend theme by default

When you install Drupal 9.4.0, it will look quite different from previous releases because it uses the new modern Olivero frontend theme. While the theme looks beautiful, it also has superb accessibility and adapts well to various display sizes.

The theme is named after Rachel Olivero (1982-2019). She was the head of the organizational technology group at the National Federation of the Blind, a well-known accessibility expert, a Drupal community contributor, and a friend to many.

Drupal now uses the Claro backend theme by default!

The Claro backend theme has been in the works for a while. It became stable and the default administration theme in Drupal 9.4.0. The new theme brings a modern look to the backend interface of Drupal. It has been available as a core experimental theme for some time, so it is well-tested with contributed projects and real-world sites.

A delicious addition to the Umami demo in core is a new Borscht recipe (pictured), with a dedication to the fantastic Ukrainian Drupal community.

New experimental Starterkit theme and theme generator

Drupal 9.4.0 ships with a new experimental Starterkit theme and theme generator. The new Starterkit theme is used as a basis to generate new standalone themes, rather than being extended at runtime like the Classy core base theme. Currently, the markup provided by the Starterkit theme is the same as Classy's, but its markup can be improved in future minor releases (whereas Classy's can't), so once it becomes stable, Starterkit will replace Classy. For more information, read the blog post on how the new starterkit will change theme creation in Drupal 10!

New lazy loading configuration option added to image fields

A new lazy loading configuration option is added to image fields in 9.4.0 and most image fields shipped in core are now configured to lazy load. This helps browsers to delay downloading and displaying them until they become visible, which speeds up general page display.

Easier permission management for content types, vocabularies, etc.

When editing content types, vocabularies, and so on, site administrators previously had no way to control permissions in context for these entity bundles in the same interface. With Drupal 9.4.0 a new "Manage permissions" tab displays the permissions that depend on the given type, making them easier to configure correctly.

Improvements to drupal/core-recommended for security update management

The drupal/core-recommended metapackage now allows patch-level updates for Composer dependencies. This means that site owners using drupal/core-recommended can now install most Composer dependency security updates themselves, without needing to wait for an upstream release of Drupal core that updates the affected package.

What does this release mean for me? Drupal 8 site owners

Drupal 8 is end of life as of November 17, 2021. Upgrade from Drupal 8 to at least Drupal 9.3.x as soon as possible to continue receiving security coverage. Upgrading is supported directly from 8.8.x and 8.9.x.

Drupal 7 site owners

Drupal 7 support was extended until November 1, 2023, and it will continue to receive bug and security fixes throughout this time. On the other hand, the migration path for Drupal 7 sites to Drupal 9 is stable. Read more about the migration to Drupal 9.

Translation, module, and theme contributors

Minor releases like Drupal 9.4.0 include backwards-compatible API additions for developers as well as new features.

Since minor releases are backwards-compatible, modules, themes, and translations that supported Drupal 9.3.x and earlier will be compatible with 9.4.x as well. However, the new version does include some changes to strings, user interfaces, internal APIs, and API deprecations. This means that some small updates may be required for your translations, modules, and themes. Read the 9.4.0 release notes for a full list of changes that may affect your modules and themes.

This release has further advanced the Drupal project and represents the efforts of hundreds of volunteers and contributors from various organizations. Thank you to everyone who contributed to Drupal 9.4.0!

Categories: FLOSS Project Planets

Nonprofit Drupal posts: June Drupal for Nonprofits Chat

Planet Drupal - Wed, 2022-06-15 11:55

We are looking for an additional co-moderator for this group! Reach out to Jess or Johanna to learn more about what's involved.

Our normally scheduled call to chat about all things Drupal and nonprofits will happen TOMORROW, Thursday, June 16 at 1pm ET / 10am PT. (Convert to your local time zone.)

No pre-defined topics on the agenda this month, so join us for an informal chat about anything at the intersection of Drupal and nonprofits. Got something specific on your mind? Feel free to share ahead of time in our collaborative Google doc: https://nten.org/drupal/notes!

All nonprofit Drupal devs and users, regardless of experience level, are always welcome on this call.

This free call is sponsored by NTEN.org and open to everyone. 

  • Join the call: https://us02web.zoom.us/j/81817469653

    • Meeting ID: 818 1746 9653
      Passcode: 551681

    • One tap mobile:
      +16699006833,,81817469653# US (San Jose)
      +13462487799,,81817469653# US (Houston)

    • Dial by your location:
      +1 669 900 6833 US (San Jose)
      +1 346 248 7799 US (Houston)
      +1 253 215 8782 US (Tacoma)
      +1 929 205 6099 US (New York)
      +1 301 715 8592 US (Washington DC)
      +1 312 626 6799 US (Chicago)

    • Find your local number: https://us02web.zoom.us/u/kpV1o65N

  • Follow along on Google Docs: https://nten.org/drupal/notes
  • Follow along on Twitter: #npdrupal

View notes of previous months' calls.

Categories: FLOSS Project Planets

Real Python: Build Your Python Project Documentation With MkDocs

Planet Python - Wed, 2022-06-15 10:00

In this tutorial, you’ll learn how to quickly build documentation for a Python package using MkDocs and mkdocstrings. These tools allow you to generate nice-looking and modern documentation from Markdown files and your code’s docstrings.

Maintaining auto-generated documentation means less effort because you’re linking information between your code and the documentation pages. However, good documentation is more than just the technical description pulled from your code! Your project will appeal more to users if you guide them through examples and connect the dots between the docstrings.

The Material for MkDocs theme makes your documentation look good without any extra effort and is used by popular projects such as Typer CLI and FastAPI.

In this tutorial, you’ll:

  • Work with MkDocs to produce static pages from Markdown
  • Pull in code documentation from docstrings using mkdocstrings
  • Follow best practices for project documentation
  • Use the Material for MkDocs theme to make your documentation look good
  • Host your documentation on GitHub Pages

If you use the auto-generation features of MkDocs together with mkdocstrings, then you can create good documentation with less effort. Start your documentation with docstrings in your code, then build it into a deployed and user-friendly online resource that documents your Python project.

Ready to go? Then click the link below to get the source code for the project:

Get Source Code: Click here to get access to the source code that you’ll use to build your documentation.

Demo

In this tutorial, you’ll build project documentation that’s partly auto-generated from docstrings in your code. The example code package is intentionally simplistic, so you can focus your attention on learning how to use MkDocs and the associated libraries.

After you set up your project documentation locally, you’ll learn how to host it on GitHub Pages, so it’ll be available for everyone to see:

You can use the example project documentation that you’ll build in this tutorial as a blueprint to create documentation for your own Python projects.

Project Overview

You’ll build project documentation for a toy package called calculator that contains only one module named calculations.py, which has a couple of example Python functions.

Note: The provided code doesn’t offer any new functionality and is only meant as a basis to learn how to use existing project code to build your documentation.

You’ll follow a guideline for project documentation called the Diátaxis documentation framework, which has widespread adoption in the Python community and is used by large projects such as Django and NumPy.

This system suggests splitting up your documentation into four different parts with different orientations:

  1. Tutorials: Learning-oriented
  2. How-To Guides: Problem-oriented
  3. Reference: Information-oriented
  4. Explanation: Understanding-oriented

Splitting your project documentation into these four different purposes with different orientations will help you create comprehensive documentation for your Python project.

From a technical perspective, you’ll build your documentation using three Python packages:

  1. MkDocs for building static pages from Markdown
  2. mkdocstrings for auto-generating documentation from docstrings in your code
  3. Material for MkDocs for styling your documentation

When you want to use MkDocs for auto-generating parts of your documentation from your docstrings, you’ll need to add the mkdocstrings package.

Note: Sphinx, another popular tool for Python project documentation, can auto-generate text from your docstrings without additional extensions. However, Sphinx primarily uses reStructuredText instead of Markdown and is overall less straightforward to work with than MkDocs.

You don’t absolutely need to add the Material for MkDocs theme for building your project documentation, but it’ll help to render the documentation in a user-friendly manner.

Read the full article at https://realpython.com/python-project-documentation-with-mkdocs/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Mike Driscoll: What's New in Python 3.11 (Video)

Planet Python - Wed, 2022-06-15 09:56

In this video, Mike Driscoll talks about what to expect in Python's newest release, which is coming in Fall 2022

  • Better Performance
  • Improved error messages
  • Exception groups
  • New type hints
  • The new `tomllib` module

See the Python website for full details

The post What's New in Python 3.11 (Video) appeared first on Mouse Vs Python.

Categories: FLOSS Project Planets

Podcast.__init__: Intelligent Dependency Resolution For Optimal Compatibility And Security With Project Thoth

Planet Python - Wed, 2022-06-15 09:43
Building any software project is going to require relying on dependencies that you and your team didn't write or maintain, and many of those will have dependencies of their own. This has led to a wide variety of potential and actual issues ranging from developer ergonomics to application security. In order to provide a higher degree of confidence in the optimal combinations of direct and transitive dependencies a team at Red Hat started Project Thoth. In this episode Fridolín Pokorný explains how the Thoth resolver uses multiple signals to find the best combination of dependency versions to ensure compatibility and avoid known security issues.Summary

Building any software project is going to require relying on dependencies that you and your team didn’t write or maintain, and many of those will have dependencies of their own. This has led to a wide variety of potential and actual issues ranging from developer ergonomics to application security. In order to provide a higher degree of confidence in the optimal combinations of direct and transitive dependencies a team at Red Hat started Project Thoth. In this episode Fridolín Pokorný explains how the Thoth resolver uses multiple signals to find the best combination of dependency versions to ensure compatibility and avoid known security issues.

Announcements
  • Hello and welcome to Podcast.__init__, the podcast about Python’s role in data and science.
  • When you’re ready to launch your next app or want to try a project you hear about on the show, you’ll need somewhere to deploy it, so take a look at our friends over at Linode. With their managed Kubernetes platform it’s easy to get started with the next generation of deployment and scaling, powered by the battle tested Linode platform, including simple pricing, node balancers, 40Gbit networking, dedicated CPU and GPU instances, and worldwide data centers. And now you can launch a managed MySQL, Postgres, or Mongo database cluster in minutes to keep your critical data safe with automated backups and failover. Go to pythonpodcast.com/linode and get a $100 credit to try out a Kubernetes cluster of your own. And don’t forget to thank them for their continued support of this show!
  • Need to automate your Python code in the cloud? Want to avoid the hassle of setting up and maintaining infrastructure? Shipyard is the premier orchestration platform built to help you quickly launch, monitor, and share python workflows in a matter of minutes with 0 changes to your code. Shipyard provides powerful features like webhooks, error-handling, monitoring, automatic containerization, syncing with Github, and more. Plus, it comes with over 70 open-source, low-code templates to help you quickly build solutions with the tools you already use. Go to dataengineeringpodcast.com/shipyard to get started automating with a free developer plan today!
  • Your host as usual is Tobias Macey and today I’m interviewing Fridolín Pokorný about Project Thoth, a resolver service that computes the optimal combination of versions for your dependencies
Interview
  • Introductions
  • How did you get introduced to Python?
  • Can you describe what Project Thoth is and the story behind it?
  • What are some examples of the types of problems that can be introduced by mismanaged dependency versions?
  • The Python ecosystem has seen a number of dependency management tools introduced recently. What are the capabilities that Thoth offers that make it stand out?
    • How does it compare to e.g. pip, Poetry, pip-tools, etc.?
    • How do those other tools approach resolution of dependencies?
  • Can you describe how Thoth is implemented?
    • How have the scope and design of the project evolved since it was started?
  • What are the sources of information that it relies on for generating the possible solution space?
    • What are the algorithms that it relies on for finding an optimal combination of packages?
  • Can you describe how Thoth fits into the workflow of a developer while selecting a set of dependencies and keeping them up to date over the life of a project?
  • What are the opportunities for expanding Thoth’s application to other language ecosystems?
  • What are the interfaces available for extending or integrating with Thoth?
  • What are the most interesting, innovative, or unexpected ways that you have seen Thoth used?
  • What are the most interesting, unexpected, or challenging lessons that you have learned while working on Thoth?
  • When is Thoth the wrong choice?
  • What do you have planned for the future of Thoth?
Keep In Touch Picks Links

The intro and outro music is from Requiem for a Fish The Freak Fandango Orchestra / CC BY-SA

Categories: FLOSS Project Planets

Python for Beginners: Doubly Linked List in Python

Planet Python - Wed, 2022-06-15 09:00

Linked lists are used in various applications in python programming. In this article, we will implement a doubly linked list in python. To understand doubly linked lists, you will need to have the knowledge of simple linked lists. Therefore, if you don’t know about singly linked lists, you can read about them in this article on linked lists in python.

Table of Contents
  1. What Is a Doubly Linked List?
  2. How to Create a Doubly Linked List in Python?
  3. Check if a Doubly Linked List Is Empty in Python
  4. Find the Length of a Doubly Linked List in Python
  5. Search an Element in a Doubly Linked List in Python
  6. Insert an Element in a Doubly Linked List in Python
    1. Insert at the Beginning of a Doubly Linked List
    2. Insert at the End of the Doubly Linked List
    3. Insert After an Element of the Doubly Linked List 
    4. Insert at a Given Position in the Doubly Linked List
  7. Print the Elements of a Doubly Linked List in Python
  8. Update an Element in a Doubly Linked List in Python
    1. Update Element at a Given Position
  9. Delete an Element From a Doubly Linked List in Python
    1. Delete From Beginning of the Doubly Linked List
    2. Delete the Last Element of the Doubly Linked List
    3. Delete a Given Element in the Doubly Linked List
    4. Delete From a Given Position in the Doubly Linked List
  10. Complete Implementation of Doubly Linked List in Python
  11. Conclusion
What Is a Doubly Linked List?

A doubly linked list is a linked list in which the nodes are linked to each other using two references.

Each node in a doubly linked list consists of three attributes namely data, next, and previous. You can visualize a node in a doubly linked list as follows.

Node of a Doubly Linked List in Python

Here,

  • The previous attribute is used to refer to the previous node in the linked list.
  • The data attribute is used to store the data in the node.
  • The next attribute is used to refer to the next node in the linked list.

In a doubly linked list, there can be any number of nodes. All the nodes in the linked list are connected to each other using the previous and next attributes. You can visualize a doubly linked list having three nodes as follows.

Doubly Linked List in Python

In the above image, we have created a doubly linked list containing three nodes.

  • The first node contains 5 in its data attribute and p1 and n1 as its previous and next attribute respectively.
  • The second node contains 10 in its data attribute and p2 and n2 as its previous and next attribute respectively.
  • The third node contains 15 in its data attribute and p3 and n3 as its previous and next attribute respectively.
  • The head node does not contain any data and is used to refer to the first node in the linked list.
  • The previous attribute of the first node does not refer to any other node. It points to None. Similarly, the next attribute of the last node does not refer to any other node. It also points to None.

As we have a general idea of what a doubly linked list looks like, let us try to implement a doubly linked list in Python.

How to Create a Doubly Linked List in Python?

To create a doubly linked list in python, we will first create a node for a doubly linked list as shown below.

class Node: def __init__(self, value): self.previous = None self.data = value self.next = None

Here, the Node class contains the attribute data to store the data in the linked list. The previous and next attributes are used to connect the nodes in the doubly linked list in python.

After creating a node, we will create a DoublyLinkedList class to implement a doubly linked list in python. The class will contain a head attribute that is initialized to None.

class DoublyLinkedList: def __init__(self): self.head = None

After creating the empty linked list, we can create a node and assign it to the head attribute. To add more nodes to the linked list, you can manually assign the next and the previous attributes of the nodes.

Instead of manually assigning the nodes to the linked list, we can write a method to insert an element in a doubly linked list in python. We can also perform different operations like updating and deleting the elements from the doubly linked list in python. Let us discuss each operation one by one.

Check if a Doubly Linked List Is Empty in Python

To check if a doubly linked list is empty in python, we just have to check if the head attribute of the linked list points to None. If yes, we will say that the linked list is empty. Otherwise not.

For this operation, we will implement an isEmpty() method. The isEmpty() method, when invoked on a doubly linked list, will check whether the head attribute of the linked list points to None. If yes, it returns True. Otherwise, it returns False. 

Following is the python implementation of the isEmpty() method to check if a doubly linked list is empty or not.

def isEmpty(self): if self.head is None: return True return False Find the Length of a Doubly Linked List in Python

To find the length of the doubly linked list, we will follow the following steps.

  • First, we will create a temporary variable temp and a counter variable count.
  • We will initialize the variable temp with the head of the doubly linked list. Also, we will initialize the variable count to 0.
  • Now we will traverse through the linked list using a while loop and the temp variable. 
  • While traversal, we will first check if the current Node (temp) is None. If yes, we will get out of the loop. Otherwise, we will first increment the count by 1. After that, we will assign the temp variable to the next node of the current node. 

After execution of the while loop, we will get the length of the doubly linked list in the variable count. 

Following is the implementation of the length() method for doubly linked lists in python. When invoked on a doubly linked list, it calculates the length of the linked list and returns the value.

def length(self): temp = self.head count = 0 while temp is not None: temp = temp.next count += 1 return count Search an Element in a Doubly Linked List in Python

To search an element in a doubly linked list in python, we will use the following algorithm.

  • First, we will define a variable temp and initialize it to the head attribute of the linked list.
  • We will also define a variable isFound and initialize it to False. This variable will be used to check if the given element is found in the doubly linked list or not.
  • After that, we will iterate through the nodes of the linked list using a while loop and the temp variable. 
  • While iterating the nodes of the doubly linked list, we will perform the following operations.
    • We will check if the current node is None. If yes, it means that we have reached the end of the linked list. Hence, we will move out of the while loop.
    • If the current node is not None, we will check if the data in the current node is equal to the value we are searching for. 
    • If the element in the current node is equal to the element we are searching for, we will assign the value True to the isFound variable. After that, we will move out of the while loop using the break statement. Otherwise, we will move to the next node in the linked list.

After execution of the while loop, if the isFound variable has the value True, the element is said to be found in the linked list. Otherwise not.

Following is the implementation of the search() method. The search() method, when invoked on a doubly linked list, takes an element as its input argument. After execution, it returns True if the element is found in the linked list. Otherwise, it returns False.

def search(self, value): temp = self.head isFound = False while temp is not None: if temp.data == value: isFound = True break temp = temp.next return isFound Insert an Element in a Doubly Linked List in Python

While inserting an element in a doubly linked list in python, there can be four situations.

  1. We need to insert an element at the beginning of the linked list.
  2. We need to insert an element at a given position in the linked list.
  3. We need to insert an element after an element in the linked list.
  4. We need to insert an element at the end of the linked list.

Let us discuss each of the cases one by one.

Insert at the Beginning of a Doubly Linked List

To insert an element at the beginning of a doubly linked list in python, we will first check if the linked list is empty. You can check if the linked list is empty using the isEmpty() method discussed in the previous section.

If the doubly linked list is empty, we will simply create a new node with the given data and assign it to the head attribute of the linked list.

If the linked list is not empty, we will follow the following steps.

  • First, we will create a new node with the given data that has to be inserted into the linked list.
  • After that, we will assign the node referred by the head attribute of the linked list to the next attribute of the new node.
  • Then, we will assign the new node to the previous attribute of the node referred by the head attribute of the linked list. 
  • Finally, we will assign the new node to the head attribute of the linked list.

After executing the above steps, the new element will be added to the doubly linked list at its beginning.

Following is the implementation of the insertAtBeginning() method in python. The insertAtBeginning() method, when invoked on a doubly-linked list, takes an element as its input argument and inserts it into the linked list at the beginning of the linked list.

def insertAtBeginning(self, value): new_node = Node(value) if self.isEmpty(): self.head = new_node else: new_node.next = self.head self.head.previous = new_node self.head = new_node Insert at the End of the Doubly Linked List

To insert an element at the end of a doubly linked list in python, we will use the following algorithm.

  • First, we will create a new node with the given element.
  • After that, we will check if the doubly linked list is empty. If yes, we will use the insertAtBeginning() method to add the new element to the list.
  • Otherwise, we will define a variable temp and assign the head attribute to it. After that, we will move to the last node of the doubly linked list using a while loop.
  • In the while loop, we will check if the next attribute of the current node points to None. If yes, we have reached the last node of the list. Hence, we will move out of the loop. Otherwise, we will move to the next node.
  • After reaching the last node(temp), we will assign the new node to the next attribute of the last node.  Then, we will assign temp to the previous attribute of the new node.

After execution of the above steps, the new element will be added to the end of the doubly linked list.

Following is the implementation of the insertAtEnd() method. The insertAtEnd() method, when invoked on a linked list, takes an element as its input argument and adds it to the end of the linked list.

def insertAtEnd(self, value): new_node = Node(value) if self.isEmpty(): self.insertAtBeginning(value) else: temp = self.head while temp.next is not None: temp = temp.next temp.next = new_node new_node.previous = temp Insert After an Element of the Doubly Linked List 

To insert a new element after another element in a doubly linked list in python, we will use the following steps.

First, we will define a variable temp and initialize it to the head attribute of the linked list. After that, we will iterate through the nodes of the linked list using a while loop and the temp variable.  While iterating the nodes of the doubly linked list, we will perform the following operations.

  • We will check if the current node is None. If yes, it means that we have reached the end of the linked list. Hence, we will move out of the while loop.
  • If the current node is not None, we will check if the data in the current node is equal to the element after which we have to insert the new element.
  • If the element in the current node is equal to the element after which we have to insert the new element, we will move out of the while loop using the break statement. Otherwise, we will move to the next node in the linked list.

After execution of the while loop, there can be two situations.

  • If the temp variable contains the value None, it means that the element after which we have to insert the new value doesn’t exist in the linked list. In this case, we will print that the element cannot be inserted in the doubly linked list.
  • If the temp variable is not None, we will insert the element in the linked list. For this, we will follow the following steps.
    • First, we will create a new node with the element that has to be inserted.
    • We will assign the next node of the current node to the next attribute of the new node. After that, we will assign the current node to the previous attribute of the new node.
    • Then, we will assign the new node to the previous attribute of the next node of the current node.
    • Finally, we will assign the new node to the next attribute of the current node. 

After executing the above steps, the new element will be inserted into the doubly linked list after the given value.

Following is the implementation of insertAfterElement() method. The insertAfterElement() method takes two values as its input argument. The first argument is the new value that is to be inserted into the linked list.  The second argument is the element after which the new value has to be inserted.

After execution, the insertAfterElement() method inserts the new element to the doubly linked list if the element after which the new value has to be inserted is present in the doubly linked list. Otherwise, it prints that the new element cannot be inserted into the linked list.

def insertAfterElement(self, value, element): temp = self.head while temp is not None: if temp.data == element: break temp = temp.next if temp is None: print("{} is not present in the linked list. {} cannot be inserted into the list.".format(element, value)) else: new_node = Node(value) new_node.next = temp.next new_node.previous = temp temp.next.previous = new_node temp.next = new_node Insert at a Given Position in the Doubly Linked List

To insert an element at a given position N in a doubly linked list in python, we will follow the following steps.

If N==1, it means that the element has to be inserted at the first position. We will insert the element in the doubly linked list using the insertAtBeginning() method. Otherwise, we will follow the following steps.

  • First, we will define a variable temp and initialize it to the head attribute of the linked list. Then, we will initialize a variable count to 1.
  • After that, we will iterate through the nodes of the linked list using a while loop and the temp variable. 
  • While iterating the nodes of the doubly linked list, we will perform the following operations.
    • We will check if the current node is None. If yes, it means that we have reached the end of the linked list. Hence, we will move out of the while loop.
    • If the current node is not None, we will check if the variable count has a value equal to N-1. If yes, we will move out of the while loop using the break statement. Otherwise, we will move to the next node in the linked list.

After execution of the while loop, there can be two situations.

  • If the temp variable contains the value None, it means that there are less than N-1 elements in the linked list. In this case, we cannot insert the new node at the Nth position. Hence, we will print that the element cannot be inserted in the doubly linked list.
  • If the temp variable is not None, we will insert the element in the linked list. For this, we will have two choices. 
  • First, we will check if the next node of the current node(temp) is None, if yes, we need to insert the new element at the end of the linked list. Therefore, we will use the insertAtEnd() method for the same.
  • If the next node of the current node is not None, we will use the following steps to insert the new element at the given position.
    • First, we will create a new node with the element that has to be inserted.
    • We will assign the next node of the current node to the next attribute of the new node.
    • After that, we will assign the current node to the previous attribute of the new node.
    • Then, we will assign the new node to the previous attribute of the next node of the current node.
    • Finally, we will assign the new node to the next attribute of the current node. 

After executing the above steps, the new element will be inserted into the doubly linked list at the given position.

Following is the implementation of insertAtPosition() method. The insertAtPosition() method takes two values as its input argument. The first argument is the new value that is to be inserted into the linked list.  The second argument is the position at which the new value has to be inserted.

After execution, the insertAtPosition() method inserts the new element at the desired position in the doubly linked list. Otherwise, it prints that the new element cannot be inserted into the linked list.

def insertAtPosition(self, value, position): temp = self.head count = 0 while temp is not None: if count == position - 1: break count += 1 temp = temp.next if position == 1: self.insertAtBeginning(value) elif temp is None: print("There are less than {}-1 elements in the linked list. Cannot insert at {} position.".format(position, position)) elif temp.next is None: self.insertAtEnd(value) else: new_node = Node(value) new_node.next = temp.next new_node.previous = temp temp.next.previous = new_node temp.next = new_node Print the Elements of a Doubly Linked List in Python

To print the elements of a doubly linked list in python, we will first define a variable temp and assign the head of the linked list to it. After that, we will use a while loop to iterate through the nodes of the linked list.

While iteration, we will first check if the current node in the doubly linked list is None. If yes, we will move out of the while loop. Otherwise, we will print the data attribute of the current node. Finally, we will move to the next node in the linked list using the next attribute of the nodes.

Following is the implementation of printLinkedList() method. The printLinkedList() method, when invoked on a doubly linked list, prints all the elements of the linked list.

def printLinkedList(self): temp = self.head while temp is not None: print(temp.data) temp = temp.next Update an Element in a Doubly Linked List in Python

To update an element in a doubly linked list in python, we will first define a variable temp and assign the head of the linked list to it. We will also define a variable isUpdated and initialize it to False. After that, we will use a while loop to iterate through the nodes of the linked list.

While iteration, we will first check if the current node in the doubly linked list is None. If yes, we will move out of the while loop. Otherwise, we will check if the data attribute in the current node is equal to the value that needs to be replaced with a new value. If yes, we will update the data attribute in the current Node and will update the value in isUpdated to True. Finally, we will move out of the while loop using break statement.

After executing the while loop, we will check if the isUpdated is False. If yes, we will print that the value is not updated. Otherwise, we will print that the value is updated.

Following is the implementation of updateElement() method. The updateElement() method takes two values as its input argument. The first argument is the old value that is to be updated.  The second argument is the new value.

After execution, the updateElement() method updates the given element to the new value.

def updateElement(self, old_value, new_value): temp = self.head isUpdated = False while temp is not None: if temp.data == old_value: temp.data = new_value isUpdated = True temp = temp.next if isUpdated: print("Value Updated in the linked list") else: print("Value not Updated in the linked list") Update Element at a Given Position

To update an element at a given position N in a doubly linked list in python, we will use the following algorithm.

First, we will define a variable temp and initialize it to the head attribute of the linked list. Then, we will initialize a variable count to 1. After that, we will iterate through the nodes of the linked list using a while loop and the temp variable. 

While iterating the nodes of the doubly linked list, we will perform the following operations.

  • We will check if the current node is None. If yes, it means that we have reached the end of the linked list. Hence, we will move out of the while loop.
  • If the current node is not None, we will check if the variable count has the value equal to N. If yes, we will move out of the while loop using the break statement. Otherwise, we will move to the next node in the linked list.

After execution of the while loop, there can be two situations.

  • If the temp variable contains the value None, it means that there are less than N elements in the linked list. In this case, we cannot update an element at the Nth position in the linked list. Hence, we will print that the element cannot be updated.
  • If the temp variable is not None, we will update the Nth element in the linked list by updating the data attribute of the current node.

Following is the implementation of updateAtPosition() method. The updateAtPosition() method takes two values as its input argument. The first argument is the new value that is to be updated in the linked list.  The second argument is the position at which the new value has to be assigned.

After execution, the updateAtPosition() method updates the element at the desired position in the doubly linked list.

def updateAtPosition(self, value, position): temp = self.head count = 0 while temp is not None: if count == position: break count += 1 temp = temp.next if temp is None: print("Less than {} elements in the linked list. Cannot update.".format(position)) else: temp.data = value print("Value updated at position {}".format(position)) Delete an Element From a Doubly Linked List in Python

While deleting an element from a doubly linked list in python, there can be four cases.

  1. We need to delete an element from the start of the linked list.
  2. We need to delete an element from the end of the linked list.
  3. We need to delete a specific element.
  4. We need to delete an element from the given position in the linked list.

Let us discuss each of the cases one by one.

Delete From Beginning of the Doubly Linked List

To delete an element from the beginning of a doubly linked list, we will first check if the doubly linked list is empty. If yes, we will say that we cannot delete any element from the linked list.

Otherwise, we will check if there is only one element in the linked list i.e. the next attribute of the head node points to None or not. If the next attribute of the head node is None, we will assign None to the head.

If there are more than one element in the linked list, we will move the head of the linked list to the next node of the current head. After that, we will assign None to the previous attribute of the new head node. 

After executing the above steps, the first node of the linked list will be deleted.

Following is the implementation of the deleteFromBeginning() method. The deleteFromBeginning() method, when invoked on a doubly linked list, deletes the first node of the linked list.

def deleteFromBeginning(self): if self.head is None: print("Linked List is empty. Cannot delete elements.") elif self.head.next is None: self.head = None else: self.head = self.head.next self.head.previous = None Delete the Last Element of the Doubly Linked List

To delete the last element of a doubly linked list in python, we will use the following steps.

  • First, we will check if the doubly linked list is empty. If yes, we will say that we cannot delete any element from the linked list.
  • Otherwise, we will check if the linked list has only one element i.e. the next attribute of the head node points to None or not. If the next attribute of the head node is None, we will assign None to the head.
  • If there are more than one elements in the linked list, we will create a variable temp and assign the head to the variable. After that, we will traverse the doubly linked list till we reach the last node of the list i.e. the next attribute of the current node becomes None.
  • After reaching the last node, we will assign None to the next attribute of the previous node of the current node. Subsequently, we will assign None to the previous attribute of the current node. 

By executing the above steps, the last element of the doubly linked list will be deleted from the linked list.

Following is the implementation of deleteFromLast() method. The deleteFromLast() method, when invoked on a doubly linked list, deletes the last node of the linked list. 

def deleteFromLast(self): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif self.head.next is None: self.head = None else: temp = self.head while temp.next is not None: temp = temp.next temp.previous.next = None temp.previous = None Delete a Given Element in the Doubly Linked List

To delete a given element from a doubly linked list in python, we will use the following steps.

  • First, we will check if the doubly linked list is empty. If yes, we will say that we cannot delete any element from the linked list.
  • Otherwise, we will check if the linked list has only one element i.e. the next attribute of the head node points to None or not. 
  • If the next attribute of the head node is None, we will check if the element in the first node is the element to be deleted. If yes, we will assign None to the head. 
  • If there are more than one element in the linked list, we will create a variable temp and assign the head to the variable. 
  • After that, we will traverse the doubly linked list till last using the temp variable and a while loop. 
  • While traversing the linked list, we will first check if the current node is None i.e. we have reached the end of the linked list. If yes, we will move out of the while loop. 
  • If the current node is not None, we will check if the current node contains the element that needs to be deleted. If yes, we will move out of the while loop using a break statement. 

After executing the while loop, there can be two situations.

  • If the temp variable contains the value None, it means that we have reached the end of the doubly linked list. In this case, we cannot delete an element from the linked list. Hence, we will print that the element cannot be deleted from the doubly linked list.
  • If the temp variable is not none, we will delete the element from the linked list. For this, we will have two choices. 
    • First, we will check if the current node is the last node of the linked list i.e. the next node of the current node(temp) is None, if yes, we basically need to delete the last node of the linked list. Therefore, we will use the deleteFromLast() method for the same.
    • If the next node of the current node is not None, we will use the following steps to delete the given element.
    • We will assign the next node of the temp node to the next attribute of the previous node of temp.
    • Then, we will assign the previous node of the temp node to the previous attribute of the next node of temp.
    • Finally, we will assign None to the previous and next attributes of the temp node.

By executing the above steps, any given element, if present in the linked list, will be deleted from the linked list.

Following is the implementation of delete() method. The delete() method, when invoked on a doubly linked list, takes an element as an input argument and deletes its first occurrence from the linked list. 

def delete(self, value): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif self.head.next is None: if self.head.data == value: self.head = None else: temp = self.head while temp is not None: if temp.data == value: break temp = temp.next if temp is None: print("Element not present in linked list. Cannot delete element.") elif temp.next is None: self.deleteFromLast() else: temp.next = temp.previous.next temp.next.previous = temp.previous temp.next = None temp.previous = None Delete From a Given Position in the Doubly Linked List

To delete an element from a given position N in a doubly linked list in python, we will use the following algorithm.

First, we will check if the linked list is empty. If yes, we will say that we cannot delete any element.

After that, we will check If N==1, which means that we need to delete the element from the first position. If yes, we will delete the first element using the deleteFromBeginning() method.

Otherwise, we will follow the following steps.

  • First, we will define a variable temp and initialize it to the head attribute of the linked list. Then, we will initialize a variable count to 1.
  • After that, we will iterate through the nodes of the linked list using a while loop and the temp variable. 
  • While iterating the nodes of the doubly linked list, we will perform the following operations.
    • We will check if the current node is None. If yes, it means that we have reached the end of the linked list. Hence, we will move out of the while loop.
    • If the current node is not None, we will check if the variable count has the value equal to N. If yes, we will move out of the while loop using the break statement. Otherwise, we will move to the next node in the linked list.

After execution of the while loop, there can be two situations.

  • If the temp variable contains the value None, it means that there are less than N elements in the linked list. In this case, we cannot delete an element from Nth position in the linked list. Hence, we will print that the element cannot be deleted from the doubly linked list.
  • If the temp variable is not none, we will delete the Nth element from the linked list. For this, we will have two choices. 
    • First, we will check if the next node of the current node(temp) is None. If yes, the Nth element is the last element of the linked list. Therefore, we will use the deleteFromEnd() method to delete the element.
    • If the Nth element isn’t the last element of the doubly linked list, we will perform the following operations to delete the Nth element.
    • We will assign the next node of the temp (the current node) to the next attribute of the previous node of temp.
    • Then, we will assign the previous node of the temp node to the previous attribute of the next node of temp.
    • Finally, we will assign None to the previous and next attributes of the temp node.

By executing the above steps, any given element, if present in the linked list, will be deleted from the linked list.

Following is the implementation of deleteFromPosition() method. The deleteFromPosition() method, when invoked on a doubly linked list, takes the position from which the element has to be deleted as its input argument. After execution, it deletes the element from the specified position in the doubly linked list.

def deleteFromPosition(self, position): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif position == 1: self.deleteFromBeginning() else: temp = self.head count = 1 while temp is not None: if count == position: break temp = temp.next if temp is None: print("There are less than {} elements in linked list. Cannot delete element.".format(position)) elif temp.next is None: self.deleteFromLast() temp.previous.next = temp.next temp.next.previous = temp.previous temp.next = None temp.previous = None Complete Implementation of Doubly Linked List in Python

Now that we have discussed all the methods to implement a doubly linked list in python, let us execute the program to observe the implementation.

class Node: def __init__(self, value): self.previous = None self.data = value self.next = None class DoublyLinkedList: def __init__(self): self.head = None def isEmpty(self): if self.head is None: return True return False def length(self): temp = self.head count = 0 while temp is not None: temp = temp.next count += 1 return count def search(self, value): temp = self.head isFound = False while temp is not None: if temp.data == value: isFound = True break temp = temp.next return isFound def insertAtBeginning(self, value): new_node = Node(value) if self.isEmpty(): self.head = new_node else: new_node.next = self.head self.head.previous = new_node self.head = new_node def insertAtEnd(self, value): new_node = Node(value) if self.isEmpty(): self.insertAtBeginning(value) else: temp = self.head while temp.next is not None: temp = temp.next temp.next = new_node new_node.previous = temp def insertAfterElement(self, value, element): temp = self.head while temp is not None: if temp.data == element: break temp = temp.next if temp is None: print("{} is not present in the linked list. {} cannot be inserted into the list.".format(element, value)) else: new_node = Node(value) new_node.next = temp.next new_node.previous = temp temp.next.previous = new_node temp.next = new_node def insertAtPosition(self, value, position): temp = self.head count = 0 while temp is not None: if count == position - 1: break count += 1 temp = temp.next if position == 1: self.insertAtBeginning(value) elif temp is None: print("There are less than {}-1 elements in the linked list. Cannot insert at {} position.".format(position, position)) elif temp.next is None: self.insertAtEnd(value) else: new_node = Node(value) new_node.next = temp.next new_node.previous = temp temp.next.previous = new_node temp.next = new_node def printLinkedList(self): temp = self.head while temp is not None: print(temp.data, sep=",") temp = temp.next def updateElement(self, old_value, new_value): temp = self.head isUpdated = False while temp is not None: if temp.data == old_value: temp.data = new_value isUpdated = True temp = temp.next if isUpdated: print("Value Updated in the linked list") else: print("Value not Updated in the linked list") def updateAtPosition(self, value, position): temp = self.head count = 0 while temp is not None: if count == position: break count += 1 temp = temp.next if temp is None: print("Less than {} elements in the linked list. Cannot update.".format(position)) else: temp.data = value print("Value updated at position {}".format(position)) def deleteFromBeginning(self): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif self.head.next is None: self.head = None else: self.head = self.head.next self.head.previous = None def deleteFromLast(self): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif self.head.next is None: self.head = None else: temp = self.head while temp.next is not None: temp = temp.next temp.previous.next = None temp.previous = None def delete(self, value): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif self.head.next is None: if self.head.data == value: self.head = None else: temp = self.head while temp is not None: if temp.data == value: break temp = temp.next if temp is None: print("Element not present in linked list. Cannot delete element.") elif temp.next is None: self.deleteFromLast() else: temp.next = temp.previous.next temp.next.previous = temp.previous temp.next = None temp.previous = None def deleteFromPosition(self, position): if self.isEmpty(): print("Linked List is empty. Cannot delete elements.") elif position == 1: self.deleteFromBeginning() else: temp = self.head count = 1 while temp is not None: if count == position: break temp = temp.next if temp is None: print("There are less than {} elements in linked list. Cannot delete element.".format(position)) elif temp.next is None: self.deleteFromLast() temp.previous.next = temp.next temp.next.previous = temp.previous temp.next = None temp.previous = None x = DoublyLinkedList() print(x.isEmpty()) x.insertAtBeginning(5) x.printLinkedList() x.insertAtEnd(10) x.printLinkedList() x.deleteFromLast() x.printLinkedList() x.insertAtEnd(25) x.printLinkedList() x.deleteFromLast() x.deleteFromBeginning() x.insertAtEnd(100) x.printLinkedList()

Output:

True 5 5 10 5 5 25 100 Conclusion

In this article, we have discussed the implementation of doubly linked list in python. I hope this article helps you learn all the concepts of doubly linked lists in python. If you find any bugs or improvements in the implementation, do let us know in the comments.

To learn more about python programming, you can read this article on how to find the index of max value in a list in python. You might also like this article on how to sort list of objects in python.

Stay tuned for more informative articles.

Happy Learning!

The post Doubly Linked List in Python appeared first on PythonForBeginners.com.

Categories: FLOSS Project Planets

Andy Wingo: defragmentation

GNU Planet! - Wed, 2022-06-15 08:47

Good morning, hackers! Been a while. It used to be that I had long blocks of uninterrupted time to think and work on projects. Now I have two kids; the longest such time-blocks are on trains (too infrequent, but it happens) and in a less effective but more frequent fashion, after the kids are sleeping. As I start writing this, I'm in an airport waiting for a delayed flight -- my first since the pandemic -- so we can consider this to be the former case.

It is perhaps out of mechanical sympathy that I have been using my reclaimed time to noodle on a garbage collector. Managing space and managing time have similar concerns: how to do much with little, efficiently packing different-sized allocations into a finite resource.

I have been itching to write a GC for years, but the proximate event that pushed me over the edge was reading about the Immix collection algorithm a few months ago.

on fundamentals

Immix is a "mark-region" collection algorithm. I say "algorithm" rather than "collector" because it's more like a strategy or something that you have to put into practice by making a concrete collector, the other fundamental algorithms being copying/evacuation, mark-sweep, and mark-compact.

To build a collector, you might combine a number of spaces that use different strategies. A common choice would be to have a semi-space copying young generation, a mark-sweep old space, and maybe a treadmill large object space (a kind of copying collector, logically; more on that later). Then you have heuristics that determine what object goes where, when.

On the engineering side, there's quite a number of choices to make there too: probably you make some parts of your collector to be parallel, maybe the collector and the mutator (the user program) can run concurrently, and so on. Things get complicated, but the fundamental algorithms are relatively simple, and present interesting fundamental tradeoffs.


figure 1 from the immix paper

For example, mark-compact is most parsimonious regarding space usage -- for a given program, a garbage collector using a mark-compact algorithm will require less memory than one that uses mark-sweep. However, mark-compact algorithms all require at least two passes over the heap: one to identify live objects (mark), and at least one to relocate them (compact). This makes them less efficient in terms of overall program throughput and can also increase latency (GC pause times).

Copying or evacuating spaces can be more CPU-efficient than mark-compact spaces, as reclaiming memory avoids traversing the heap twice; a copying space copies objects as it traverses the live object graph instead of after the traversal (mark phase) is complete. However, a copying space's minimum heap size is quite high, and it only reaches competitive efficiencies at large heap sizes. For example, if your program needs 100 MB of space for its live data, a semi-space copying collector will need at least 200 MB of space in the heap (a 2x multiplier, we say), and will only run efficiently at something more like 4-5x. It's a reasonable tradeoff to make for small spaces such as nurseries, but as a mature space, it's so memory-hungry that users will be unhappy if you make it responsible for a large portion of your memory.

Finally, mark-sweep is quite efficient in terms of program throughput, because like copying it traverses the heap in just one pass, and because it leaves objects in place instead of moving them. But! Unlike the other two fundamental algorithms, mark-sweep leaves the heap in a fragmented state: instead of having all live objects packed into a contiguous block, memory is interspersed with live objects and free space. So the collector can run quickly but the allocator stops and stutters as it accesses disparate regions of memory.

allocators

Collectors are paired with allocators. For mark-compact and copying/evacuation, the allocator consists of a pointer to free space and a limit. Objects are allocated by bumping the allocation pointer, a fast operation that also preserves locality between contemporaneous allocations, improving overall program throughput. But for mark-sweep, we run into a problem: say you go to allocate a 1 kilobyte byte array, do you actually have space for that?

Generally speaking, mark-sweep allocators solve this problem via freelist allocation: the allocator has an array of lists of free objects, one for each "size class" (say 2 words, 3 words, and so on up to 16 words, then more sparsely up to the largest allocatable size maybe), and services allocations from their appropriate size class's freelist. This prevents the 1 kB free space that we need from being "used up" by a 16-byte allocation that could just have well gone elsewhere. However, freelists prevent objects allocated around the same time from being deterministically placed in nearby memory locations. This increases variance and decreases overall throughput for both the allocation operations but also for pointer-chasing in the course of the program's execution.

Also, in a mark-sweep collector, we can still reach a situation where there is enough space on the heap for an allocation, but that free space broken up into too many pieces: the heap is fragmented. For this reason, many systems that perform mark-sweep collection can choose to compact, if heuristics show it might be profitable. Because the usual strategy is mark-sweep, though, they still use freelist allocation.

on immix and mark-region

Mark-region collectors are like mark-sweep collectors, except that they do bump-pointer allocation into the holes between survivor objects.

Sounds simple, right? To my mind, though the fundamental challenge in implementing a mark-region collector is how to handle fragmentation. Let's take a look at how Immix solves this problem.


part of figure 2 from the immix paper

Firstly, Immix partitions the heap into blocks, which might be 32 kB in size or so. No object can span a block. Block size should be chosen to be a nice power-of-two multiple of the system page size, not so small that common object allocations wouldn't fit. Allocating "large" objects -- greater than 8 kB, for Immix -- go to a separate space that is managed in a different way.

Within a block, Immix divides space into lines -- maybe 128 bytes long. Objects can span lines. Any line that does not contain (a part of) an object that survived the previous collection is part of a hole. A hole is a contiguous span of free lines in a block.

On the allocation side, Immix does bump-pointer allocation into holes. If a mutator doesn't have a hole currently, it scans the current block (obtaining one if needed) for the next hole, via a side-table of per-line mark bits: one bit per line. Lines without the mark are in holes. Scanning for holes is fairly cheap, because the line size is not too small. Note, there are also per-object mark bits as well; just because you've marked a line doesn't mean that you've traced all objects on that line.

Allocating into a hole has good expected performance as well, as it's bump-pointer, and the minimum size isn't tiny. In the worst case of a hole consisting of a single line, you have 128 bytes to work with. This size is large enough for the majority of objects, given that most objects are small.

mitigating fragmentation

Immix still has some challenges regarding fragmentation. There is some loss in which a single (piece of an) object can keep a line marked, wasting any free space on that line. Also, when an object can't fit into a hole, any space left in that hole is lost, at least until the next collection. This loss could also occur for the next hole, and the next and the next and so on until Immix finds a hole that's big enough. In a mark-sweep collector with lazy sweeping, these free extents could instead be placed on freelists and used when needed, but in Immix there is no such facility (by design).

One mitigation for fragmentation risks is "overflow allocation": when allocating an object larger than a line (a medium object), and Immix can't find a hole before the end of the block, Immix allocates into a completely free block. So actually mutator threads allocate into two blocks at a time: one for small objects and medium objects if possible, and the other for medium objects when necessary.

Another mitigation is that large objects are allocated into their own space, so an Immix space will never be used for blocks larger than, say, 8kB.

The other mitigation is that Immix can choose to evacuate instead of mark. How does this work? Is it worth it?

stw

This question about the practical tradeoffs involving evacuation is the one I wanted to pose when I started this article; I have gotten to the point of implementing this part of Immix and I have some doubts. But, this article is long enough, and my plane is about to land, so let's revisit this on my return flight. Until then, see you later, allocators!

Categories: FLOSS Project Planets

Qt for MCUs 2.2 LTS released

Planet KDE - Wed, 2022-06-15 07:55

We're excited to announce that Qt for MCUs 2.2 LTS is now available! It is the first release of Qt for MCUs that will be long-term supported; bug fixes for it will be released for a period of 18 months. This release adds improvements to text rendering,  new APIs to handle errors in GUI code, new tools to generate C++ from QML independently from CMake, and a new performance and footprint guide to help you get the best results with Qt for MCUs.

Categories: FLOSS Project Planets

Kushal Das: Story of a space

Planet Python - Wed, 2022-06-15 06:03

In my case the story continued for around 2 hours. Yesterday I was trying to implement something from a given SPEC, and tried to match my output (from Rust) with the output from the Python code written by Dr. Fett.

The problem

I had to get the JSON encoding from an ordered Array (in Python it is a simple list), say ["kushal", "das"], and get the base64urlencode(sha256sum(JSON_ENCODED_STRING)) in Rust using serde_json.

Easy, isn’t?

Let us see what the encoded string looks like:

"[\"6Ij7tM-a5iVPGboS5tmvVA\",\"John\"]"

And the final checksum is otcxXDbO4_xzK0DXL_DO2INIFXIJdO1PqVM-n_gt-3g.

But, the reference implementation in Python has the checksum as fUMdn88aaoyKTHrvZd6AuLmPraGhPJ0zF5r_JhxCVZs.

It took me a few hours to notice the space after comma in the JSON encoded string from Python:

"[\"6Ij7tM-a5iVPGboS5tmvVA\", \"John\"]"

This is due to the following line from JSON spec

Insignificant whitespace is allowed before or after any of the six structural characters.

Yes, I know I should have known better, and read the original spec properly. But, in this case I learned it in the hard way.

Categories: FLOSS Project Planets

Got something to say about KDE? Say it at Akademy 2022

Planet KDE - Wed, 2022-06-15 04:29



Akademy 2022 will be a hybrid event, in Barcelona and online that will be held from Saturday 1st to Friday 7th October. The Call for Participation is still open! Submit your talk ideas and abstracts as the deadline has been extended until the 19th June.

Why talk at #Akademy2022?

Akademy attracts people from all over the world, not only from the global KDE Community, but also from companies, Free Software developers and pundits from other projects, public institutions, and more. We all meet together to discuss and plan the future of the Community and its technology. You will meet people that are receptive to your ideas and will help you with their skills and experience.

How do you get started?

You do not have to worry about details or a presentation right now. Just think of an idea and submit some basic details about your talk. You can edit your abstract after the initial submission.

All topics relevant to the KDE Community are welcome. Here are a few ideas to get you started on your talk:

  • What KDE projects have you contributed to? Tell us about your work, and get feedback and support!
  • Where do you think should KDE go next?
  • How has KDE software impacted you/people you know in real life?
  • Are there particular problems with KDE you think you can give us some insight on? How would you solve it?
  • How can we improve KDE for people with disabilities?
  • How can we increase the participation of women in our community?
  • How can we accelerate full support on Wayland?
  • What is next for Plasma mobile?
  • What is the current state of the project you are working on?
  • Have you used any KDE applications in ingenious ways or created something great with them? Tell us about your work!

These are just some ideas to get the ball rolling. However, you can submit a proposal on any topic as long as you can make it relevant to KDE. For more ideas for talks, check out the videos from previous years: 2021, 2020, 2019, 2018, 2017, 2016, and 2015.

For more details and information visit our Call for Participation page.

Dot Categories: Community and Events
Categories: FLOSS Project Planets

Web Wash: Add Preview Page to Forms using Webform in Drupal

Planet Drupal - Wed, 2022-06-15 04:00

When you create a form using the Webform module, you may need a “preview” step. This is a page or step which allows the user who’s submitting the form to preview what’s being submitted.

If they see a mistake, they can go back to the form and fix the error. Once the form has been filled out and previewed then, it can be submitted.

Webform allows you to quickly create a preview step in any form, and in this tutorial, you’ll learn how to implement it.

Categories: FLOSS Project Planets

PyCoder’s Weekly: Issue #529 (June 14, 2022)

Planet Python - Tue, 2022-06-14 15:30

#529 – JUNE 14, 2022
View in Browser »

A First Look at PyScript: Python in the Web Browser

In this tutorial, you’ll learn about PyScript, a new framework that allows for running Python in the web browser with few or no code modifications and excellent performance. You’ll leverage browser APIs and JavaScript libraries to build rich, highly interactive web applications with Python.
REAL PYTHON

Understand Django: Debugging Tips And Techniques

Your Django app is up. You’ve got users. Your users are hitting bugs. How do you debug to fix the problems? This article dives deep into to how to find and squish the bugs in your Django code.
MATT LAYMAN • Shared by Matt Layman

Connect, Integrate, & Automate Your Data—From Python, or Any Other Application or Tool

CData makes it easier to unlock the value of data — simplifying connectivity between applications and data sources. Our SQL-based connectors streamline data access making it easy to access real-time data from on-premise and cloud databases, SaaS, APIs, NoSQL and more. Visit cdata.com to learn more →
CDATA SOFTWARE sponsor

Secure Password Handling in Python

Lots of applications require some form of authentication, password handling, or the use of secure credentials. What are the best practices in Python for dealing with this?
MARTIN HEINZ

Python 3.10.5 Bug Release

CPYTHON DEV BLOG

Python Developers Survey 2021 Results

PYTHON SOFTWARE FOUNDATION

EuroSciPy August 29-September 2, Basel, Switzerland

EUROSCIPY.ORG • Shared by Darya Chyzhyk

Discussions Raising Exceptions or Returning Error Objects in Python

Luke Plant’s article Raising exceptions or returning error objects in Python has folks weighing in.
HACKER NEWS

MicroPython: Python for Microcontrollers

HACKER NEWS

Python Jobs Senior Full-Stack Web Developer (Anywhere)

MonetizeMore

Backend Software Developer (Draper, UT, USA)

Canopy

Gameful Learning Developer (Ann Arbor, MI, USA)

University of Michigan

Python Technical Architect (USA)

Blenderbox

Software Engineer (Los Angeles or Dallas) (Los Angeles, CA, USA)

Causeway Capital Management LLC

DevOps Engineer (Ann Arbor, MI, USA)

University of Michigan

Academic Innovation Developer (Ann Arbor, MI, USA)

University of Michigan

Software Development Lead (Ann Arbor, MI, USA)

University of Michigan

Senior Backend Engineer (Anywhere)

Doist

Senior Storytelling Framework Engineer - Python (France)

GoPro

Senior Software Engineer - Python Full Stack (USA)

Blenderbox

Principal Python Engineer (100% Remote) (San Francisco, CA, USA)

Nira

More Python Jobs >>>

Articles & Tutorials Build a Quiz Application With Python

In this step-by-step project, you’ll build a Python quiz application for the terminal. Your app will ask you multiple-choice questions that you can use to strengthen your own knowledge or challenge your friends to test theirs.
REAL PYTHON

Pandas Vectorization: Sometimes Slower with Bloated Memory

When you’re processing data with Pandas, vectorized operations can speed up your code. In some cases though, they can actually make it slower, or at least no faster and memory hungry. Learn when it is helpful and when it is harmful to use vectorization.
ITAMAR TURNER-TRAURING

Merge Faster with WorkerB for PRs Chrome Extension

The average pull request sits idle for two days! Add context to your PR & merge faster with WorkerB magic links. Get estimated review time, number of changes, & related issues in one click! Install it today →
LINEARB sponsor

Write and Test a Python Function: Interview Practice

In this interview practice session, you’ll tackle creating a function that will double every character within a string. This challenge is typical of what you might encounter in a Python job interview. You’ll explore how to add tests to your code.
REAL PYTHON course

Dates And Times And Types

Dates and times in code can be more complicated than they first appear. Consider how datetime and date interact and how incorrect use can result in a TypeError even though they’re considered correct by type annotations.
GLYPH LEFKOWITZ

Understanding Sampling With and Without Replacement

Sampling can be done with and without replacement: when an item is sampled it may or may not be returned to the population for the next sample. Learn the differences and how it effects your statistical code.
MICHAEL GALARNY • Shared by Michael Galarnyk

Shipping to Production

“How you ship your code to production in a way that is fast and reliable, is a question more engineers and engineering leaders should educate themselves on.” Read on for a comparison between two extremes.
GERGELY OROSZ

Caching Connection Objects in Python

Three different mechanisms are common for having a single instance of a DB connection in your Python code: module level imports, the lru_cache decorator, or through singletons. See examples of each.
REDOWAN DELOWAR

Reach 100,000+ Pythonistas Every Week

Sponsor PyCoder’s Weekly and reach 100,000+ passionate Python developers, data scientists & machine learning engineers each week. Learn more about sponsorship packages →
PYCODER'S WEEKLY sponsor

Random Python: Secrets and Random Values Made Easy

Needing a random value happens a lot when you’re coding. This article describes different ways of getting random information in Python and how to choose amongst them.
JOHN LOCKWOOD

Handling Concurrency Without Locks

Through the use of an example Django web application, this article illustrates a variety of concurrency issues and how to handle them without locks.
HAKI BENITA

Projects & Code python-syntax-errors: Version Specific No-Ops

GITHUB.COM/JWILK

libgravatar: Python 3 Interface for Gravatar APIs

GITHUB.COM/PABLUK

arsenal: Inventory & Launcher for Penetration Testing

GITHUB.COM/ORANGE-CYBERDEFENSE

pikepdf: Read and Write PDF, Powered by QPDF

GITHUB.COM/PIKEPDF

django-pgpubsub: Distributed Tasks with Postgres NOTIFY

GITHUB.COM/OPUS10 • Shared by Paul Gilmartin

Events Software Craftsmanship

June 15, 2022
MEETUP.COM

PiterPy Breakfast

June 15, 2022
TIMEPAD.RU

Weekly Real Python Office Hours Q&A (Virtual)

June 15, 2022
REALPYTHON.COM

PyData Bristol Meetup

June 16, 2022
MEETUP.COM

PyLadies Dublin

June 16, 2022
PYLADIES.COM

Karlsruhe Python User Group (KaPy)

June 17, 2022
BL0RG.NET

GeoPython 2022

June 20 to June 23, 2022
GEOPYTHON.NET

Happy Pythoning!
This was PyCoder’s Weekly Issue #529.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

Categories: FLOSS Project Planets

Higher Ed needs to step up to stay relevant as Open Source floods the IT world

Open Source Initiative - Tue, 2022-06-14 14:44

By Patrick Masson, General Manager, Apereo Foundation This post is in support of the Open Apereo...

The post Higher Ed needs to step up to stay relevant as Open Source floods the IT world first appeared on Voices of Open Source.

Categories: FLOSS Research

Kdenlive 22.04.2 Released

Planet KDE - Tue, 2022-06-14 11:02

The polishing and stability effort of this release cycle continues with the release Kdenlive 22.04.2 which comes with bug fixes to the AppImage and Mac packages, render widget, same track transitions, subtitles and project loading issues.

 

  • Fix icon color change in some situations (eg. Appimage). Commit. Fixes bug #450556
  • Fix incorrect lambda capture leading to crash. Commit.
  • Fix AppImage icons. Commit. See bug #451406
  • Online resources: only show warning about loading time once. Commit. See bug #454470
  • Clang format fixes. Commit.
  • Fix crash clicking ok in empty transcoding dialog. Commit.
  • Fix possible crash when load task is running on exit. Commit.
  • Fix file watcher broken, changed clips were not detected anymore. Commit.
  • Fix timeremap clip always using proxies on rendering. Commit. Fixes bug #454089
  • Ensure internal effects like subtitles stay on top so that they are not affected by color or transform effects. Commit.
  • Fix crash on undo center keyframe. Commit.
  • Fix crash changing clip monitor bg color when no clip is selected. Commit.
  • Fix crash on undo selected clip insert. Commit.
  • Fix nvenc codec. Commit. See bug #454469
  • Fix clip thumbs not discarded on property change. Commit.
  • On document loading, also check images for changes. Commit.
  • Fix tests and mix direction regression. Commit.
  • Fix major corruption on undo/redo clip cut, with tests. Commit.
  • Project loading: detect and fix corruption if audio or video clips on the same track use a different producer. Commit.
  • Fix crash dropping an effect on the clip monitor. Commit.
  • Speedup maker search. Commit.
  • Fix cannot put monitor in fullscreen with mirrored screens. Commit.
  • Fix mix on very short AV clips broken, with test. Commit.
  • Fix Slide mix not correctly updated when creating a new mix on the previous clip, add tests. Commit. See bug #453770
  • Fix mix mix not correctly reversed in some cases and on undo. Commit.
  • Fix slide composition going in wrong direction (mix is still todo). Commit. See bug #453770
  • Fix several small glitches in bin selection. Commit.
  • Fix clip height not aligned to its track. Commit.
  • Fix speech to text on Mac. Commit.
  • Fix crash/corruption in overwrite mode when moving grouped clips above or below existing tracks. Commit.
  • Fix missing audio with “WebM-VP9/Opus (libre)” preset. Commit. See bug #452950
  • [Render Widget] Allow more steps for quality slider. Commit.
  • [Render Presets] Fix wrongly reversed quality with custom presets. Commit.
  • [Render Presets] Add more speed preset steps for x254 and x256. Commit.
  • Fix mixers don’t display levels if a track was added/removed with collapsed mixer. Commit.
  • Fix possible crash in transcoding dialog if there are no clips to convert. Commit.
  • [RenderWidget] Add scrollbar to improve experience on small screens. Commit.

The post Kdenlive 22.04.2 Released appeared first on Kdenlive.

Categories: FLOSS Project Planets

Mike Driscoll: Data Science Packages in Python (Video)

Planet Python - Tue, 2022-06-14 10:46

In this tutorial, I will talk about some of the many different data science packages you can use in Python.

Packages mentioned: pandas, matplotlib, numpy, scipy, mahotas, OpenCV, Keras, PyTorch, Tensorflow, Theano, Bokeh, Plotly, Scikit-learn, Scikit-image

The post Data Science Packages in Python (Video) appeared first on Mouse Vs Python.

Categories: FLOSS Project Planets

PyBites: How to make a nice graph using Django and Chart.js

Planet Python - Tue, 2022-06-14 10:41

In this article I will show you how to make a beautiful yet simple graph using Django + Chart.js. 

The code for this project is here.

Getting the data

We are going to plot the number of Bite exercises that get completed per month on our platform

For this I exported the live data from the corresponding table in our database using this handy Postgres command:

postgres@0:pybites> \copy (select * from bites_table) to '~/Downloads/bites.csv' WITH (FORMAT CSV, HEADER)

This will export the content of the table to a csv file.

Setting up Django

Next we will make a Django project and app so we can share it with this post:

$ mkdir bite_stats && cd $_ √ bite_stats $ python3.10 -m venv venv && source venv/bin/activate (venv) √ bite_stats $ pip install django python-dateutil # latter for date parsing ... (venv) √ bite_stats $ django-admin startproject mysite . (venv) √ bite_stats $ django-admin startapp stats (venv) √ bite_stats $ tree . ├── manage.py ├── mysite │ ├── __init__.py │ ├── asgi.py │ ├── settings.py │ ├── urls.py │ └── wsgi.py └── stats ├── __init__.py ├── admin.py ├── apps.py ├── migrations │ └── __init__.py ├── models.py ├── tests.py └── views.py 3 directories, 13 files

Note the trailing dot (.) to the startproject command. I always do that in Django to not get the extra nested directory.

Create a model (database table)

Next let’s make the model that will hold the stats for our graph. I keep it very simple because the purpose of this post is to make a simple graph:

from django.db import models class BiteStat(models.Model): exercise = models.PositiveSmallIntegerField() # 0 to 32767 completed = models.DateField() # I don't care about time here level = models.PositiveSmallIntegerField(null=True, blank=True) # optional, not every Bite has user feedback

I add the stats app to my INSTALLED_APPS in settings.py and run python manage.py makemigrations to make the migration file, then python manage.py migrate to sync it (and all other pending Django migrations) to the database.

Note we just use the standard sqlite3 database that Django provides out of the box.

Also note I did not update the secret key and other env variables, because this is a toy app example. Check this video how to do this in a regular Django project.

Django importer command

Next let’s get the stats csv loaded in using a Django command.

First make the required directory structure:

(venv) √ bite_stats $ mkdir -p stats/management/commands

Then make a module in that directory. The name you’ll give it will become the new switch to manage.py, I am going with stats/management/commands/import_stats.py so I can run it like: python manage.py import_stats

As per the docs we need to subclass BaseCommand and implement the handle() method.

I actually did a complete walk through of Django commands here.

I am adding a command line argument -c (--csv) to point the csv file on my computer. 

Note that I parse the date string to a datetime object using python-dateutil and I need to convert rows without a user level to 0 to stay consistent with the column type:

import csv from django.core.management.base import BaseCommand from dateutil.parser import parse from stats.models import BiteStat class Command(BaseCommand): help = 'Import bite exercise stats' def add_arguments(self, parser): parser.add_argument('-c', '--csv', required=True) def handle(self, *args, **options): file = options["csv"] with open(file) as f: reader = csv.DictReader(f) for row in reader: completed = row["first_completed"] if not completed: continue level = row["user_level"] if not level: level = 0 date = parse(completed) stat, created = BiteStat.objects.get_or_create( exercise=row["bite_id"], completed=date, level=level, ) if created: self.stdout.write(f"{stat} created") else: self.stderr.write(f"{stat} already in db")

Some more observations:

  • We get the command line arg through the options dict.
  • csv.DictReader() is really nice to load in a csv file and associate each row with the column names (keys).
  • Django ORM’s get_or_create() is a nice helper to only create the object if it does not exist yet. This makes the script idempotent (I can run it again without getting duplicate records).
  • We use the super class’ stdout and stderr objects for nicer output formatting.

Let’s run it:

python manage.py import_stats -c ~/Downloads/bites.csv

This is literally scrolling over the terminal for a good while as I am writing this, lot of Bites completed on our platform

Let’s now plot this data in a simple view using Chart.js …

Create a route to a new view

I am creating a view and route first:

from django.contrib import admin from django.urls import path from stats import views urlpatterns = [ path('', views.index), # new path('admin/', admin.site.urls), ]

To keep it simple I just do this in the main router (urls.py), not making a new one at the app level.

Create a view to get the data and link to a template

In stats/views.py I create a simple (function based) view to retrieve the records we just imported and build up the x and y values for the graph.

Btw I much prefer function based views over class based ones, see this resource why …

from collections import Counter from math import ceil from django.shortcuts import render from stats.models import BiteStat def index(request): stats = BiteStat.objects.order_by('completed') data = Counter() for row in stats: yymm = row.completed.strftime("%Y-%m") data[yymm] += 1 # unpack dict keys / values into two lists labels, values = zip(*data.items()) context = { "labels": labels, "values": values, } return render(request, "graph.html", context)

For counting things collections.Counter() is a Standard Library staple and zip(*list_of_tuples) is a nice way to unpack values to a list of labels + values.

Create the template with the graph using Chart.js

I render a graph.html template we’ll create next. I put this template in mysite/templates, a directory I created.

I took the basic example from the docs page, linked to the library using its CDN, and used Django’s templating language to populate the labels and data attributes as per the (context) variables passed in from the view.

Note that normally we would abstract repeated template code in a base.html and use inheritance. I just used a single template to keep things simple.

<!DOCTYPE html> <html> <head> <title>Bite exercise stats</title> </head> <body> <canvas id="myChart" width="800" height="400"></canvas> <script src="https://cdn.jsdelivr.net/npm/chart.js"></script> <script> const ctx = document.getElementById('myChart').getContext('2d'); const myChart = new Chart(ctx, { type: 'bar', data: { labels : [{% for item in labels %}"{{ item }}",{% endfor %}], datasets: [{ label: "Bite exercises complete per month", data : [{% for item in values %}{{ item }},{% endfor %}], backgroundColor: 'rgba(75, 192, 192, 0.2)', borderColor: 'rgba(75, 192, 192, 1)', borderWidth: 1 }] }, options: { scales: { y: { beginAtZero: true } } } }); </script> </body> </html>

In order for Django to find this template I put the path to the template directory to DIRS in the TEMPLATES list in settings.py:

TEMPLATES = [ ... 'DIRS': [Path(BASE_DIR) / 'mysite' / 'templates'], ...

Again check out the source code of this project in this repo.

The resulting graph

Now let’s run python manage.py runserver and see if it works …

Navigating to localhost:8000 I see this – sweet!

Bite exercises completed per month since we started with the platform. Hovering over the bars with your mouse you get nice tooltips. Wrap up

Cool! There are two peaks, what were those? It turns out those were the two months we participated in some massive Humble Bundle promotions where we offered Bite exercise packages.

A lot of people signed up to the platform then and started to redeem their Bite tokens and hence code the Bites. It’s really cool to see those trends using data visualization.

Overall we’re blown away with a steady rate of > 1K exercises being solved every month!

So there you go, I hope this gave you a little template you can use to easily build your own graph using Django.

We challenge you!

Note that I also imported the user_level (scale 1-10) per Bite. What if you create a second graph to show the Bites on the x-axis and their average user_level (difficulty) ratings on the y-axis?

Remember deliberate practice is key, so it’s time to fork the repo and get your hands dirty

Feel free to PR (pull request) your solution …

Good luck and keep calm and code in Python!

– Bob

Categories: FLOSS Project Planets

Real Python: Combining Data in pandas With concat() and merge()

Planet Python - Tue, 2022-06-14 10:00

The Series and DataFrame objects in pandas are powerful tools for exploring and analyzing data. Part of their power comes from a multifaceted approach to combining separate datasets. With pandas, you can merge and concatenate your datasets, allowing you to unify and better understand your data as you analyze it.

In this video course, you’ll learn how and when to combine your data in pandas with:

  • merge() for combining data on common columns or indices
  • concat() for combining DataFrames across rows or columns

If you have some experience using DataFrame and Series objects in pandas and you’re ready to learn how to combine them, then this video course will help you do exactly that. If you’re feeling a bit rusty, then you can watch a quick refresher on DataFrames before proceeding.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

Plasma 5.25: Amazement Guaranteed

Planet KDE - Tue, 2022-06-14 07:45

Plasma 5.25 brings new features and concepts to the desktop environment. Check out how you can now handle your desktop through gestures on your touchpad or touchscreen, the advanced and easy to use customization features, an improved and very useful Overview, and much, much more.

Find out more about this release

Music: Lightness By Nomyn, distributed under a CC By license.

Special thanks to: Maksym Hazevych, Ryhor Aŭrukievič, Manuel Jesús de la Fuente, Freya, Aditya Yadav, Rachit, David McCullah, Aura Herrero Ruiz

Categories: FLOSS Project Planets

Pages