FLOSS Project Planets

Promet Source: A Transformative Approach to Drupal Migration

Planet Drupal - Thu, 2023-12-14 12:48
Over the past several months, much of the focus within the Drupal community has concerned the imperative of Drupal migration.  Between the Nov. 1, 2003 end of life for Drupal 9, and the announcement at DrupalCon last Spring that Jan. 5, 2025 was the final, final end-of-life date for Drupal 7, there has been a steady stream of messaging concerning the Why, the When, and the How of upgrading to Drupal 10. 
Categories: FLOSS Project Planets

Arturo Borrero González: OpenTofu: handcrafted include-file mechanism with YAML

Planet Debian - Thu, 2023-12-14 10:00

I recently started playing with Terraform/OpenTofu almost on a daily basis.

The other day I was working with Amazon Managed Prometheus (or AMP), and wanted to define prometheus alert rules on YAML files.

I decided that I needed a way to put the alerts on a bunch of files, and then load them by the declarative code, on the correct AMP workspace.

I came up with this code pattern that I’m sharing here, for my future reference, and in case it is interesting to someone else.

The YAML file where I specify the AMP workspace, and where the alert rule files live:

--- alert_files: my_alerts_production: amp: workspace: "production" files: "alert_rules/production/*.yaml" my_alerts_staging: amp: workspace: "staging" files: "alert_rules/staging/*.yaml"

Note the files entry contains a file pattern. I will later expand the pattern using the fileset() function.

Each rule file would be something like this:

--- name: "my_rule_namespace" rule_data: | # this is prometheus-specific config groups: - name: "example_alert_group" rules: - alert: Example_Alert_Cpu # just arbitrary values, to produce an example alert expr: avg(rate(ecs_cpu_seconds_total{container=~"something"}[2m])) > 1 for: 10s annotations: summary: "CPU usage is too high" description: "The container average CPU usage is too high."

I’m interested in the data structure mutating into something similar to this:

--- alert_files: my_alerts_production: amp: workspace: "production" alerts_data: - name: rule_namespace_1 rule_data: | # actual alert definition here [..] - name: rule_namespace_2 rule_data: | # actual alert definition here [..] my_alerts_staging: amp: workspace: "staging" alerts_data: - name: rule_namespace_1 rule_data: | # actual alert definition here [..] - name: rule_namespace_2 rule_data: | # actual alert definition here [..]

This is the algorithm that does the trick:

locals { alerts_config = { for x, y in { for k, v in local.config.alert_files : k => { amp : (v.amp), files : fileset("", v.files) } } : x => { amp : (y.amp), alertmanager_data : [ for z in(y.files) : yamldecode(file(z)) ] } } }

Because the declarative nature of the Terraform/OpenTofu language, I needed to implement 3 different for loops. Each loop reads the map and transforms it in some way, passing the result into the next loop. A bit convoluted if you ask me.

To explain the logic, I think it makes more sense to read it from inside out.

First loop:

for k, v in local.config.alert_files : k => { amp : (v.amp), files : fileset("", v.files) }

This loop iterates the input YAML map in key-value pairs, remapping each amp entry, and expanding the file globs using the fileset() into a temporal files entry.

Second loop:

for x, y in { # previous fileset() loop } : x => { amp : (y.amp), alertmanager_data : [ # yamldecode() loop ] }

This intermediate loop is responsible for building the final data structure. It iterates the previous fileset() loop to remap it calling the next loop, the yamldecode() one. Note how the amp entry is being “rebuilt” in each remap (first loop and this one), otherwise we would lose it!

Third loop:

alertmanager_data : [ for z in(y.files) : yamldecode(file(z)) ]

And finally, this is maybe the easiest loop of the 3, we iterate the temporal file entry that was created in the first loop, calling yamldecode() for each of the file names generated by fileset().

The resulting data structure should allow you to easily create resources later in a for_each loop.

Categories: FLOSS Project Planets

Andy Wingo: service update

GNU Planet! - Thu, 2023-12-14 09:28

Late last year I switched blog entries and comments to be written in a dialect of markdown, but there was a bug that I never noticed: if a text consisted only of a single paragraph or block, it would trigger an error that got reported back to the user in a very strange way, and which would prevent the comment from being posted.

I had never seen the error myself because blog posts are generally more than a paragraph, but it must have been quite irritating when commenting. Sorry about that; it should be fixed now. Should you experience more strange errors, please do send me an email with the comment to wingo@igalia.com. Cheers.

Categories: FLOSS Project Planets

Drupal Association blog: Pitch-burgh updates end 2023 (November & December)

Planet Drupal - Thu, 2023-12-14 06:54

The year is coming to a close and activity is already ramping down as we approach the Christmas festivities, so this month my PitchBurgh update will cover November and December.

But, before I jump into those updates, and as we are reaching important milestones and PitchBurgh is proving its value to Drupal and the community, let me highlight the importance of the companies that have made PitchBurgh possible, without whom anything of this would have been possible. Those companies are:

As I mentioned, PitchBurgh projects are crossing some important milestones, reaching their midpoint projects, and some are even getting to the end of the projects themselves. Let’s cover some important of the past weeks:

Access Policy

Kristiaan’s work on Access Policy API got merged https://www.drupal.org/project/drupal/issues/3376843

This was a relatively complex project, not just because of the technical side, but also because of what it did imply in terms of communication and coordination with the different people involved, and the fact that it needed approval from the core team to get this work committed into Core.

The big concern with this PitchBurgh project, as well as the JSON API one, was having the investment and all the work done, and preventing the work from getting stuck waiting for approvals and merges for months.

So, seeing how this project is the first to cross the line and get merged to core, is not just a relief, but proof that we can get complex things done as a community. It just takes a little bit of organization and communication between all parties involved, in this case the Drupal Association was the one ensuring that this communication and updates were happening often.

This project was finally committed on 17 November 2023 and will be in Drupal 10.3. You can watch a video/demo of the project here. We are now working to ensure that Drupal core itself also implements the new API by the time 10.3 is released.

JSON API

JSON api has completed their work as well

You can watch their video here:

https://www.previousnext.com.au/blog/pitchburgh-diaries-decoupled-layout-builder-sprint-3-4

Decoupled Layout Builder

Decoupled LB reached project mid point, so right now the work is being reviewed before the funds get released for this part of the project.

You can read all the updates and the rest of the information of the project on their Drupal.org issue:  https://www.drupal.org/project/drupal/issues/3375422  

You can also read the project wrap up here.

Drupal API

It has been an eventful month for the Drupal API Client. 

Reaching a big milestone for the project, we’ve published a 0.1.0 release on npm which represents our Vertical Slice POC. Now that we have a small sample of the client out in the real world, we’ve created an issue for soliciting feedback from the community. Any feedback small or large would be extremely helpful for the project. We’ve also been spreading the word through an ‘Update on the Drupal API Client’ blog post and a session at New England Drupal Camp.

We are also happy to announce that coby.sher and pratik_kamble have joined the project as maintainers. They both played a critical role in making our POC a reality and we’re excited to have their leadership going forward.

As we gather feedback, we’ll continue work on the JSON:API Client 1.0 release meta issue. We could still use contributors. If you’d like to participate, join us in the #api-client channel in Drupal Slack.

Gutenberg in Drupal

Gutenberg project is progressing well. After the different conversations and alignments with the team in Automattic, there is a date for the workshop and a provisional agenda. Here is a sneak peak at said agenda:

  • In depth go-through of how we built Drupal Gutenberg
  • The Wordpress build scripts
  • Handling permissions within Gutenberg
  • Short term structured data with JSON field
  • Possible long term structured Gutenberg
  • Gutenberg for single field editing
  • Collaborative editing

Across each of our progress we've paid out a total of 38,037.75 for milestones achieved, with 2 projects already finished, one that has crossed midpoint, and the rest advancing at a good pace.

Categories: FLOSS Project Planets

Russell Coker: Fat Finger Shell

Planet Debian - Thu, 2023-12-14 05:35

I’ve been trying out the Fat Finger Shell which is a terminal emulator for Linux on touch screen devices where the keyboard is overlayed with the terminal output. This means that instead of having a tiny keyboard and a tiny terminal output you have the full screen for both. There is a YouTube video showing how the Fat Finger Shell works [1].

Here is a link to the Github page [2], which hasn’t changed much in the last 11 years.

Currently the shell is hard-coded to a 80*24 terminal and a 640*480 screen which doesn’t match any modern hardware. Some parts of this are easy to change but then there’s the comment “I ran once XGetGeometry and I am harcoded (bad) values for x, y, etc..” which is followed by some magic numbers that are not easy to change which are hacked into the source of xvt.

The configuration of this is almost great. It has a plain text file where each line has 4 numbers representing the X and Y coordinates of opposite corners of a rectangle and additional information on what the key is, which is relatively easy to edit. But then it has an image which has to match that, the obvious improvement would be to not have an image but to just display rectangles for each pair of corner coordinates and display the glyph of the character in question inside it.

I think there is a real need for a terminal like this for use on devices like the PinePhonePro, it won’t be to everyone’s taste but the people who like it will really like it. The features that such a shell needs for modern use are being based on Wayland, supporting a variety of screen resolutions and particularly the commonly used ones like 720*1440 and 1920*1080 (with terminal resolution matching the combination of screen resolution and font), and having code derived from a newer terminal emulator. As a final note it would be good for such a terminal to also take input from a regular keyboard so when you plug your Linux phone into a dock you don’t need to close your existing terminal sessions.

There is a Debian RFP/ITP bug for this [3] which I think should be closed due to nothing happening for 11 years and the fact that so much work is required to make this usable.

The current Fat Finger Shell code is a good demonstration of the concept, but I don’t think it makes sense to move on with this code base. One of the many possible ways of addressing this with modern graphics technology might be to have a semi transparent window overlaying the screen and generating virtual keyboard events for whichever window happens to be below it so instead of being limited to one terminal program by the choice of input method have that input work for any terminal that the user may choose as well as any other text based program (email, IM, etc).

Related posts:

  1. Running a Shell in a Daemon Domain allow unconfined_t logrotate_t:process transition; allow logrotate_t { shell_exec_t bin_t }:file...
  2. Logging Shell Commands In response to my previous post about logging in directly...
  3. Some Tips for Shell Code that Won’t Destroy Your OS When writing a shell script you need to take some...
Categories: FLOSS Project Planets

Talk Python to Me: #442: Ultra High Speed Message Parsing with msgspec

Planet Python - Thu, 2023-12-14 03:00
If you're a fan of Pydantic or dataclasses, you'll definitely be interested in this episode. We are talking about a super fast data modeling and validation framework called msgspec. Some of the types in here might even be better for general purpose use than Python's native classes. Join me and Jim Crist-Harif to talk about his data exchange framework, mspspec.<br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Jim Crist-Harif</b>: <a href="https://jcristharif.com" target="_blank" rel="noopener">jcristharif.com</a><br/> <b>Jim @ GitHub</b>: <a href="https://github.com/jcrist" target="_blank" rel="noopener">github.com</a><br/> <b>Jim @ Mastdon</b>: <a href="https://hachyderm.io/@jcristharif" target="_blank" rel="noopener">@jcristharif@hachyderm.io</a><br/> <br/> <b>msgspec</b>: <a href="https://github.com/jcrist/msgspec" target="_blank" rel="noopener">github.com</a><br/> <b>Projects using msgspec</b>: <a href="https://github.com/jcrist/msgspec/network/dependents?dependent_type=REPOSITORY" target="_blank" rel="noopener">github.com</a><br/> <b>msgspec on Conda Forge</b>: <a href="https://anaconda.org/conda-forge/msgspec" target="_blank" rel="noopener">anaconda.org</a><br/> <b>msgspec on PyPI</b>: <a href="https://pypi.org/project/msgspec/" target="_blank" rel="noopener">pypi.org</a><br/> <b>Litestar web framework</b>: <a href="https://litestar.dev" target="_blank" rel="noopener">litestar.dev</a><br/> <b>Litestar episode</b>: <a href="https://talkpython.fm/episodes/show/433/litestar-effortlessly-build-performant-apis" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>Pydantic V2 episode</b>: <a href="https://talkpython.fm/episodes/show/376/pydantic-v2-the-plan" target="_blank" rel="noopener">talkpython.fm</a><br/> <b>JSON parsing with msgspec article</b>: <a href="https://pythonspeed.com/articles/faster-python-json-parsing/" target="_blank" rel="noopener">pythonspeed.com</a><br/> <br/> <b>msgspec bencharmks</b>: <a href="https://jcristharif.com/msgspec/benchmarks.html" target="_blank" rel="noopener">jcristharif.com</a><br/> <b>msgspec vs. pydantic v1 and pydantic v2</b>: <a href="https://gist.github.com/jcrist/d62f450594164d284fbea957fd48b743" target="_blank" rel="noopener">github.com</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=9Qq2q-9HHqs" target="_blank" rel="noopener">youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/442/ultra-high-speed-message-parsing-with-msgspec" target="_blank" rel="noopener">talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" rel="noopener">youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" rel="noopener"><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div><br/> <strong>--- Episode sponsors ---</strong><br/> <a href='https://talkpython.fm/posit'>Posit</a><br> <a href='https://talkpython.fm/training'>Talk Python Training</a>
Categories: FLOSS Project Planets

LostCarPark Drupal Blog: Drupal Advent Calendar day 14 - DDEV

Planet Drupal - Thu, 2023-12-14 02:00
Drupal Advent Calendar day 14 - DDEV james Thu, 12/14/2023 - 07:00

It’s day 14, and we’re wrapping up the second week with a big topic. Ryan Price (liberatr) joins us to talk about DDEV, the local development environment.

When you first boot up your computer and start working on a local copy of your site, what does that look like? Is it easy? Is it a chore? (please tell me you’re working locally if you’re doing any theme or module development)

If you got a new laptop tomorrow, would you know how to get your local copies of your sites running on that machine? The idea of setting up a new machine for development used to scare me. Now that I use docker-based tools…

Tags
Categories: FLOSS Project Planets

Dirk Eddelbuettel: RProtoBuf 0.4.21 on CRAN: Updated Upstream Support!

Planet Debian - Wed, 2023-12-13 19:49

An exciting new release 0.4.21 of RProtoBuf arrived on CRAN earlier today. RProtoBuf provides R with bindings for the Google Protocol Buffers (“ProtoBuf”) data encoding and serialization library used and released by Google, and deployed very widely in numerous projects as a language and operating-system agnostic protocol.

ProtoBuf development, following what seemed like a multi-year lull, all of a sudden picked up again with a vengeance a little while ago. And the library releases we rely on for convenience and provided by the Linux distributions are lagging. So last summer we received an excellent, and focussed, pull request #93 offering to update the package to the newer ProtoBuf 22.0 and beyond. (Aside: When a library ditches its numbering scheme you know changes are ‘for real’. My Ubuntu 23.10 box is still at 3.21 in a different counting scheme .) But it wasn’t until last weekend the issue ticket #95 by Sebastian ran into the same issue, but recognized it … and contained a container recipe! So now all of a sudden we were able to build under a newer ProtoBuf which made accepting the PR #93 much easier! We added this as an additional continuous unit test, and made a few other smaller updates to documentation and style.

The following section from the NEWS.Rd file has full details.

Changes in RProtoBuf version 0.4.21 (2022-12-13)
  • Package now builds with ProtoBuf >= 22.x thanks to Matteo Gianella (#93 addressing #92).

  • An Alpine 3.19-based workflow was added to test this in continuous integration thanks to a suggestion by Sebastian Meyer.

  • A large number of old-style .Call were updated (#96).

  • Several packaging, dcoumentation and testing items were updated.

Thanks to my CRANberries, there is a diff to the previous release. The RProtoBuf page has copies of the (older) package vignette, the ‘quick’ overview vignette, and the pre-print of our JSS paper. Questions, comments etc should go to the GitHub issue tracker off the GitHub repo.

If you like this or other open-source work I do, you can sponsor me at GitHub.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Seth Michael Larson: Python listed as memory-safe language in latest CISA recommendations

Planet Python - Wed, 2023-12-13 19:00
Python listed as memory-safe language in latest CISA recommendations AboutBlogNewsletterLinks Python listed as memory-safe language in latest CISA recommendations

Published 2023-12-14 by Seth Larson
Reading time: minutes

This critical role would not be possible without funding from the OpenSSF Alpha-Omega project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem!

Memory-safety is clearly a top priority for software security, making up 70% of vulnerabilities in popular software like Chrome and Windows. The US Government RFI on open source security had a focus area on memory-safety to which the Python Software Foundation provided a response, for which I was the primary author on this topic.

The US government organization CISA (Cybersecurity and Infrastructure Security Agency) issued new recommendations this week regarding securing software products through memory safety. You can read the full information sheet here.

From the recommendation I have the following quote, emphasis mine:

“Recommended memory safe programming languages mentioned in the CSI include C#, Go, Java, Python, Rust, and Swift.”

This is awesome news for Python users! 🥳

More support will be needed to migrate Python's extensive packaging ecosystem from memory-unsafe languages to memory-safe languages. I've made recommendations on how best the US government can aid in that effort in the PSF's RFI response. My recommendations included:

  • Provide resources (financial or time) to aid in discovery, improving tooling (like PyO3), and the actual migration of projects.
  • Prioritizing projects for migration based on security-sensitivity and criticality.
  • Learning from the experiences of projects which have migrated from C to Rust like cryptography.
  • Hardening for projects which can't migrate to memory-safe languages for a variety of reasons.

You can see the current usage of memory-unsafe languages in Python projects in a previous article.

Other items
  • PyPI Safety and Security Engineer Mike Fielder published an announcement about 2FA enforcement of PyPI starting January 1st, 2024. All users of PyPI will be required to use 2FA.
  • Initial pull request adding Software Bill-of-Materials for CPython dependencies has been merged. I've created a list of sub-issues on different projects for where the work will be progressing next.
  • Łukasz will dry-run the automation for the CPython release process for CPython 3.13.0a3 (according to PEP 719 will be in about a week).
  • Created the CVE record and advisory for CVE-2023-6507. This vulnerability affects only CPython 3.12.0.
  • Submitted my year-end report to OpenSSF Alpha-Omega. Subscribe to the Alpha-Omega blog to be notified when the year-end report is published.
  • Submitted to the PyCon US CFP.

That's all for this week! 👋 If you're interested in more you can read last week's report.

Thanks for reading! ♡ Did you find this article helpful and want more content like it? Get notified of new posts by subscribing to the RSS feed or the email newsletter.

This work is licensed under CC BY-SA 4.0

Categories: FLOSS Project Planets

KDE Ships Frameworks 5.113.0

Planet KDE - Wed, 2023-12-13 19:00

Thursday, 14 December 2023

KDE today announces the release of KDE Frameworks 5.113.0.

KDE Frameworks are 83 addon libraries to Qt which provide a wide variety of commonly needed functionality in mature, peer reviewed and well tested libraries with friendly licensing terms. For an introduction see the KDE Frameworks release announcement.

This release is part of a series of planned monthly releases making improvements available to developers in a quick and predictable manner.

New in this version Baloo
  • [ExtractorProcess] Handle signal mangling by DrKonqi (bug 421317)
  • [ExtractorProcess] Test the various processing states
  • [ExtractorProcess] Make the extractor process path a constructor argument
  • [BasicIndexingBenchmark] Allow to keep temporary DB
  • [QueryTest] Add test case for terms folded to empty strings (bug 407664)
  • [DocumentUrlDb] Reduce allocations during path reconstruction
  • Shortcut non-matching phrase queries
  • [EngineQuery] Remove AND/OR operations
  • [QueryParser] Remove it, as it is no longer used
  • [SearchStore] Always use TermGenerator instead of QueryParser (bug 412421)
  • [Extractor] Do not emit startedIndexingFile for skipped documents (bug 462009)
  • Remove dead registerBalooWatcher DBus method
  • [FileWatchTest] Check attr changed signal when XAttr is not available
  • [FileWatchTest] Replace common boilerplate with RAII
  • [KInotify] Fix _k_addWatches helper when hitting descriptor limit
  • [KInotify] Simplify dirIter logic, use unique_ptr
  • [KInotify] Always add all watches in the event loop
  • [KInotify] Silence EventMoveSelf warning when not relevant
  • [KInotify] Removed unused and obsolete available() method
Extra CMake Modules
  • ECMQtDeclareLoggingCategory: support kdebugsettings files w/ . in basename
  • Add Find7Zip, deprecate Find7z, revert Find7z broken Linux support
  • Find7z: make it work also on non-Windows systems
Framework Integration
  • Reflect identifier change of oxygen-icons
KActivitiesStats
  • Adapt to renamed activities library repo
  • Adapt to move out of Frameworks
KActivities
  • Adapt to move out of Frameworks
KArchive
  • Fix broken bzip2 with new shared-mime-info
KConfig
  • Make KConfigWatcher noop for in-memory configs
KConfigWidgets
  • KRecentFileActions: Fix use after free (bug 476312)
KCoreAddons
  • kdirwatch: don't crash after moving threads (bug 472862)
  • kdirwatch: don't leave relative entries dangling
  • kdirwatch: expand ref counting system to keep account of public instances
  • kdirwatch: always unref d, and unset d from inside d (bug 472862)
KFileMetaData
  • Output generic file types from dump utility
  • Check if file path passed to dump utility is a readable file
  • [FFmpegExtractor] Bail out on first missing component during find_package (bug 458313)
  • Bump minimum Exiv2 version to 0.26
  • [TaglibExtractor] Add support for Ogg stream with FLAC (audio/x-flac+ogg) (bug 475352)
  • Generate fromName hash table programmatically from PropertyInfo table
  • Use proxy class for lowercase property name lookup
  • Reduce PropertyInfo::fromName overhead
  • [FFMpegExtractor] Support and check video/vnd.avi
  • [TaglibExtractor] Adapt code to SMI changes for audio/x-wav -> /vnd.wave
KIO
  • Add KF5 include for KPropertiesDialogPlugin
  • Add build option for KF6 coinstallability
  • Adaptations for shared-mime-info >= 2.3
KRunner
  • Adapt to plasma-framework being renamed to libplasma
  • Deprecate old QueryMatch::setType which will be removed in KF6
  • Backport KF6 API to set categoryRelevance for QueryMatch
  • Adapt to plasma-framework moving to Plasma
KService
  • Remove warning about empty Exec field from KService::exec() (bug 430157)
Plasma Framework
  • Adapt to renamed activities library repo
  • Adjust to KWayland moving to Plasma
  • Adapt to kactivities moving to Plasma
  • Reflect identifier change of oxygen-icons
Sonnet
  • Use the cmake variables rather than if(TARGET)
Security information

The released code has been GPG-signed using the following key: pub rsa2048/58D0EE648A48B3BB 2016-09-05 David Faure faure@kde.org Primary key fingerprint: 53E6 B47B 45CE A3E0 D5B7 4577 58D0 EE64 8A48 B3BB

Categories: FLOSS Project Planets

James Bennett: Database views in Django

Planet Python - Wed, 2023-12-13 15:56

This is part of a series of posts I’m doing as a sort of Python/Django Advent calendar, offering a small tip or piece of information each day from the first Sunday of Advent through Christmas Eve. See the first post for an introduction.

A view to a database

Most databases support creating and working with views, which, if you’ve never encountered them before, are like a virtual table — instead of being defined by …

Read full entry

Categories: FLOSS Project Planets

Jonathan Dowland: equivalence problems with StreamGraph

Planet Debian - Wed, 2023-12-13 15:51

I've been tackling an equivalence problem with rewritten programs in StrIoT, our proof-of-concept stream-processing system.

The StrIoT Logical Optimiser applies a set of rewrite rules to a stream-processing program, generating a set of variants that can be reasoned about, ranked, and deployed. The problem I've been tackling is that a variant may appear to be semantically equivalent to another, but compare (with ==) as distinct.

The issue relates to the design of our data-type representing programs, in particular, a consequence of our choice to outsource the structural aspect to a 3rd-party library (Algebra.Graph). The Graph library deems nodes that compare as equivalent (again with ==) to be the same. Since a stream-processing program may contain many operators which are equivalent, but distinct, we needed to add a field to our payload type to differentiate them: so we opted for an Integer field, vertexId (something I've described as a "wart" elsewhere)

Here's a simplified example of our existing payload type, StreamVertex:

data StreamVertex = StreamVertex { vertexId :: Int , operator :: StreamOperator , parameters :: [ExpQ] , intype :: String , outtype :: String }

A rewrite rule might introduce or eliminate operators from a stream-processing program. For example, consider the rule which "hoists" a filter upstream from a merge operator. In pseudo-Haskell,

streamFilter p . streamMerge [a , b, ...] => streamMerge [ streamFilter p a , streamFilter p b , ...]

The original streamFilter is removed, and new streamFilters are introduced, one per stream arriving at streamMerge. In general, rules may need to synthesise new operators, and thus new vertexIds.

Another rewrite rule might perform the reverse operation. But the individual rules operate in isolation: and so, the program variant that results after applying a rule and then applying an inverse rule may not have the same vertexIds, or the same order of vertexIds, as the original program.

I thought of the outline of two possible solutions to this.

"well-numbered" StreamGraphs

The first was to encode (and enforce) some rules about how vertexIds are used. If they always began from (say) 1, and were strictly-ascending from the source operator(s), and rewrite rules guaranteed that a "well numbered" input would be "well numbered" after rewriting, this would be sufficient to rule out a rewritten-but-semantically-equivalent program being considered distinct.

The trouble with this approach is using properties of a numerical system built around vertexId as a stand-in for the real structural problem. I was not sure I could prove both that the stand-in system was sound and that it was a proper analogue for the underlying structural issue.

It feels to me more that the choice to use an external library to encode the structure of a stream-processing program was the issue: the structure itself is a fundamental part of the semantics of the program. What if we had encoded the structure of programs within the same data-type?

alternative data-type

StrIoT programs are trees. The root is the sink node: there is always exactly one. There can be multiple source (leaf nodes), but they always converge. Operators can have multiple inputs (including zero). The root node has no output, but all other operators have exactly one.

I explored transforming StreamVertex into a tree by adding a field representing incoming streams, and dispensing with Graph and vertexId. Something like this

data StreamProg = StreamProg StreamOperator [Exp] String String [StreamProg]

A uni-directional transformation from Graph StreamVertex to StreamProg is all that's needed to implement something like ==, so we don't need to keep track of vertexId mappings. Unfortunately, we can't fix the actual Eq (Graph StreamVertex) implementation this way: it delegates to Eq StreamVertex, and we just don't have enough information to fix the problem at that level. But, we can write a separate graphEq and use that instead where we need to.

could I go further?

Spoiler: I haven't. But I've been sorely tempted.

We still have a separate StreamOperator type, which it would be nice to fold in; and we still have to use a list around the incoming nodes, since different operators accept different numbers of incoming streams. It would be better to encode the correct valences in the type.

In 2020 I explored iteratively reducing the StreamVertex data-type to try and get it as close as possible to the ideal end-user API: simple functions. I wrote about one step along that path in Template Haskell and Stream-processing programs, but concluded that, since this was not my main PhD focus, I wouldn't go further. But it was nagging at my subconcious ever since.

I allowed myself a couple of days exploring some advanced concepts including typed Template Haskell (that has had some developments since 2020), generalised abstract data types (GADTs) and more generic programming to see what could be achieved.

I'll summarise all that in the next blog post.

Categories: FLOSS Project Planets

PyCharm: Buy PyCharm. Support Data Science.

Planet Python - Wed, 2023-12-13 12:06
PyCharm is joining forces with NumFOCUS to support their Year End Fundraiser

Here at PyCharm, we acknowledge the profound impact of data science in the realm of Python and the importance of projects backed by NumFOCUS for the future of data science and the Python ecosystem.

PyCharm Professional stands out with robust support for data science workflows, encompassing interactive tables, Jupyter notebooks, SQL database support, and more.

How to support data science?

Until December 23, we extend a special offer to new users—a 30% discount for PyCharm Professional annual subscriptions. All proceeds from this campaign will be donated to support NumFOCUS and advance the field of data science.

Buy PyCharm


What is NumFOCUS?

NumFOCUS is a non-profit organization that offers a stable, independent, and professional foundation for open-source projects that drive modern scientific inquiry and business processes. If you rely on essential tools such as NumPy, Jupyter, pandas, Matplotlib, conda, or various other open-source data science projects, you are directly benefiting from projects backed by NumFOCUS.

Securing financial support is crucial for the sustained growth of their organization. Alongside contributions from corporate sponsors, event proceeds, and foundation grants, the generous contributions from individual donors play a pivotal role in supplying the essential resources needed to advance their mission of promoting open and accessible scientific computing.

Your donation significantly contributes to their overarching goal of supporting open-source scientific computing projects and fostering education in this dynamic field.

Don’t miss your chance to get a professional tool and get more work done while supporting your favorite projects. Use this deal yourself or just share it to spread the word!

Let’s support Data Science together!

Categories: FLOSS Project Planets

Real Python: Build a Scalable Flask Web Project From Scratch

Planet Python - Wed, 2023-12-13 09:00

Flask is a powerful and flexible micro web framework for Python, ideal for both small and large web projects. It provides a straightforward way to get a web application up and running, with all the features that you need to get started.

Over the course of this tutorial, you’ll explore the process of creating a boilerplate for a Flask web project. This boilerplate will serve as a great starting point for any scalable Flask web app that you wish to develop in the future, from basic web pages to complex web applications.

In this tutorial, you’ll learn how to:

  • Set up a Flask project
  • Create a "Hello, World!" Flask app
  • Add multiple pages with blueprints
  • Leverage Jinja templates
  • Store static files

At the end of this tutorial, you’ll have created a working Flask web application that you can use to bootstrap your future Flask projects.

Get Your Code: Click here to download the free source code that shows you how to build a Flask project boilerplate.

Prerequisites

You don’t need any previous knowledge of Flask to complete this project. If you want to learn more about the topics that you encounter in this tutorial, then you’ll find links to resources along the way.

However, you should be comfortable using the terminal and have basic knowledge of Python. Although it helps to know about virtual environments and pip, you’ll learn how to set everything up as you work through the tutorial.

Project Overview

In this tutorial, you’ll start building a Flask project, which could become a public message board at some point. Here’s what the project will look at the end of this tutorial:

You create a basic Flask project with two pages that inherit content and style from a parent template.

The Flask project that you’ll build in this tutorial will be very generic. That way, it’s the perfect base for any of your future projects. For example, maybe you’ll want to add a database or implement logging and notification messages.

Get Started

In this section, you’ll prepare the development environment for your Flask project. First, you’ll create a virtual environment and install all the dependencies that you need for your project.

Create a Virtual Environment

In this section, you’ll build your project structure. You can name the root folder of your project any way you like. For example, you could name it rp_flask_board/ if your ultimate goal is to create a message board. If you have another project in mind, then feel free to name your folder accordingly. Create your folder and navigate into it:

Shell $ mkdir rp_flask_board $ cd rp_flask_board Copied!

In this case, you name the root folder of your project rp_flask_board/. The files and folders that you create over the course of this series will be located in either this folder or its subfolders.

After you navigate to the project folder, it’s a good idea to create and activate a virtual environment. That way, you’re installing any project dependencies not system-wide but only in your project’s virtual environment.

Select your operating system below and use your platform-specific command to set up a virtual environment:

Windows PowerShell PS> python -m venv venv PS> .\venv\Scripts\activate (venv) PS> Copied! Shell $ python -m venv venv $ source venv/bin/activate (venv) $ Copied!

With the commands shown above, you create and activate a virtual environment named venv by using Python’s built-in venv module. The parenthesized (venv) in front of the prompt indicates that you’ve successfully activated the virtual environment.

Read the full article at https://realpython.com/flask-project/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Categories: FLOSS Project Planets

DrupalEasy: How best to handle "Unsafe usage of new static()" PHPStan errors

Planet Drupal - Wed, 2023-12-13 08:29

As a Drupal module developer, if you're as big of a fan of PHPStan as I am, then you'll no doubt have run across the Unsafe usage of new static() error that PHPStan throws when analyzing code at level 0. In this article I will describe what causes this error and provide four possible solutions, weighing the pros and cons of each.

Generally, this will occur in a create() method when you are injecting a dependency into a class using either ContainerFactoryPluginInterface or ContainerInjectionInterface using the common (simplified) Drupal pattern of:

class baseClass { public function __construct(array $configuration, $plugin_id, $plugin_definition, ModuleHandlerInterface $module_handler) { parent::__construct($configuration, $plugin_id, $plugin_definition); $this->moduleHandler = $module_handler; } public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) { return new static( $configuration, $plugin_id, $plugin_definition, $container->get('module_handler'), ); } }

Basically, PHPStan is warning you that there could be an issue if the class is extended and the new class overrides the constructor with different arguments. For example, if the Token service class is added:

class newClass extends baseClass { public function __construct(array $configuration, $plugin_id, $plugin_definition, Token $token, ModuleHandlerInterface $module_handler) { parent::__construct($configuration, $plugin_id, $plugin_definition); $this->token = $token; $this->moduleHandler = $module_handler; } }

At this point, when newClass is instantiated by the container, it will fail because its inherited create() method no longer matches its overridden constructor. This is bad.

Now, this coding pattern is all over Drupal core and contributed modules, and we are generally very careful about breaking things (thank you, automated tests.) Regardless, PHPStan isn't cool with it, so let's look at some alternatives. 

Dig hole in sand, stick head in it

Easiest, but probably not a good long-term solution.

PHPStan provides the ability for certain errors to be ignored, and this is definitely a good candidate for ignoring. Additionally, it is simple enough to do with a quick addition to your phpstan.neon configuration file:

parameters: level: 0 paths: - web/modules ignoreErrors: - '#Unsafe usage of new static\(\)#'

Generally, I'm not a big fan of ignoring any code quality issues, whether they are from phpcs, PHPStan or any other code quality tool.

Pros

  • PHPStan error goes away

Cons

  • That nagging feeling that you really didn't fix anything.
Prevent others from extending the class

Easiest for custom code, with limitations.

If using new static is only an issue when the class is extended, then why not just prevent the class from being extended with the addition of the final keyword?

final class baseClass { … }

This definitely solves the issue, and in many cases, is a valid solution. I almost always use final when writing a custom module - especially when I am in control of the code base.

Using final in core or a contributed module is a bit trickier, as in most cases the assumption does have to be made that someone somewhere is going to extend that class for reasons you haven't thought of. Being the co-maintainer of several contributed modules, I'm experimenting with using it - to see if anyone opens an issue letting me know that it is blocking them in some way or another. Time will tell.

Pros

  • PHPStan is happy.
  • Code protection against someone extending the class and not properly overriding the constructor and create() methods (if necessary).

Cons

  • The class cannot be extended, possibly limiting its usefulness.
Change new static to new self

Most correct, but with potential repercussions.

Recommended by a Drupal.org documentation page, but with little explanation, this change looks like this (from our example above:)

public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition) { return new self( $configuration, $plugin_id, $plugin_definition, $container->get('module_handler'), ); }

In this case, if the class is extended and the constructor is overridden, then the code will always fail unless the create() method is also overridden. Curiously, the documentation page also suggests adding final to the class. I suspect that the documentation page should be updated, but I'm not confident enough in what is the best way to handle this to make a suggestion. Yet.

Pros

  • PHPStan is happy.
  • This is probably the "most correct" solution.

Cons

  • While the class can be extended, less experienced developers might not always recognize that they must override the create() method when overriding the constructor, potentially leading to a bug that can be tricky to figure out. For example, if the create() method isn't overridden, then the result would be that the base class is created and not the (expected) extended class. 
Make the constructor final

Similar to marking the entire class "final".

I haven't spotted this potential solution in a Drupal community discussion, but on the PHPStan blog, one of the suggestions is to just make the constructor final. Again updating our code sample from above:

final public function __construct(array $configuration, $plugin_id, $plugin_definition, ModuleHandlerInterface $module_handler) { parent::__construct($configuration, $plugin_id, $plugin_definition); $this->moduleHandler = $module_handler; }

This seems like a slightly more flexible solution than making the entire class final or changing from new static to new self, but still would limit anyone extending the class. Maybe this is a reasonable middle-of-the-road solution? I haven't seen it implemented in Drupal yet.

This method would allow the class to be extended, but the constructor wouldn't be allowed to be overridden - which means no dependency changes can be made. Obviously, this limits what extended classes would be able to do.

Similar to this method, the blog post also suggested adding the constructor signature to an interface, which would have the same effect of locking it down so that it cannot be changed.

Pros

  • PHPStan is happy.
  • Code protection against someone extending the class and not properly overriding the constructor and create() methods by not allowing those methods to be overridden. 

Cons

  • If the constructor can't be overridden, then while the class can be extended, its dependencies cannot be changed, potentially limiting its usefulness. 
Summary

So where does that leave us? Hopefully, with the ability to make a more informed decision on how to handle this particular PHPStan error. As I mentioned above, I'm not a big fan of ignoring PHPStan errors, so for now, I'll be adding final most of the time 😉.

Additional reading

Thanks to Professional Module Development course alumni James Shields and Hanane Kacemi as well as Ryan Price for reviewing this blog post.

Categories: FLOSS Project Planets

Tag1 Consulting: Gander: Performance Testing Made Easy

Planet Drupal - Wed, 2023-12-13 08:00

Join us for an update on Gander, the automated performance testing framework making waves in Drupal Core. Developed by Tag1 Consulting and the Google Chrome team, Gander is set to transform how we approach performance testing. In this Tag1 Team Talk Mariano Crivello, Nat Catchpole, and Michael Meyers share the story behind Gander's development. Learn how it's simplifying performance testing for developers, reducing the burden on maintainers, and becoming a game-changer for organizations using Drupal. Part one of our two-part series dives into Gander's history, benefits, and roadmap. Stay tuned for part two, where we'll give you a hands-on demo of Gander in action. Ready to level up your Drupal game? Dive into the Gander story now! Links: - Gander Announcement: The Future of Drupal Performance & Scalability - Google’s Core Web Vitals - Core Web Vitals: Google & Tag1 Improving Drupal’s Performance, User Experience, and Your Bottom Line - from DrupalCon Pittsburgh --- For a transcript of this...

Read more michaelemeyers Wed, 12/13/2023 - 05:00
Categories: FLOSS Project Planets

Melissa Wen: 15 Tips for Debugging Issues in the AMD Display Kernel Driver

Planet Debian - Wed, 2023-12-13 07:25

A self-help guide for examining and debugging the AMD display driver within the Linux kernel/DRM subsystem.

It’s based on my experience as an external developer working on the driver, and are shared with the goal of helping others navigate the driver code.

Acknowledgments: These tips were gathered thanks to the countless help received from AMD developers during the driver development process. The list below was obtained by examining open source code, reviewing public documentation, playing with tools, asking in public forums and also with the help of my former GSoC mentor, Rodrigo Siqueira.

Pre-Debugging Steps:

Before diving into an issue, it’s crucial to perform two essential steps:

1) Check the latest changes: Ensure you’re working with the latest AMD driver modifications located in the amd-staging-drm-next branch maintained by Alex Deucher. You may also find bug fixes for newer kernel versions on branches that have the name pattern drm-fixes-<date>.

2) Examine the issue tracker: Confirm that your issue isn’t already documented and addressed in the AMD display driver issue tracker. If you find a similar issue, you can team up with others and speed up the debugging process.

Understanding the issue:

Do you really need to change this? Where should you start looking for changes?

3) Is the issue in the AMD kernel driver or in the userspace?: Identifying the source of the issue is essential regardless of the GPU vendor. Sometimes this can be challenging so here are some helpful tips:

  • Record the screen: Capture the screen using a recording app while experiencing the issue. If the bug appears in the capture, it’s likely a userspace issue, not the kernel display driver.
  • Analyze the dmesg log: Look for error messages related to the display driver in the dmesg log. If the error message appears before the message “[drm] Display Core v...”, it’s not likely a display driver issue. If this message doesn’t appear in your log, the display driver wasn’t fully loaded and you will see a notification that something went wrong here.

4) AMD Display Manager vs. AMD Display Core: The AMD display driver consists of two components:

  • Display Manager (DM): This component interacts directly with the Linux DRM infrastructure. Occasionally, issues can arise from misinterpretations of DRM properties or features. If the issue doesn’t occur on other platforms with the same AMD hardware - for example, only happens on Linux but not on Windows - it’s more likely related to the AMD DM code.
  • Display Core (DC): This is the platform-agnostic part responsible for setting and programming hardware features. Modifications to the DC usually require validation on other platforms, like Windows, to avoid regressions.

5) Identify the DC HW family: Each AMD GPU has variations in its hardware architecture. Features and helpers differ between families, so determining the relevant code for your specific hardware is crucial.

  • Find GPU product information in Linux/AMD GPU documentation
  • Check the dmesg log for the Display Core version (since this commit in Linux kernel 6.3v). For example:
    • [drm] Display Core v3.2.241 initialized on DCN 2.1
    • [drm] Display Core v3.2.237 initialized on DCN 3.0.1
Investigating the relevant driver code:

Keep from letting unrelated driver code to affect your investigation.

6) Narrow the code inspection down to one DC HW family: the relevant code resides in a directory named after the DC number. For example, the DCN 3.0.1 driver code is located at drivers/gpu/drm/amd/display/dc/dcn301. We all know that the AMD’s shared code is huge and you can use these boundaries to rule out codes unrelated to your issue.

7) Newer families may inherit code from older ones: you can find dcn301 using code from dcn30, dcn20, dcn10 files. It’s crucial to verify which hooks and helpers your driver utilizes to investigate the right portion. You can leverage ftrace for supplemental validation. To give an example, it was useful when I was updating DCN3 color mapping to correctly use their new post-blending color capabilities, such as:

Additionally, you can use two different HW families to compare behaviours. If you see the issue in one but not in the other, you can compare the code and understand what has changed and if the implementation from a previous family doesn’t fit well the new HW resources or design. You can also count on the help of the community on the Linux AMD issue tracker to validate your code on other hardware and/or systems.

This approach helped me debug a 2-year-old issue where the cursor gamma adjustment was incorrect in DCN3 hardware, but working correctly for DCN2 family. I solved the issue in two steps, thanks for community feedback and validation:

8) Check the hardware capability screening in the driver: You can currently find a list of display hardware capabilities in the drivers/gpu/drm/amd/display/dc/dcn*/dcn*_resource.c file. More precisely in the dcn*_resource_construct() function. Using DCN301 for illustration, here is the list of its hardware caps:

/************************************************* * Resource + asic cap harcoding * *************************************************/ pool->base.underlay_pipe_index = NO_UNDERLAY_PIPE; pool->base.pipe_count = pool->base.res_cap->num_timing_generator; pool->base.mpcc_count = pool->base.res_cap->num_timing_generator; dc->caps.max_downscale_ratio = 600; dc->caps.i2c_speed_in_khz = 100; dc->caps.i2c_speed_in_khz_hdcp = 5; /*1.4 w/a enabled by default*/ dc->caps.max_cursor_size = 256; dc->caps.min_horizontal_blanking_period = 80; dc->caps.dmdata_alloc_size = 2048; dc->caps.max_slave_planes = 2; dc->caps.max_slave_yuv_planes = 2; dc->caps.max_slave_rgb_planes = 2; dc->caps.is_apu = true; dc->caps.post_blend_color_processing = true; dc->caps.force_dp_tps4_for_cp2520 = true; dc->caps.extended_aux_timeout_support = true; dc->caps.dmcub_support = true; /* Color pipeline capabilities */ dc->caps.color.dpp.dcn_arch = 1; dc->caps.color.dpp.input_lut_shared = 0; dc->caps.color.dpp.icsc = 1; dc->caps.color.dpp.dgam_ram = 0; // must use gamma_corr dc->caps.color.dpp.dgam_rom_caps.srgb = 1; dc->caps.color.dpp.dgam_rom_caps.bt2020 = 1; dc->caps.color.dpp.dgam_rom_caps.gamma2_2 = 1; dc->caps.color.dpp.dgam_rom_caps.pq = 1; dc->caps.color.dpp.dgam_rom_caps.hlg = 1; dc->caps.color.dpp.post_csc = 1; dc->caps.color.dpp.gamma_corr = 1; dc->caps.color.dpp.dgam_rom_for_yuv = 0; dc->caps.color.dpp.hw_3d_lut = 1; dc->caps.color.dpp.ogam_ram = 1; // no OGAM ROM on DCN301 dc->caps.color.dpp.ogam_rom_caps.srgb = 0; dc->caps.color.dpp.ogam_rom_caps.bt2020 = 0; dc->caps.color.dpp.ogam_rom_caps.gamma2_2 = 0; dc->caps.color.dpp.ogam_rom_caps.pq = 0; dc->caps.color.dpp.ogam_rom_caps.hlg = 0; dc->caps.color.dpp.ocsc = 0; dc->caps.color.mpc.gamut_remap = 1; dc->caps.color.mpc.num_3dluts = pool->base.res_cap->num_mpc_3dlut; //2 dc->caps.color.mpc.ogam_ram = 1; dc->caps.color.mpc.ogam_rom_caps.srgb = 0; dc->caps.color.mpc.ogam_rom_caps.bt2020 = 0; dc->caps.color.mpc.ogam_rom_caps.gamma2_2 = 0; dc->caps.color.mpc.ogam_rom_caps.pq = 0; dc->caps.color.mpc.ogam_rom_caps.hlg = 0; dc->caps.color.mpc.ocsc = 1; dc->caps.dp_hdmi21_pcon_support = true; /* read VBIOS LTTPR caps */ if (ctx->dc_bios->funcs->get_lttpr_caps) { enum bp_result bp_query_result; uint8_t is_vbios_lttpr_enable = 0; bp_query_result = ctx->dc_bios->funcs->get_lttpr_caps(ctx->dc_bios, &is_vbios_lttpr_enable); dc->caps.vbios_lttpr_enable = (bp_query_result == BP_RESULT_OK) && !!is_vbios_lttpr_enable; } if (ctx->dc_bios->funcs->get_lttpr_interop) { enum bp_result bp_query_result; uint8_t is_vbios_interop_enabled = 0; bp_query_result = ctx->dc_bios->funcs->get_lttpr_interop(ctx->dc_bios, &is_vbios_interop_enabled); dc->caps.vbios_lttpr_aware = (bp_query_result == BP_RESULT_OK) && !!is_vbios_interop_enabled; }

Keep in mind that the documentation of color capabilities are available at the Linux kernel Documentation.

Understanding the development history:

What has brought us to the current state?

9) Pinpoint relevant commits: Use git log and git blame to identify commits targeting the code section you’re interested in.

10) Track regressions: If you’re examining the amd-staging-drm-next branch, check for regressions between DC release versions. These are defined by DC_VER in the drivers/gpu/drm/amd/display/dc/dc.h file. Alternatively, find a commit with this format drm/amd/display: 3.2.221 that determines a display release. It’s useful for bisecting. This information helps you understand how outdated your branch is and identify potential regressions. You can consider each DC_VER takes around one week to be bumped. Finally, check testing log of each release in the report provided on the amd-gfx mailing list, such as this one Tested-by: Daniel Wheeler:

Reducing the inspection area:

Focus on what really matters.

11) Identify involved HW blocks: This helps isolate the issue. You can find more information about DCN HW blocks in the DCN Overview documentation. In summary:

  • Plane issues are closer to HUBP and DPP.
  • Blending/Stream issues are closer to MPC, OPP and OPTC. They are related to DRM CRTC subjects.

This information was useful when debugging a hardware rotation issue where the cursor plane got clipped off in the middle of the screen.

Finally, the issue was addressed by two patches:

12) Issues around bandwidth (glitches) and clocks: May be affected by calculations done in these HW blocks and HW specific values. The recalculation equations are found in the DML folder. DML stands for Display Mode Library. It’s in charge of all required configuration parameters supported by the hardware for multiple scenarios. See more in the AMD DC Overview kernel docs. It’s a math library that optimally configures hardware to find the best balance between power efficiency and performance in a given scenario.

Finding some clk variables that affect device behavior may be a sign of it. It’s hard for a external developer to debug this part, since it involves information from HW specs and firmware programming that we don’t have access. The best option is to provide all relevant debugging information you have and ask AMD developers to check the values from your suspicions.

  • Do a trick: If you suspect the power setup is degrading performance, try setting the amount of power supplied to the GPU to the maximum and see if it affects the system behavior with this command: sudo bash -c "echo high > /sys/class/drm/card0/device/power_dpm_force_performance_level"

I learned it when debugging glitches with hardware cursor rotation on Steam Deck. My first attempt was changing the clock calculation. In the end, Rodrigo Siqueira proposed the right solution targeting bandwidth in two steps:

Checking implicit programming and hardware limitations:

Bring implicit programming to the level of consciousness and recognize hardware limitations.

13) Implicit update types: Check if the selected type for atomic update may affect your issue. The update type depends on the mode settings, since programming some modes demands more time for hardware processing. More details in the source code:

/* Surface update type is used by dc_update_surfaces_and_stream * The update type is determined at the very beginning of the function based * on parameters passed in and decides how much programming (or updating) is * going to be done during the call. * * UPDATE_TYPE_FAST is used for really fast updates that do not require much * logical calculations or hardware register programming. This update MUST be * ISR safe on windows. Currently fast update will only be used to flip surface * address. * * UPDATE_TYPE_MED is used for slower updates which require significant hw * re-programming however do not affect bandwidth consumption or clock * requirements. At present, this is the level at which front end updates * that do not require us to run bw_calcs happen. These are in/out transfer func * updates, viewport offset changes, recout size changes and pixel depth changes. * This update can be done at ISR, but we want to minimize how often this happens. * * UPDATE_TYPE_FULL is slow. Really slow. This requires us to recalculate our * bandwidth and clocks, possibly rearrange some pipes and reprogram anything front * end related. Any time viewport dimensions, recout dimensions, scaling ratios or * gamma need to be adjusted or pipe needs to be turned on (or disconnected) we do * a full update. This cannot be done at ISR level and should be a rare event. * Unless someone is stress testing mpo enter/exit, playing with colour or adjusting * underscan we don't expect to see this call at all. */ enum surface_update_type { UPDATE_TYPE_FAST, /* super fast, safe to execute in isr */ UPDATE_TYPE_MED, /* ISR safe, most of programming needed, no bw/clk change*/ UPDATE_TYPE_FULL, /* may need to shuffle resources */ }; Using tools:

Observe the current state, validate your findings, continue improvements.

14) Use AMD tools to check hardware state and driver programming: help on understanding your driver settings and checking the behavior when changing those settings.

  • DC Visual confirmation: Check multiple planes and pipe split policy.

  • DTN logs: Check display hardware state, including rotation, size, format, underflow, blocks in use, color block values, etc.

  • UMR: Check ASIC info, register values, KMS state - links and elements (framebuffers, planes, CRTCs, connectors). Source: UMR project documentation

15) Use generic DRM/KMS tools:

  • IGT test tools: Use generic KMS tests or develop your own to isolate the issue in the kernel space. Compare results across different GPU vendors to understand their implementations and find potential solutions. Here AMD also has specific IGT tests for its GPUs that is expect to work without failures on any AMD GPU. You can check results of HW-specific tests using different display hardware families or you can compare expected differences between the generic workflow and AMD workflow.

  • drm_info: This tool summarizes the current state of a display driver (capabilities, properties and formats) per element of the DRM/KMS workflow. Output can be helpful when reporting bugs.

Don’t give up!

Debugging issues in the AMD display driver can be challenging, but by following these tips and leveraging available resources, you can significantly improve your chances of success.

Worth mentioning: This blog post builds upon my talk, “I’m not an AMD expert, but…” presented at the 2022 XDC. It shares guidelines that helped me debug AMD display issues as an external developer of the driver.

Open Source Display Driver: The Linux kernel/AMD display driver is open source, allowing you to actively contribute by addressing issues listed in the official tracker. Tackling existing issues or resolving your own can be a rewarding learning experience and a valuable contribution to the community. Additionally, the tracker serves as a valuable resource for finding similar bugs, troubleshooting tips, and suggestions from AMD developers. Finally, it’s a platform for seeking help when needed.

Remember, contributing to the open source community through issue resolution and collaboration is mutually beneficial for everyone involved.

Categories: FLOSS Project Planets

Qt for MCUs 2.2.5 LTS Released

Planet KDE - Wed, 2023-12-13 06:27

Qt for MCUs 2.2.5 LTS (Long-Term Support) has been released and is available for download. As a patch release, Qt for MCUs 2.2.5 LTS provides bug fixes and other improvements, and maintains source compatibility with Qt for MCUs 2.2.x. It does not add any new functionality.

Categories: FLOSS Project Planets

Jonathan Dowland: William Basinski, Gateshead, 2022

Planet Debian - Wed, 2023-12-13 04:05

I was looking over the list of live music I'd seen this year and realised that avante-garde composer William Basinski was actually last year and I had forgotten to write about it!

In November 2022, Basinski headlined a night of performances which otherwise featured folk from the venue's "Arists in Residence" programme, with some affiliation to Newcastle's DIY music scene.

Unfortunately we arrived too late to catch any of the other acts: partly because of the venue's sometimes doggiest insistence that people can only enter or leave the halls during intervals, and partly because the building works surrounding it had made the southern entrance effectively closed, so we had to walk to the north side of the building to get in1.

Basinski was performing work from Lamentations. Basinski himself presented very unexpectedly to how I imagined him: he's got the Texas drawl, mediated through a fair amount of time spent in New York; very camp, in a glittery top; he kicked off the gig complaining about how tired he was, before a mini rant about the state of the world, riffing on a title from the album: Please, This Shit Has Got To Stop.

We were in Hall 1, the larger of the two, and it was sparsely attended; a few people walked out mid performance. My gig-buddy Rob (a useful barometer for me on how things have gone) remarked that it was one of the most unique and unusual gigs he'd been to. I recognised snatches of the tracks from the album, but I'm hard-placed to name or sequence them, and they flowed into each other. I don't know how much of what we were hearing was "live" or what, if anything, was being decided during the performance, but Basinski's set-up included what looked like archaic tape equipment, with exposed loops of tape running between spools that could be interfered with by other tools.

The encore was a unique, unreleased mix of Melancholia (II), which (making no apologies) Basinski hit play on before retiring backstage.

I didn't take any photos. From memory, I think the venue had specifically stated filming or photos were not allowed for this performance. People at prior shows in New York and London filmed some of their shows; which were substantially similar: I've included embeds of them above.

Lots of Basinski's work is on Bandcamp; the three pieces I particularly enjoy are Lamentations,

Lamentations by William Basinski

On Time Out of Time,

On Time Out of Time by William Basinski

and his best-known work, The Disintegration Loops.

The Disintegration Loops by William Basinski
  1. I don't want to speak ill of the venue, though: The Sage, as it was, and the Glasshouse, as it is now known, has ended up being the venue I've attended most this year (2023), and it's such a civilised place: plenty of bars, great drinks selection (both alcoholic and not, hot and cold), loads of clean toilets, a free cloakroom, fantastic accoustics, polite staff; the list goes on.
Categories: FLOSS Project Planets

LostCarPark Drupal Blog: Drupal Advent Calendar day 13 - Mentoring

Planet Drupal - Wed, 2023-12-13 02:00
Drupal Advent Calendar day 13 - Mentoring james Wed, 12/13/2023 - 07:00

It’s day 13 and we’ve fewer than half of the doors remaining. For today’s door we’re taking a break from modules to talk about mentoring with Anto Jose (antojose).

I would like to use this calendar day to tell you about my experience mentoring at DrupalCons.

I signed up for mentoring at DrupalCon Lille, not because I thought of myself particularly as mentoring material, but because of the joy I found years back, when trying to help some people get their first taste of contributing, first during DrupalCon Asia 2016, and then during DrupalCon Amsterdam 2019, and then in a couple of online…

Tags
Categories: FLOSS Project Planets

Pages