Feeds

As I said on linked in, this week my brain was fried and also buzzing while working on getting a beta release of the LocalGov Live Preview module.

In this article, I'll show you how to import data from CSV files into Drupal. This is often a step in creating a new website, and the solutions shown here will come in handy when transferring data from an old system to a new one. I’ll also demonstrate how to handle such a situation using the Feeds module and its Feeds Tamper extension and walk you through importing data using three types of data as examples: products, users, and blog posts.

In this quiz, you’ll test your understanding of The Python Standard REPL: Try Out Code and Ideas Quickly.

The Python REPL allows you to run Python code interactively, which is useful for testing new ideas, exploring libraries, refactoring and debugging code, and trying out examples.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Implementing Pagination in Drupal admin Thu, 10/24/2024 - 14:43

Changing your mind about dependency injection

When I start writing a class that has a dependency injection, I have a clear idea about which services it needs. I generate it -- the plugin, form, controller, or service -- and specify those services.

Then nearly always, unless it's something really very simple, I find that no matter how much I thought about it and planned it, I need to add more services. Maybe remove some too.

Fortunately, because Module Builder saves the configuration of the module code you've generated, it's easy to go back to it and edit it to add more services:

1. Edit your module in Module Builder
2. Add to the injected services for your component
3. Ensure your code file is committed to version control
4. Generate the code, and write the updated version of the code file
5. Add and commit the new DI code, while discarding the changes that remove your code. (I find it helps to use a git GUI for things like this, though git add -p works too.)

But I tend to find that I make this mistake several times as the class developers, and so I adopt the approach of using the \Drupal::service() function to get my services, and only when I'm fairly confident I'm not going to need to make any more changes to DI, I update the injected services in one go, converting all the service calls to use the service properties.

I was asked yesterday at Drupal Drinks about how to do that, and it occurred to me that there's a way of doing this so after you've updated the dependency injection with Module Builder, it's a simple find and replace to update your code.

If you write your code like this whenever you need a service:

$service_entityTypeManager = \Drupal::service('entity_type.manager'); $stuff = $service_entityTypeManager->doSomething();

Then you need to do only two find and replace operations to convert this to DI:

1. Replace '^.+Drupal::service.+\n' with ''. This removes all the lines where you get the service from the Drupal class.
2. Replace '\$service_(\w+)' with '$this->$1'. This replaces all the service variables with the class property.

Up until now I'd been calling the service variables something like $entityTypeManager so that I could easily change that to $this->entityTypeManager manually, but prefixing the variable name with a camel case 'service_' gives you something to find with a regular expression.

If you want to be really fancy, you can use a regular expression like '(?<=::service..)[\w.]+' (using dots to avoid having to escape the open bracket and the quote mark) to find all the services that you need to add to the class's dependency injection.

Something like this:

$ ag -G MyClass.php '(?<=::service..)[\w.]+' -o --nonumbers --nofilename | sort | uniq | tr "\n" ", "

will give you a list of service names that you can copy-paste into the Module Builder form. This is probably overkill for something you can do pretty quickly with the search in a text editor or IDE, but it's a nice illustration of the power of unix tools: ag has options to output just the found text, then sort and uniq eliminate duplicates, and finally tr turns it into a comma-separated list.

joachim Thu, 10/24/2024 - 11:25 Tags

• dependency injection
• drupal code builder
• module builder

Drupal 7: Navigating the Maintenance Maze bhavinhjoshi Thu, 10/24/2024 - 15:52

As the technology advances, and where change is the only constant, the question arises: why remain anchored to Drupal 7? This familiar platform has served many well, but as digital ecosystems grow, the costs of inaction multiply. The world of web development is never static, and adhering to outdated technology can impede your progress and competitive edge.

Let's dive deeper into the specific challenges and opportunities for those still using Drupal 7. One crucial aspect is the complexity of customisations made out of necessity during its heyday. While these custom modules and tweaks may have added significant value at the time, they now serve as barriers to progress—highlighting the risks of deviating from best practices.

 

 

Custom modules often become the lifeblood of a Drupal 7 site, but they also present hurdles. Each line of bespoke code demands scrutiny. This isn't merely a case of asking whether the customisations still function, but whether they are still fit for purpose in a more modern context. As we look towards Drupal 10 and beyond, it's worth questioning whether there exists a current core or contributed module that could replace complex custom work. This requires an expert evaluation, which isn’t optional but essential for a smooth migration. Understanding the intent and function of each customisation empowers you to streamline and possibly automate future updates.

Contributed modules add another layer of complexity. While it's tempting to wait for updates, proactive engagement within the Drupal community can unlock solutions, paving the way for an easier migration. The Drupal community is one of the platform's strongest assets—by getting involved, you can influence the direction of module development and even participate in updating modules critical to your website. This active participation ensures that you’re not only a passive consumer of someone else's labour but an active contributor to your digital landscape.

Security remains a paramount concern. Maintaining Drupal 7 involves more than just patching security holes. It’s about optimising performance, refining user experience, and bolstering defences. Old modules can introduce vulnerabilities, and outdated practices might not comply with current security standards. Here, the role of a dedicated hosting expert becomes invaluable. Collaborating with hosting partners who specialise in Drupal can ease the administrative burden, ensuring your site remains secure and efficient without derailing your internal resources.

The decision to stay with Drupal 7 should be informed by strategic foresight, not stagnation. It’s about recognising the potential for transformation rather than clinging to what's comfortable. Embracing newer versions does more than just upgrade your tech stack—it revitalises your business processes and user interactions.

An upgrade to Drupal 10 isn't just a technical necessity; it's a strategic advantage. It positions your organisation to leverage the latest innovations, enhance security, and improve scalability. The transition is not merely a swap of versions but an opportunity to realign with the evolving digital landscape.

Migration may seem daunting, but with a clear strategy, it becomes a journey of growth. Start by assessing your existing architecture, identify key stakeholders, and set clear objectives. Assemble a team of experts who can guide this transition, ensuring that each step aligns with your business goals.

In the end, Drupal 7's dilemma isn't just about the platform itself. It's a reflection of how we choose to adapt to change. Commit to evolution and unlock the full potential of what modern technology can offer. The future awaits those willing to embrace it with open arms and strategic intent.

Drupal 7 Drupal Drupal Planet Add new comment

 

The PSF is pleased to announce its second batch of PSF Fellows for 2024! Let us welcome the new PSF Fellows for Q2! The following people continue to do amazing things for the Python community:

Leonard Richardson

Blog

Winnie Ke 

Facebook, LinkedIn

Thank you for your continued contributions. We have added you to our Fellow roster.

The above members help support the Python ecosystem by being phenomenal leaders, sustaining the growth of the Python scientific community, maintaining virtual Python communities, maintaining Python libraries, creating educational material, organizing Python events and conferences, starting Python communities in local regions, and overall being great mentors in our community. Each of them continues to help make Python more accessible around the world. To learn more about the new Fellow members, check out their links above.

Let's continue recognizing Pythonistas all over the world for their impact on our community. The criteria for Fellow members is available online: https://www.python.org/psf/fellows/. If you would like to nominate someone to be a PSF Fellow, please send a description of their Python accomplishments and their email address to psf-fellow at python.org. Quarter 3 nominations are currently in review. We are accepting nominations for Quarter 4 through November 20th, 2024.

Are you a PSF Fellow and want to help the Work Group review nominations? Contact us at psf-fellow at python.org.

Do you struggle to make sure your code is always correct before you check it in? What about your team members' code? That one person who never wants to run the linter? Tired of dealing with tons of conflicts and spurious git changes? You need git pre-commit hooks. We're lucky to have Stefanie Molin on this episode who has done a bunch of writing and teaching of git hooks.<br/> <br/> <strong>Episode sponsors</strong><br/> <br/> <a href='https://talkpython.fm/sentry'>Sentry Error Monitoring, Code TALKPYTHON</a><br> <a href='https://talkpython.fm/bluehost'>Bluehost</a><br> <a href='https://talkpython.fm/training'>Talk Python Courses</a><br/> <br/> <strong>Links from the show</strong><br/> <br/> <div><b>Stefanie Molin</b>: <a href="https://stefaniemolin.com/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <br/> <b>Talk Python Blog</b>: <a href="https://talkpython.fm/blog/" target="_blank" >talkpython.fm/blog</a><br/> <br/> <b>How to Set Up Pre-Commit Hooks</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/setup-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Common Pre-Commit Errors and How to Solve Them</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/troubleshooting-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>A Behind-the-Scenes Look at How Pre-Commit Works</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/behind-the-scenes/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Pre-Commit Hook Creation Guide</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/hook-creation-guide/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>(Pre-)Commit to Better Code Workshop</b>: <a href="https://stefaniemolin.com/workshops/pre-commit-workshop/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>exif-stripper</b>: <a href="https://stefaniemolin.com/articles/devx/pre-commit/exif-stripper/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>exif-stripper on GitHub</b>: <a href="https://github.com/stefmolin/exif-stripper?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>docstring-validation-using-pre-commit-hook</b>: <a href="https://numpydoc.readthedocs.io/en/latest/validation.html#docstring-validation-using-pre-commit-hook" target="_blank" >numpydoc.readthedocs.io</a><br/> <b>Data Morph: Moving Beyond the Datasaurus Dozen</b>: <a href="https://stefaniemolin.com/articles/data-science/introducing-data-morph/?featured_on=talkpython" target="_blank" >stefaniemolin.com</a><br/> <b>Data Morph on GitHub</b>: <a href="https://github.com/stefmolin/data-morph?featured_on=talkpython" target="_blank" >github.com</a><br/> <b>Watch this episode on YouTube</b>: <a href="https://www.youtube.com/watch?v=EzlzX1OL92w" target="_blank" >youtube.com</a><br/> <b>Episode transcripts</b>: <a href="https://talkpython.fm/episodes/transcript/482/pre-commit-hooks-for-python-devs" target="_blank" >talkpython.fm</a><br/> <br/> <b>--- Stay in touch with us ---</b><br/> <b>Subscribe to us on YouTube</b>: <a href="https://talkpython.fm/youtube" target="_blank" >youtube.com</a><br/> <b>Follow Talk Python on Mastodon</b>: <a href="https://fosstodon.org/web/@talkpython" target="_blank" ><i class="fa-brands fa-mastodon"></i>talkpython</a><br/> <b>Follow Michael on Mastodon</b>: <a href="https://fosstodon.org/web/@mkennedy" target="_blank" ><i class="fa-brands fa-mastodon"></i>mkennedy</a><br/></div>

NOTE: pacman v7 is currently in [libre-testing]; but it will be promoted to libre soon

from arch:

With the release of [version 7.0.0] pacman has added support for downloading packages as a separate user with dropped privileges.

For users with local repos however this might imply that the download user does not have access to the files in question, which can be fixed by assigning the files and folder to the alpm group and ensuring the executable bit (+x) is set on the folders in question.

$ chown :alpm -R /path/to/local/repo

Remember to [merge the .pacnew] files to apply the new default.

Pacman also introduced [a change] to improve checksum stability for git repos that utilize .gitattributes files. This might require a one-time checksum change for PKGBUILDs that use git sources.

What does “pushing code” mean for your business? Read on to discover how easy it is to get your code from development to production without headaches.

Posted on October 24, 2024
Tags: madeof:atoms, madeof:bits, craft:zine

I have no idea either.

Happy Maladay1 to those who celebrate it, I guess.

If you care about the how, it started as china ink on tracing paper, with the help of a template (and a correction sheet for one page where I used the wrong line on the template).

A rubber stamp was carved with the author’s signature and stamped on white paper because the ink from the pad wasn’t working well on tracing paper.

Then everything was scanned (with the correction on top of the wrong page) asemic_zine_scans.tar.

Imported in Inkscape and traced asemic_zine_svg.tar.

Printed, cut in half, folded and stapled. The magenta lines weren’t by design, but are there because my printer is currently2 cursed.

And finally, asemic_zine.pdf was created, joining the pages together with pdfjam, for convenience in case somebody wants to download the full thing.

All the .tar and .pdf downloads from this page are released under the WTFPL, or All Rites Reversed..

1. it’s still technically Maladay when I write this, even if by the time you’ll get this it’s probably the 6th of The Aftermath.↩︎

2. I mean, all printers are always cursed, but at different times they can be cursed in different and novel ways.↩︎

Learn to develop an effective content strategy for your Drupal site with expert advice. Drive traffic, engage users, and boost conversions with strategic content planning.

Introduction

A content strategy is essential for any modern Drupal site as it aligns content with business goals, optimizes user experience, ensures quality and consistency, and leverages Drupal's advanced content management features for efficient governance and workflow management. Your organization’s content strategy will incorporate SEO best practices to amplify online visibility and prepare the organization for future digital trends and multi-channel distribution. 

Ultimately, a well-crafted content strategy is a linchpin in transforming a Drupal CMS into a dynamic asset that supports engagement, lead conversion, and customer loyalty, thus driving an organization's digital success.

I’m going to change up the Calamares release process a little. It’s been slow going as a community-maintained project – which isn’t to say that that is a bad thing. Just slow. I’ve decided to make releases marginally more predictable than “when [ade] has a relaxed kind of Tuesday” and have marked a couple of issues with the Calamares 3.3 milestone. When the milestone is empty again, then there will be a release. After the next release, I’ll put a couple more issues on the milestone, and the recipe can be repeated.

Many years ago I was involved in software-quality research – the SQO-OSS project and things like that. That work begat the code-quality checking scripts that we in the KDE community called “the EBN”, or EnglishBreakfastNetwork. I was a tea-drinker then. The EBN stuff has been surpassed by Klazy and many other software-quality-checking tools. But the EBN domain carries on. Although I haven’t got anything to put on it I just renewed the domain again for two years – just in case.

As is probably apparent from the sequence of blog posts about the topic in the last year, I have been thinking about and working on integer optimizations in the JIT compiler a lot. This work was mainly motivated by Pydrofoil, where integer operations matter a lot more than for your typical Python program.

In this post I'll describe my most recent change, which is a new small domain specific language that I implemented to specify peephole optimizations on integer operations in the JIT. It uses pattern matching to specify how (sequences of) integer operations should be simplified and optimized. The rules are then compiled to RPython code that then becomes part of the JIT's optimization passes.

To make it less likely to introduce incorrect optimizations into the JIT, the rules are automatically proven correct with Z3 as part of the build process (for a more hands-on intro to how that works you can look at the knownbits post). In this blog post I want to motivate why I introduced the DSL and give an introduction to how it works.

Motivation

This summer, after I wrote my scripts to mine JIT traces for missed optimization opportunities, I started implementing a few of the integer peephole rewrite that the script identified. Unfortunately, doing so led to the problem that the way we express these rewrites up to now is very imperative and verbose. Here's a snippet of RPython code that shows some rewrites for integer multiplication (look at the comments to see what the different parts actually do). You don't need to understand the code in detail, but basically it's in very imperative style and there's quite a lot of boilerplate.

def optimize_INT_MUL(self, op): arg0 = get_box_replacement(op.getarg(0)) b0 = self.getintbound(arg0) arg1 = get_box_replacement(op.getarg(1)) b1 = self.getintbound(arg1) if b0.known_eq_const(1): # 1 * x == x self.make_equal_to(op, arg1) elif b1.known_eq_const(1): # x * 1 == x self.make_equal_to(op, arg0) elif b0.known_eq_const(0) or b1.known_eq_const(0): # 0 * x == x * 0 == 0 self.make_constant_int(op, 0) else: for lhs, rhs in [(arg0, arg1), (arg1, arg0)]: lh_info = self.getintbound(lhs) if lh_info.is_constant(): x = lh_info.get_constant_int() if x & (x - 1) == 0: # x * (2 ** c) == x << c new_rhs = ConstInt(highest_bit(lh_info.get_constant_int())) op = self.replace_op_with(op, rop.INT_LSHIFT, args=[rhs, new_rhs]) self.optimizer.send_extra_operation(op) return elif x == -1: # x * -1 == -x op = self.replace_op_with(op, rop.INT_NEG, args=[rhs]) self.optimizer.send_extra_operation(op) return else: # x * (1 << y) == x << y shiftop = self.optimizer.as_operation(get_box_replacement(lhs), rop.INT_LSHIFT) if shiftop is None: continue if not shiftop.getarg(0).is_constant() or shiftop.getarg(0).getint() != 1: continue shiftvar = get_box_replacement(shiftop.getarg(1)) shiftbound = self.getintbound(shiftvar) if shiftbound.known_nonnegative() and shiftbound.known_lt_const(LONG_BIT): op = self.replace_op_with( op, rop.INT_LSHIFT, args=[rhs, shiftvar]) self.optimizer.send_extra_operation(op) return return self.emit(op)

Adding more rules to these functions is very tedious and gets super confusing when the functions get bigger. In addition I am always worried about making mistakes when writing this kind of code, and there is no feedback at all about which of these rules are actually applied a lot in real programs.

Therefore I decided to write a small domain specific language with the goal of expressing these rules in a more declarative way. In the rest of the post I'll describe the DSL (most of that description is adapted from the documentation about it that I wrote).

The Peephole Rule DSL Simple transformation rules

The rules in the DSL specify how integer operation can be transformed into cheaper other integer operations. A rule always consists of a name, a pattern, and a target. Here's a simple rule:

add_zero: int_add(x, 0) => x

The name of the rule is add_zero. It matches operations in the trace of the form int_add(x, 0), where x will match anything and 0 will match only the constant zero. After the => arrow is the target of the rewrite, i.e. what the operation is rewritten to, in this case x.

The rule language has a list of which of the operations are commutative, so add_zero will also optimize int_add(0, x) to x.

Variables in the pattern can repeat:

sub_x_x: int_sub(x, x) => 0

This rule matches against int_sub operations where the two arguments are the same (either the same box, or the same constant).

Here's a rule with a more complicated pattern:

sub_add: int_sub(int_add(x, y), y) => x

This pattern matches int_sub operations, where the first argument was produced by an int_add operation. In addition, one of the arguments of the addition has to be the same as the second argument of the subtraction.

The constants MININT, MAXINT and LONG_BIT (which is either 32 or 64, depending on which platform the JIT is built for) can be used in rules, they behave like writing numbers but allow bit-width-independent formulations:

is_true_and_minint: int_is_true(int_and(x, MININT)) => int_lt(x, 0)

It is also possible to have a pattern where some arguments needs to be a constant, without specifying which constant. Those patterns look like this:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete # more goes here => int_sub(x, C)

Variables in the pattern that start with a C match against constants only. However, in this current form the rule is incomplete, because the variable C that is being used in the target operation is not defined anywhere. We will see how to compute it in the next section.

Computing constants and other intermediate results

Sometimes it is necessary to compute intermediate results that are used in the target operation. To do that, there can be extra assignments between the rule head and the rule target.:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete C = C1 + C1 => int_sub(x, C)

The right hand side of such an assignment is a subset of Python syntax, supporting arithmetic using +, -, *, and certain helper functions. However, the syntax allows you to be explicit about unsignedness for some operations. E.g. >>u exists for unsigned right shifts (and I plan to add >u, >=u, <u, <=u for comparisons).

Here's an example of a rule that uses >>u:

urshift_lshift_x_c_c: uint_rshift(int_lshift(x, C), C) mask = (-1 << C) >>u C => int_and(x, mask) Checks

Some rewrites are only true under certain conditions. For example, int_eq(x, 1) can be rewritten to x, if x is known to store a boolean value. This can be expressed with checks:

eq_one: int_eq(x, 1) check x.is_bool() => x

A check is followed by a boolean expression. The variables from the pattern can be used as IntBound instances in checks (and also in assignments) to find out what the abstract interpretation of the JIT knows about the value of a trace variable (IntBound is the name of the abstract domain that the JIT uses for integers, despite the fact that it also stores knownbits information nowadays).

Here's another example:

mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

It expresses that x * (1 << y) can be rewritten to x << y but checks that y is known to be between 0 and LONG_BIT.

Checks and assignments can be repeated and combined with each other:

mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift)

In addition to calling methods on IntBound instances, it's also possible to access their attributes, like in this rule:

and_x_c_in_range: int_and(x, C) check x.lower >= 0 and x.upper <= C & ~(C + 1) => x Rule Ordering and Liveness

The generated optimizer code will give preference to applying rules that produce a constant or a variable as a rewrite result. Only if none of those match do rules that produce new result operations get applied. For example, the rules sub_x_x and sub_add are tried before trying sub_add_consts, because the former two rules optimize to a constant and a variable respectively, while the latter produces a new operation as the result.

The rule sub_add_consts has a possible problem, which is that if the intermediate result of the int_add operation in the rule head is used by some other operations, then the sub_add_consts rule does not actually reduce the number of operations (and might actually make things slightly worse due to increased register pressure). However, currently it would be extremely hard to take that kind of information into account in the optimization pass of the JIT, so we optimistically apply the rules anyway.

Checking rule coverage

Every rewrite rule should have at least one unit test where it triggers. To ensure this, the unit test file that mainly checks integer optimizations in the JIT has an assert at the end of a test run, that every rule fired at least once.

Printing rule statistics

The JIT can print statistics about which rule fired how often in the jit-intbounds-stats logging category, using the PYPYLOG mechanism. For example, to print the category to stdout at the end of program execution, run PyPy like this:

PYPYLOG=jit-intbounds-stats:- pypy ...

The output of that will look something like this:

int_add add_reassoc_consts 2514 add_zero 107008 int_sub sub_zero 31519 sub_from_zero 523 sub_x_x 3153 sub_add_consts 159 sub_add 55 sub_sub_x_c_c 1752 sub_sub_c_x_c 0 sub_xor_x_y_y 0 sub_or_x_y_y 0 int_mul mul_zero 0 mul_one 110 mul_minus_one 0 mul_pow2_const 1456 mul_lshift 0 ... Termination and Confluence

Right now there are unfortunately no checks that the rules actually rewrite operations towards "simpler" forms. There is no cost model, and also nothing that prevents you from writing a rule like this:

neg_complication: int_neg(x) # leads to infinite rewrites => int_mul(-1, x)

Doing this would lead to endless rewrites if there is also another rule that turns multiplication with -1 into negation.

There is also no checking for confluence (yet?), i.e. the property that all rewrites starting from the same input trace always lead to the same output trace, no matter in which order the rules are applied.

Proofs

It is very easy to write a peephole rule that is not correct in all corner cases. Therefore all the rules are proven correct with Z3 before compiled into actual JIT code, by default. When the proof fails, a (hopefully minimal) counterexample is printed. The counterexample consists of values for all the inputs that fulfil the checks, values for the intermediate expressions, and then two different values for the source and the target operations.

E.g. if we try to add the incorrect rule:

mul_is_add: int_mul(a, b) => int_add(a, b)

We get the following counterexample as output:

Could not prove correctness of rule 'mul_is_add' in line 1 counterexample given by Z3: counterexample values: a: 0 b: 1 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 0 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 1

If we add conditions, they are taken into account and the counterexample will fulfil the conditions:

mul_is_add: int_mul(a, b) check a.known_gt_const(1) and b.known_gt_const(2) => int_add(a, b)

This leads to the following counterexample:

Could not prove correctness of rule 'mul_is_add' in line 46 counterexample given by Z3: counterexample values: a: 2 b: 3 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 6 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 5

Some IntBound methods cannot be used in Z3 proofs because their control flow is too complex. If that is the case, they can have Z3-equivalent formulations defined (in every case this is done, it's a potential proof hole if the Z3 friendly reformulation and the real implementation differ from each other, therefore extra care is required to make very sure they are equivalent).

It's possible to skip the proof of individual rules entirely by adding SORRY_Z3 to its body (but we should try not to do that too often):

eq_different_knownbits: int_eq(x, y) SORRY_Z3 check x.known_ne(y) => 0 Checking for satisfiability

In addition to checking whether the rule yields a correct optimization, we also check whether the rule can ever apply. This ensures that there are some runtime values that would fulfil all the checks in a rule. Here's an example of a rule violating this:

never_applies: int_is_true(x) check x.known_lt_const(0) and x.known_gt_const(0) # impossible condition, always False => x

Right now the error messages if this goes wrong are not completely easy to understand. I hope to be able to improve this later:

Rule 'never_applies' cannot ever apply in line 1 Z3 did not manage to find values for variables x such that the following condition becomes True: And(x <= x_upper, x_lower <= x, If(x_upper < 0, x_lower > 0, x_upper < 0)) Implementation Notes

The implementation of the DSL is done in a relatively ad-hoc manner. It is parsed using rply, there's a small type checker that tries to find common problems in how the rules are written. Z3 is used via the Python API, like in the previous blog posts that are using it. The pattern matching RPython code is generated using an approach inspired by Luc Maranget's paper Compiling Pattern Matching to Good Decision Trees. See this blog post for an approachable introduction.

Conclusion

Now that I've described the DSL, here are the rules that are equivalent to the imperative code in the motivation section:

mul_zero: int_mul(x, 0) => 0 mul_one: int_mul(x, 1) => x mul_minus_one: int_mul(x, -1) => int_neg(x) mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift) mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

The current status of the DSL is that it got merged to PyPy's main branch. I rewrote a part of the integer rewrites into the DSL, but some are still in the old imperative style (mostly for complicated reasons, the easily ported ones are all done). Since I've only been porting optimizations that had existed prior to the existence of the DSL, performance numbers of benchmarks didn't change.

There are a number of features that are still missing and some possible extensions that I plan to work on in the future:

• All the integer operations that the DSL handles so far are the variants that do not check for overflow (or where overflow was proven to be impossible to happen). In regular Python code the overflow-checking variants int_add_ovf etc are much more common, but the DSL doesn't support them yet. I plan to fix this, but don't completely understand how the correctness proofs for them should be done correctly.

• A related problem is that I don't understand what it means for a rewrite to be correct if some of the operations are only defined for a subset of the input values. E.g. division isn't defined if the divisor is zero. In theory, a division operation in the trace should always be preceded by a check that the divisor isn't zero. But sometimes other optimization move the check around and the connection to the division gets lost or muddled. What optimizations can we still safely perform on the division? There's lots of prior work on this question, but I still don't understand what the correct approach in our context would be.

• Ordering comparisons like int_lt, int_le and their unsigned variants are not ported to the DSL yet. Comparisons are an area where the JIT is not super good yet at optimizing away operations. This is a pretty big topic and I've started a project with Nico Rittinghaus to try to improve the situation a bit more generally.

• A more advanced direction of work would be to implement a simplified form of e-graphs (or ae-graphs). The JIT has like half of an e-graph data structure already, and we probably can't afford a full one in terms of compile time costs, but maybe we can have two thirds or something?

Acknowledgements

Thank you to Max Bernstein and Martin Berger for super helpful feedback on drafts of the post!

In this episode of the Python Show Podcast, David Mertz is our guest. David is a prolific writer about the Python programming language. From his extremely popular IPM Developerworks articles to his multiple books on the Python language, David has been a part of the Python community for decades.

We ended up chatting about:

• The history of Python

• Book writing

• Conference speaking

• The PSF

• and more!

Show Links

• David Mertz’s website

• PyDev of the Week: David Mertz on Mouse vs Python

• David Mertz on InformIT / Pearson

Python threading allows you to run parts of your code concurrently, making the code more efficient. However, when you introduce threading to your code without knowing about thread safety, you may run into issues such as race conditions. You solve these with tools like locks, semaphores, events, conditions, and barriers.

By the end of this tutorial, you’ll be able to identify safety issues and prevent them by using the synchronization primitives in Python’s threading module to make your code thread-safe.

In this tutorial, you’ll learn:

• What thread safety is
• What race conditions are and how to avoid them
• How to identify thread safety issues in your code
• What different synchronization primitives exist in the threading module
• How to use synchronization primitives to make your code thread-safe

To get the most out of this tutorial, you’ll need to have basic experience working with multithreaded code using Python’s threading module and ThreadPoolExecutor.

Get Your Code: Click here to download the free sample code that you’ll use to learn about thread safety techniques in Python.

Take the Quiz: Test your knowledge with our interactive “Python Thread Safety: Using a Lock and Other Techniques” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Python Thread Safety: Using a Lock and Other Techniques

In this quiz, you'll test your understanding of Python thread safety. You'll revisit the concepts of race conditions, locks, and other synchronization primitives in the threading module. By working through this quiz, you'll reinforce your knowledge about how to make your Python code thread-safe.

Threading in Python

In this section, you’ll get a general overview of how Python handles threading. Before discussing threading in Python, it’s important to revisit two related terms that you may have heard about in this context:

• Concurrency: The ability of a system to handle multiple tasks by allowing their execution to overlap in time but not necessarily happen simultaneously.
• Parallelism: The simultaneous execution of multiple tasks that run at the same time to leverage multiple processing units, typically multiple CPU cores.

Python’s threading is a concurrency framework that allows you to spin up multiple threads that run concurrently, each executing pieces of code. This improves the efficiency and responsiveness of your application. When running multiple threads, the Python interpreter switches between them, handing the control of execution over to each thread.

By running the script below, you can observe the creation of four threads:

Python threading_example.py import threading import time from concurrent.futures import ThreadPoolExecutor def threaded_function(): for number in range(3): print(f"Printing from {threading.current_thread().name}. {number=}") time.sleep(0.1) with ThreadPoolExecutor(max_workers=4, thread_name_prefix="Worker") as executor: for _ in range(4): executor.submit(threaded_function) Copied!

In this example, threaded_function prints the values zero to two that your for loop assigns to the loop variable number. Using a ThreadPoolExecutor, four threads are created to execute the threaded function. ThreadPoolExecutor is configured to run a maximum of four threads concurrently with max_workers=4, and each worker thread is named with a “Worker” prefix, as in thread_name_prefix="Worker".

In print(), the .name attribute on threading.current_thread() is used to get the name of the current thread. This will help you identify which thread is executed each time. A call to sleep() is added inside the threaded function to increase the likelihood of a context switch.

You’ll learn what a context switch is in just a moment. First, run the script and take a look at the output:

Shell $ python threading_example.py Printing from Worker_0. number=0 Printing from Worker_1. number=0 Printing from Worker_2. number=0 Printing from Worker_3. number=0 Printing from Worker_0. number=1 Printing from Worker_2. number=1 Printing from Worker_1. number=1 Printing from Worker_3. number=1 Printing from Worker_0. number=2 Printing from Worker_2. number=2 Printing from Worker_1. number=2 Printing from Worker_3. number=2 Copied!

Each line in the output represents a print() call from a worker thread, identified by Worker_0, Worker_1, Worker_2, and Worker_3. The number that follows the worker thread name shows the current iteration of the loop each thread is executing. Each thread takes turns executing the threaded_function, and the execution happens in a concurrent rather than sequential manner.

For example, after Worker_0 prints number=0, it’s not immediately followed by Worker_0 printing number=1. Instead, you see outputs from Worker_1, Worker_2, and Worker_3 printing number=0 before Worker_0 proceeds to number=1. You’ll notice from these interleaved outputs that multiple threads are running at the same time, taking turns to execute their part of the code.

This happens because the Python interpreter performs a context switch. This means that Python pauses the execution state of the current thread and passes control to another thread. When the context switches, Python saves the current execution state so that it can resume later. By switching the control of execution at specific intervals, multiple threads can execute code concurrently.

You can check the context switch interval of your Python interpreter by typing the following in the REPL:

Python >>> import sys >>> sys.getswitchinterval() 0.005 Copied!

The output of calling the getswitchinterval() is a number in seconds that represents the context switch interval of your Python interpreter. In this case, it’s 0.005 seconds or five milliseconds. You can think of the switch interval as how often the Python interpreter checks if it should switch to another thread.

An interval of five milliseconds doesn’t mean that threads switch exactly every five milliseconds, but rather that the interpreter considers switching to another thread at these intervals.

The switch interval is defined in the Python docs as follows:

Read the full article at https://realpython.com/python-thread-lock/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Компьютеры и сети содействуют нам в борьбе за свободу: они помогают посвятить время и силы важным общественным инициативам, организовывать протесты, защищаться от цензуры.

Но свободны ли наши компьютеры?  И свободны ли мы как пользователи?

Обсудим эти вопросы в Открытом пространстве с Глебом Ерофеевым — активистом движения за свободные программы и волонтёром проекта "ГНУ", который в 1983 году запустил философ и активист Ричард Столлман.

Команда проекта "ГНУ" занимается разработкой свободного софта и техноэтическим активизмом, чтобы дать пользователям контроль над их компьютерами и искоренить несправедливость, которую приносят в общество собственнические программы.

Адрес: Плетешковский пер., 8с1 (м. "Бауманская").

Участие бесплатно.  Приветствуются пожертвования в пользу пространства.

The navigation menu is a crucial element of any website, guiding users through content, enhancing their experience, and playing a vital role in the site's overall usability and success.

Creating a menu that's accessible, responsive, and easy to navigate is a non-trivial task, no matter how simple or complex your navigation is. However, people usually underestimate the efforts required to create a navigation menu that provides a good user experience, and that it can take several iterations to do it right.

In this quiz, you’ll test your understanding of Python Class Constructors.

By working through this quiz, you’ll revisit the internal instantiation process, object initialization using .__init__(), and fine-tuning object creation by overriding .__new__().

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]