Feeds

Many years ago I was involved in software-quality research – the SQO-OSS project and things like that. That work begat the code-quality checking scripts that we in the KDE community called “the EBN”, or EnglishBreakfastNetwork. I was a tea-drinker then. The EBN stuff has been surpassed by Klazy and many other software-quality-checking tools. But the EBN domain carries on. Although I haven’t got anything to put on it I just renewed the domain again for two years – just in case.

As is probably apparent from the sequence of blog posts about the topic in the last year, I have been thinking about and working on integer optimizations in the JIT compiler a lot. This work was mainly motivated by Pydrofoil, where integer operations matter a lot more than for your typical Python program.

In this post I'll describe my most recent change, which is a new small domain specific language that I implemented to specify peephole optimizations on integer operations in the JIT. It uses pattern matching to specify how (sequences of) integer operations should be simplified and optimized. The rules are then compiled to RPython code that then becomes part of the JIT's optimization passes.

To make it less likely to introduce incorrect optimizations into the JIT, the rules are automatically proven correct with Z3 as part of the build process (for a more hands-on intro to how that works you can look at the knownbits post). In this blog post I want to motivate why I introduced the DSL and give an introduction to how it works.

Motivation

This summer, after I wrote my scripts to mine JIT traces for missed optimization opportunities, I started implementing a few of the integer peephole rewrite that the script identified. Unfortunately, doing so led to the problem that the way we express these rewrites up to now is very imperative and verbose. Here's a snippet of RPython code that shows some rewrites for integer multiplication (look at the comments to see what the different parts actually do). You don't need to understand the code in detail, but basically it's in very imperative style and there's quite a lot of boilerplate.

def optimize_INT_MUL(self, op): arg0 = get_box_replacement(op.getarg(0)) b0 = self.getintbound(arg0) arg1 = get_box_replacement(op.getarg(1)) b1 = self.getintbound(arg1) if b0.known_eq_const(1): # 1 * x == x self.make_equal_to(op, arg1) elif b1.known_eq_const(1): # x * 1 == x self.make_equal_to(op, arg0) elif b0.known_eq_const(0) or b1.known_eq_const(0): # 0 * x == x * 0 == 0 self.make_constant_int(op, 0) else: for lhs, rhs in [(arg0, arg1), (arg1, arg0)]: lh_info = self.getintbound(lhs) if lh_info.is_constant(): x = lh_info.get_constant_int() if x & (x - 1) == 0: # x * (2 ** c) == x << c new_rhs = ConstInt(highest_bit(lh_info.get_constant_int())) op = self.replace_op_with(op, rop.INT_LSHIFT, args=[rhs, new_rhs]) self.optimizer.send_extra_operation(op) return elif x == -1: # x * -1 == -x op = self.replace_op_with(op, rop.INT_NEG, args=[rhs]) self.optimizer.send_extra_operation(op) return else: # x * (1 << y) == x << y shiftop = self.optimizer.as_operation(get_box_replacement(lhs), rop.INT_LSHIFT) if shiftop is None: continue if not shiftop.getarg(0).is_constant() or shiftop.getarg(0).getint() != 1: continue shiftvar = get_box_replacement(shiftop.getarg(1)) shiftbound = self.getintbound(shiftvar) if shiftbound.known_nonnegative() and shiftbound.known_lt_const(LONG_BIT): op = self.replace_op_with( op, rop.INT_LSHIFT, args=[rhs, shiftvar]) self.optimizer.send_extra_operation(op) return return self.emit(op)

Adding more rules to these functions is very tedious and gets super confusing when the functions get bigger. In addition I am always worried about making mistakes when writing this kind of code, and there is no feedback at all about which of these rules are actually applied a lot in real programs.

Therefore I decided to write a small domain specific language with the goal of expressing these rules in a more declarative way. In the rest of the post I'll describe the DSL (most of that description is adapted from the documentation about it that I wrote).

The Peephole Rule DSL Simple transformation rules

The rules in the DSL specify how integer operation can be transformed into cheaper other integer operations. A rule always consists of a name, a pattern, and a target. Here's a simple rule:

add_zero: int_add(x, 0) => x

The name of the rule is add_zero. It matches operations in the trace of the form int_add(x, 0), where x will match anything and 0 will match only the constant zero. After the => arrow is the target of the rewrite, i.e. what the operation is rewritten to, in this case x.

The rule language has a list of which of the operations are commutative, so add_zero will also optimize int_add(0, x) to x.

Variables in the pattern can repeat:

sub_x_x: int_sub(x, x) => 0

This rule matches against int_sub operations where the two arguments are the same (either the same box, or the same constant).

Here's a rule with a more complicated pattern:

sub_add: int_sub(int_add(x, y), y) => x

This pattern matches int_sub operations, where the first argument was produced by an int_add operation. In addition, one of the arguments of the addition has to be the same as the second argument of the subtraction.

The constants MININT, MAXINT and LONG_BIT (which is either 32 or 64, depending on which platform the JIT is built for) can be used in rules, they behave like writing numbers but allow bit-width-independent formulations:

is_true_and_minint: int_is_true(int_and(x, MININT)) => int_lt(x, 0)

It is also possible to have a pattern where some arguments needs to be a constant, without specifying which constant. Those patterns look like this:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete # more goes here => int_sub(x, C)

Variables in the pattern that start with a C match against constants only. However, in this current form the rule is incomplete, because the variable C that is being used in the target operation is not defined anywhere. We will see how to compute it in the next section.

Computing constants and other intermediate results

Sometimes it is necessary to compute intermediate results that are used in the target operation. To do that, there can be extra assignments between the rule head and the rule target.:

sub_add_consts: int_sub(int_add(x, C1), C2) # incomplete C = C1 + C1 => int_sub(x, C)

The right hand side of such an assignment is a subset of Python syntax, supporting arithmetic using +, -, *, and certain helper functions. However, the syntax allows you to be explicit about unsignedness for some operations. E.g. >>u exists for unsigned right shifts (and I plan to add >u, >=u, <u, <=u for comparisons).

Here's an example of a rule that uses >>u:

urshift_lshift_x_c_c: uint_rshift(int_lshift(x, C), C) mask = (-1 << C) >>u C => int_and(x, mask) Checks

Some rewrites are only true under certain conditions. For example, int_eq(x, 1) can be rewritten to x, if x is known to store a boolean value. This can be expressed with checks:

eq_one: int_eq(x, 1) check x.is_bool() => x

A check is followed by a boolean expression. The variables from the pattern can be used as IntBound instances in checks (and also in assignments) to find out what the abstract interpretation of the JIT knows about the value of a trace variable (IntBound is the name of the abstract domain that the JIT uses for integers, despite the fact that it also stores knownbits information nowadays).

Here's another example:

mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

It expresses that x * (1 << y) can be rewritten to x << y but checks that y is known to be between 0 and LONG_BIT.

Checks and assignments can be repeated and combined with each other:

mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift)

In addition to calling methods on IntBound instances, it's also possible to access their attributes, like in this rule:

and_x_c_in_range: int_and(x, C) check x.lower >= 0 and x.upper <= C & ~(C + 1) => x Rule Ordering and Liveness

The generated optimizer code will give preference to applying rules that produce a constant or a variable as a rewrite result. Only if none of those match do rules that produce new result operations get applied. For example, the rules sub_x_x and sub_add are tried before trying sub_add_consts, because the former two rules optimize to a constant and a variable respectively, while the latter produces a new operation as the result.

The rule sub_add_consts has a possible problem, which is that if the intermediate result of the int_add operation in the rule head is used by some other operations, then the sub_add_consts rule does not actually reduce the number of operations (and might actually make things slightly worse due to increased register pressure). However, currently it would be extremely hard to take that kind of information into account in the optimization pass of the JIT, so we optimistically apply the rules anyway.

Checking rule coverage

Every rewrite rule should have at least one unit test where it triggers. To ensure this, the unit test file that mainly checks integer optimizations in the JIT has an assert at the end of a test run, that every rule fired at least once.

Printing rule statistics

The JIT can print statistics about which rule fired how often in the jit-intbounds-stats logging category, using the PYPYLOG mechanism. For example, to print the category to stdout at the end of program execution, run PyPy like this:

PYPYLOG=jit-intbounds-stats:- pypy ...

The output of that will look something like this:

int_add add_reassoc_consts 2514 add_zero 107008 int_sub sub_zero 31519 sub_from_zero 523 sub_x_x 3153 sub_add_consts 159 sub_add 55 sub_sub_x_c_c 1752 sub_sub_c_x_c 0 sub_xor_x_y_y 0 sub_or_x_y_y 0 int_mul mul_zero 0 mul_one 110 mul_minus_one 0 mul_pow2_const 1456 mul_lshift 0 ... Termination and Confluence

Right now there are unfortunately no checks that the rules actually rewrite operations towards "simpler" forms. There is no cost model, and also nothing that prevents you from writing a rule like this:

neg_complication: int_neg(x) # leads to infinite rewrites => int_mul(-1, x)

Doing this would lead to endless rewrites if there is also another rule that turns multiplication with -1 into negation.

There is also no checking for confluence (yet?), i.e. the property that all rewrites starting from the same input trace always lead to the same output trace, no matter in which order the rules are applied.

Proofs

It is very easy to write a peephole rule that is not correct in all corner cases. Therefore all the rules are proven correct with Z3 before compiled into actual JIT code, by default. When the proof fails, a (hopefully minimal) counterexample is printed. The counterexample consists of values for all the inputs that fulfil the checks, values for the intermediate expressions, and then two different values for the source and the target operations.

E.g. if we try to add the incorrect rule:

mul_is_add: int_mul(a, b) => int_add(a, b)

We get the following counterexample as output:

Could not prove correctness of rule 'mul_is_add' in line 1 counterexample given by Z3: counterexample values: a: 0 b: 1 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 0 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 1

If we add conditions, they are taken into account and the counterexample will fulfil the conditions:

mul_is_add: int_mul(a, b) check a.known_gt_const(1) and b.known_gt_const(2) => int_add(a, b)

This leads to the following counterexample:

Could not prove correctness of rule 'mul_is_add' in line 46 counterexample given by Z3: counterexample values: a: 2 b: 3 operation int_mul(a, b) with Z3 formula a*b has counterexample result vale: 6 BUT target expression: int_add(a, b) with Z3 formula a + b has counterexample value: 5

Some IntBound methods cannot be used in Z3 proofs because their control flow is too complex. If that is the case, they can have Z3-equivalent formulations defined (in every case this is done, it's a potential proof hole if the Z3 friendly reformulation and the real implementation differ from each other, therefore extra care is required to make very sure they are equivalent).

It's possible to skip the proof of individual rules entirely by adding SORRY_Z3 to its body (but we should try not to do that too often):

eq_different_knownbits: int_eq(x, y) SORRY_Z3 check x.known_ne(y) => 0 Checking for satisfiability

In addition to checking whether the rule yields a correct optimization, we also check whether the rule can ever apply. This ensures that there are some runtime values that would fulfil all the checks in a rule. Here's an example of a rule violating this:

never_applies: int_is_true(x) check x.known_lt_const(0) and x.known_gt_const(0) # impossible condition, always False => x

Right now the error messages if this goes wrong are not completely easy to understand. I hope to be able to improve this later:

Rule 'never_applies' cannot ever apply in line 1 Z3 did not manage to find values for variables x such that the following condition becomes True: And(x <= x_upper, x_lower <= x, If(x_upper < 0, x_lower > 0, x_upper < 0)) Implementation Notes

The implementation of the DSL is done in a relatively ad-hoc manner. It is parsed using rply, there's a small type checker that tries to find common problems in how the rules are written. Z3 is used via the Python API, like in the previous blog posts that are using it. The pattern matching RPython code is generated using an approach inspired by Luc Maranget's paper Compiling Pattern Matching to Good Decision Trees. See this blog post for an approachable introduction.

Conclusion

Now that I've described the DSL, here are the rules that are equivalent to the imperative code in the motivation section:

mul_zero: int_mul(x, 0) => 0 mul_one: int_mul(x, 1) => x mul_minus_one: int_mul(x, -1) => int_neg(x) mul_pow2_const: int_mul(x, C) check C > 0 and C & (C - 1) == 0 shift = highest_bit(C) => int_lshift(x, shift) mul_lshift: int_mul(x, int_lshift(1, y)) check y.known_ge_const(0) and y.known_le_const(LONG_BIT) => int_lshift(x, y)

The current status of the DSL is that it got merged to PyPy's main branch. I rewrote a part of the integer rewrites into the DSL, but some are still in the old imperative style (mostly for complicated reasons, the easily ported ones are all done). Since I've only been porting optimizations that had existed prior to the existence of the DSL, performance numbers of benchmarks didn't change.

There are a number of features that are still missing and some possible extensions that I plan to work on in the future:

• All the integer operations that the DSL handles so far are the variants that do not check for overflow (or where overflow was proven to be impossible to happen). In regular Python code the overflow-checking variants int_add_ovf etc are much more common, but the DSL doesn't support them yet. I plan to fix this, but don't completely understand how the correctness proofs for them should be done correctly.

• A related problem is that I don't understand what it means for a rewrite to be correct if some of the operations are only defined for a subset of the input values. E.g. division isn't defined if the divisor is zero. In theory, a division operation in the trace should always be preceded by a check that the divisor isn't zero. But sometimes other optimization move the check around and the connection to the division gets lost or muddled. What optimizations can we still safely perform on the division? There's lots of prior work on this question, but I still don't understand what the correct approach in our context would be.

• Ordering comparisons like int_lt, int_le and their unsigned variants are not ported to the DSL yet. Comparisons are an area where the JIT is not super good yet at optimizing away operations. This is a pretty big topic and I've started a project with Nico Rittinghaus to try to improve the situation a bit more generally.

• A more advanced direction of work would be to implement a simplified form of e-graphs (or ae-graphs). The JIT has like half of an e-graph data structure already, and we probably can't afford a full one in terms of compile time costs, but maybe we can have two thirds or something?

Acknowledgements

Thank you to Max Bernstein and Martin Berger for super helpful feedback on drafts of the post!

In this episode of the Python Show Podcast, David Mertz is our guest. David is a prolific writer about the Python programming language. From his extremely popular IPM Developerworks articles to his multiple books on the Python language, David has been a part of the Python community for decades.

We ended up chatting about:

• The history of Python

• Book writing

• Conference speaking

• The PSF

• and more!

Show Links

• David Mertz’s website

• PyDev of the Week: David Mertz on Mouse vs Python

• David Mertz on InformIT / Pearson

Python threading allows you to run parts of your code concurrently, making the code more efficient. However, when you introduce threading to your code without knowing about thread safety, you may run into issues such as race conditions. You solve these with tools like locks, semaphores, events, conditions, and barriers.

By the end of this tutorial, you’ll be able to identify safety issues and prevent them by using the synchronization primitives in Python’s threading module to make your code thread-safe.

In this tutorial, you’ll learn:

• What thread safety is
• What race conditions are and how to avoid them
• How to identify thread safety issues in your code
• What different synchronization primitives exist in the threading module
• How to use synchronization primitives to make your code thread-safe

To get the most out of this tutorial, you’ll need to have basic experience working with multithreaded code using Python’s threading module and ThreadPoolExecutor.

Get Your Code: Click here to download the free sample code that you’ll use to learn about thread safety techniques in Python.

Take the Quiz: Test your knowledge with our interactive “Python Thread Safety: Using a Lock and Other Techniques” quiz. You’ll receive a score upon completion to help you track your learning progress:

Interactive Quiz

Python Thread Safety: Using a Lock and Other Techniques

In this quiz, you'll test your understanding of Python thread safety. You'll revisit the concepts of race conditions, locks, and other synchronization primitives in the threading module. By working through this quiz, you'll reinforce your knowledge about how to make your Python code thread-safe.

Threading in Python

In this section, you’ll get a general overview of how Python handles threading. Before discussing threading in Python, it’s important to revisit two related terms that you may have heard about in this context:

• Concurrency: The ability of a system to handle multiple tasks by allowing their execution to overlap in time but not necessarily happen simultaneously.
• Parallelism: The simultaneous execution of multiple tasks that run at the same time to leverage multiple processing units, typically multiple CPU cores.

Python’s threading is a concurrency framework that allows you to spin up multiple threads that run concurrently, each executing pieces of code. This improves the efficiency and responsiveness of your application. When running multiple threads, the Python interpreter switches between them, handing the control of execution over to each thread.

By running the script below, you can observe the creation of four threads:

Python threading_example.py import threading import time from concurrent.futures import ThreadPoolExecutor def threaded_function(): for number in range(3): print(f"Printing from {threading.current_thread().name}. {number=}") time.sleep(0.1) with ThreadPoolExecutor(max_workers=4, thread_name_prefix="Worker") as executor: for _ in range(4): executor.submit(threaded_function) Copied!

In this example, threaded_function prints the values zero to two that your for loop assigns to the loop variable number. Using a ThreadPoolExecutor, four threads are created to execute the threaded function. ThreadPoolExecutor is configured to run a maximum of four threads concurrently with max_workers=4, and each worker thread is named with a “Worker” prefix, as in thread_name_prefix="Worker".

In print(), the .name attribute on threading.current_thread() is used to get the name of the current thread. This will help you identify which thread is executed each time. A call to sleep() is added inside the threaded function to increase the likelihood of a context switch.

You’ll learn what a context switch is in just a moment. First, run the script and take a look at the output:

Shell $ python threading_example.py Printing from Worker_0. number=0 Printing from Worker_1. number=0 Printing from Worker_2. number=0 Printing from Worker_3. number=0 Printing from Worker_0. number=1 Printing from Worker_2. number=1 Printing from Worker_1. number=1 Printing from Worker_3. number=1 Printing from Worker_0. number=2 Printing from Worker_2. number=2 Printing from Worker_1. number=2 Printing from Worker_3. number=2 Copied!

Each line in the output represents a print() call from a worker thread, identified by Worker_0, Worker_1, Worker_2, and Worker_3. The number that follows the worker thread name shows the current iteration of the loop each thread is executing. Each thread takes turns executing the threaded_function, and the execution happens in a concurrent rather than sequential manner.

For example, after Worker_0 prints number=0, it’s not immediately followed by Worker_0 printing number=1. Instead, you see outputs from Worker_1, Worker_2, and Worker_3 printing number=0 before Worker_0 proceeds to number=1. You’ll notice from these interleaved outputs that multiple threads are running at the same time, taking turns to execute their part of the code.

This happens because the Python interpreter performs a context switch. This means that Python pauses the execution state of the current thread and passes control to another thread. When the context switches, Python saves the current execution state so that it can resume later. By switching the control of execution at specific intervals, multiple threads can execute code concurrently.

You can check the context switch interval of your Python interpreter by typing the following in the REPL:

Python >>> import sys >>> sys.getswitchinterval() 0.005 Copied!

The output of calling the getswitchinterval() is a number in seconds that represents the context switch interval of your Python interpreter. In this case, it’s 0.005 seconds or five milliseconds. You can think of the switch interval as how often the Python interpreter checks if it should switch to another thread.

An interval of five milliseconds doesn’t mean that threads switch exactly every five milliseconds, but rather that the interpreter considers switching to another thread at these intervals.

The switch interval is defined in the Python docs as follows:

Read the full article at https://realpython.com/python-thread-lock/ »

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Компьютеры и сети содействуют нам в борьбе за свободу: они помогают посвятить время и силы важным общественным инициативам, организовывать протесты, защищаться от цензуры.

Но свободны ли наши компьютеры?  И свободны ли мы как пользователи?

Обсудим эти вопросы в Открытом пространстве с Глебом Ерофеевым — активистом движения за свободные программы и волонтёром проекта "ГНУ", который в 1983 году запустил философ и активист Ричард Столлман.

Команда проекта "ГНУ" занимается разработкой свободного софта и техноэтическим активизмом, чтобы дать пользователям контроль над их компьютерами и искоренить несправедливость, которую приносят в общество собственнические программы.

Адрес: Плетешковский пер., 8с1 (м. "Бауманская").

Участие бесплатно.  Приветствуются пожертвования в пользу пространства.

The navigation menu is a crucial element of any website, guiding users through content, enhancing their experience, and playing a vital role in the site's overall usability and success.

Creating a menu that's accessible, responsive, and easy to navigate is a non-trivial task, no matter how simple or complex your navigation is. However, people usually underestimate the efforts required to create a navigation menu that provides a good user experience, and that it can take several iterations to do it right.

In this quiz, you’ll test your understanding of Python Class Constructors.

By working through this quiz, you’ll revisit the internal instantiation process, object initialization using .__init__(), and fine-tuning object creation by overriding .__new__().

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

Web accessibility is crucial for ensuring people of all abilities can engage with digital content without barriers. With the European Accessibility Act approaching, both the public and private sector must comply with its requirements. Learn how!

Drupal needs no introduction! It is a robust content management system (CMS) widely used for developing powerful, scalable, and secure websites. However, like any web platform, optimizing for speed and security is essential to ensure smooth performance, protect data, and improve user experience.Drupal 11 is all about enhanced scalability and security, which makes this version more dynamic and…

The companies are using the newest technologies to improve their e-commerce websites customer service and outreach. With e-commerce websites, you can offer better features to the audience that enhance the overall experience of the visitors on the website. 

And what better CMS of choice than Drupal for your large-scale websites? Drupal offers fresh features for e-commerce businesses that help their customers to have a better experience shopping online. On top of that, there is also a Drupal e-commerce module that allows you to help you engage more with the audience that visits the websites and converts them. 

In this blog, you will learn more about why you should use Drupal For your e-commerce websites and how it can be the best decision for your business. 

Russell wrote a great comment on my last post (thanks!):

What benefits do these things offer when a general purpose computer can do so many things nowadays? Is there a USB keyboard that you can connect to a laptop or phone to do these things? I presume that all recent phones have the compute power to do all the synthesis you need if you have the right software. Is it just a lack of software and infrastructure for doing it on laptops/phones that makes synthesisers still viable?

I've decided to turn my response into a post of its own.

The issue is definitely not compute power. You can indeed attach a USB keyboard to a computer and use a plethora of software synthesisers, including very faithful emulations of all the popular classics. The raw compute power of modern hardware synths is comparatively small: I’ve been told the modern Korg digital synths are on a par with a raspberry pi. I’ve seen some DSPs which are 32 bit ARMs, and other tools which are roughly equivalent to arduinos.

I can think of four reasons hardware synths remain popular with some despite the above:

1. As I touched on in my original synth post, computing dominates my life outside of music already. I really wanted something separate from that to keep mental distance from work.

2. Synths have hard real-time requirements. They don't have raw power in compute terms, but they absolutely have to do their job within microseconds of being instructed to, with no exceptions. Linux still has a long way to go for hard real-time.

3. The Linux audio ecosystem is… complex. Dealing with pipewire, pulseaudio, jack, alsa, oss, and anything else I've forgotten, as well as their failure modes, is too time consuming.

4. The last point is to do with creativity and inspiration. A good synth is more than the sum of its parts: it's an instrument, carefully designed and its components integrated by musically-minded people who have set out to create something to inspire. There are plenty of synths which aren't good instruments, but have loads of features: they’re boxes of "stuff". Good synths can't do it all: they often have limitations which you have to respond to, work around or with, creatively. This was expressed better than I could by Trent Reznor in the video archetype of a synthesiser:

In recent years, a lot has been happening to improve performance, maintainability and tooling of QML. Some of those improvements can only take full effect when your code follows modern best practices. Here are 10 things you can do in order to modernize your QML code and take full advantage of QML’s capabilities.

1. Use qt_add_qml_module CMake API

Qt6 introduced a new CMake API to create QML modules. Not only is this more convenient than what previously had to be done manually, but it is also a prerequisite for being able to exploit most of the following tips.

By using the qt_add_qml_module, your QML code is automatically processed by qmlcachegen, which not only creates QML byte code ahead of time, but also converts parts of your QML code to C++ code, improving performance. How much of your code can be compiled to C++ depends on the quality of the input code. The following tips are all about improving your code in that regard.

add_executable(myapp main.cpp) qt_add_qml_module(myapp URI "org.kde.myapp" QML_FILES Main.qml ) 2. Use declarative type registration

When creating custom types in C++ and registering them with qmlRegisterType and friends, they are not visible to the tooling at the compile time. qmlcachegen doesn’t know which types exist and which properties they have. Hence, it cannot translate to C++ the code that’s using them. Your experience with the QML Language Server will also suffer since it cannot autocomplete types and property names.

To fix this, your types should be registered declaratively using the QML_ELEMENT (and its friends, QML_NAMED_ELEMENT, QML_SINGLETON, etc) macros.

qmlRegisterType("org.kde.myapp", 1, 0, "MyThing");

becomes

class MyThing : public QObject { Q_OBJECT QML_ELEMENT };

The URL and version information are inferred from the qt_add_qml_module call.

3. Declare module dependencies

Sometimes your QML module depends on other modules. This can be due to importing it in the QML code, or more subtly by using types from another module in your QML-exposed C++ code. In the latter case, the dependency needs to be declared in the qt_add_qml_module call.

For example, exposing a QAbstractItemModel subclass to QML adds a dependency to the QtCore (that’s where QAbstractItemModel is registered) to your module. This does not only happen when subclassing a type but also when using it as a parameter type in properties or invokables.

Another example is creating a custom QQuickItem-derived type in C++, which adds a dependency on the Qt Quick module.

To fix this, add the DEPENDENCIES declaration to qt_add_qml_module:

qt_add_qml_module(myapp URI "org.kde.myapp" QML_FILES Main.qml DEPENDENCIES QtCore ) 4. Qualify property types fully

MOC needs types in C++ property definitions to be fully qualified, i.e. include the full namespace, even when inside that namespace. Not doing this will cause issues for the QML tooling.

namespace MyApp { class MyHelper : public QObject { Q_OBJECT }; class MyThing : public QObject { Q_OBJECT QML_ELEMENT Q_PROPERTY(MyHelper *helper READ helper CONSTANT) // bad Q_PROPERTY(MyApp::MyHelper *helper READ helper CONSTANT) // good ... }; } 5. Use types

In order for qmlcachegen to generate efficient code for your bindings, it needs to know the type for properties. Avoid using ‘property var’ wherever possible and use concrete types. This may be built-in types like int, double, or string, or any declaratively-defined custom type. Sometimes you want to be able to use a type as a property type in QML but don’t want the type to be creatable from QML directly. For this, you can register them using the QML_UNCREATABLE macro.

property var size: 10 // bad property int size: 10 // good property var thing // bad property MyThing thing // good 6. Avoid parent and other generic properties

qmlcachegen can only work with the property types it knows at compile time. It cannot make any assumptions about which concrete subtype a property will hold at runtime. This means that, if a property is defined with type Item, it can only compile bindings using properties defined on Item, not any of its subtypes. This is particularly relevant for properties like ‘parent’ or ‘contentItem’. For this reason, avoid using properties like these to look up items when not using properties defined on Item (properties like width, height, or visible are okay) and use look-ups via IDs instead.

Item { id: thing property int size: 10 Rectangle { width: parent.size // bad, Item has no 'size' property height: thing.height // good, lookup via id color: parent.enabled ? "red" : "black" // good, Item has 'enabled' property } } 7. Annotate function parameters with types

In order for qmlcachegen to compile JavaScript functions, it needs to know the function’s parameter and return type. For that, you need to add type annotations to the function:

function calculateArea(width: double, height: double) : double { return width * height }

When using signal handlers with parameters, you should explicitly specify the signal parameters by supplying a JS function or an arrow expression:

MouseArea { onClicked: event => console.log("clicked at", event.x, event.y) }

Not only does this make qmlcachegen happy, it also makes your code far more readable.

8. Use qualified property lookup

QML allows you to access properties from objects several times up in the parent hierarchy without explicitly specifying which object is being referenced. This is called an unqualified property look-up and generally considered bad practice since it leads to brittle and hard to reason about code. qmlcachegen also cannot properly reason about such code. So, it cannot properly compile it. You should only use qualified property lookups

Item { id: root property int size: 10 Rectangle { width: size // bad, unqualified lookup height: root.size // good, qualified lookup } }

Another area that needs attention is accessing model roles in a delegate. Views like ListView inject their model data as properties into the context of the delegate where they can be accessed with expressions like ‘foo’, ‘model.foo’, or ‘modelData.foo’. This way, qmlcachegen has no information about the types of the roles and cannot do its job properly. To fix this, you should use required properties to fetch the model data:

ListView { model: MyModel delegate: ItemDelegate { text: name // bad, lookup from context icon.name: model.iconName // more readable, but still bad required property bool active // good, using required property checked: active } } 9. Use pragma ComponentBehavior: Bound

When defining components, either explicitly via Component {} or implicitly when using delegates, it is common to want to refer to IDs outside of that component, and this generally works. However, theoretically any component can be used outside of the context it is defined in and, when doing that, IDs might refer to another object entirely. For this reason, qmlcachegen cannot properly compile such code.

To address this, we need to learn about pragma ComponentBehavior. Pragmas are file-wide switches that influence the behavior of QML. By specifying pragma ComponentBehavior: Bound at the top of the QML file, we can bind any components defined in this file to their surroundings. As a result, we cannot use the component in another place anymore but can now safely access IDs outside of it.

pragma ComponentBehavior: Bound import QtQuick Item { id: root property int delegateHeight: 10 ListView { model: MyModel delegate: Rectangle { height: root.delegateHeight // good with ComponentBehavior: Bound, bad otherwise } } }

A side effect of this is that accessing model data now must happen using required properties, as described in the previous point. Learn more about ComponentBehavior here.

10. Know your tools

A lot of these pitfalls are not obvious, even to seasoned QML programmers, especially when working with existing codebases. Fortunately, qmllint helps you find most of these issues and avoids introducing them. By using the QML Language Server, you can incorporate qmllint directly into your preferred IDE/editor such as Kate or Visual Studio Code.

While qmlcachegen can help boost your QML application’s performance, there are performance problems it cannot help with, such as scenes that are too complex, slow C++ code, or inefficient rendering. To investigate such problems, tools like the QML profiler, Hotspot for CPU profiling, Heaptrack for memory profiling, and GammaRay for analyzing QML scenes are very helpful.

About KDAB

If you like this article and want to read similar material, consider subscribing via our RSS feed.

Subscribe to KDAB TV for similar informative short video content.

KDAB provides market leading software consulting and development services and training in Qt, C++ and 3D/OpenGL. Contact us.

The post 10 Tips to Make Your QML Code Faster and More Maintainable appeared first on KDAB.

In the last weeks qmpbackup has seen a bit more improvements.

• Adds support for CEPH/RBD backed devices.
• Allows to use unique bitmaps for having multiple, separate backup chains.
• Adds support for jsonified filename configurations like often used on proxmox systems.
• Adds support for saving attached pflash/nvram devices (storing UEFI related settings)
• qmprestore can now merge the backup chain into a new image file and the new snapshotrebase command can rebase the images and after committing, creates an internal qcow snapshot, so one can easily switch between different vm states in the backup.

Ive been running it lately to backup Virtual machines on proxmox systems, where the proxmox backup server is not an option.

BOSTON (October 22, 2024) -- The Free Software Foundation (FSF) has announced today that it is working on a statement of criteria for free machine learning applications, which will require the software, as well as the raw training data and associated scripts, to grant users the four freedoms.

#652 – OCTOBER 22, 2024
View in Browser »

Structural Pattern Matching in Python

In this tutorial, you’ll learn how to harness the power of structural pattern matching in Python. You’ll explore the new syntax, delve into various pattern types, and find appropriate applications for pattern matching, all while identifying common pitfalls.
REAL PYTHON

Combinatoric Iterators From itertools

The itertools module offers four combinatoric iterators that generate different combined outputs from one or more iterable. This post covers all of them: product, permutations, combinations, and combinations_with_replacement.
JUHA-MATTI SANTALA

Scrape Web Data at Scale with a 98.7% Success Rate

Extract all the data you need from any website without getting blocked with ZenRows’ Scraper API – a complete toolkit with premium proxies, anti-CAPTCHA, cloud-based scalable browsers, and more. Start your free trial now →
ZENROWS sponsor

CPython Internals: Your Guide to the Python 3 Interpreter

Unlock the inner workings of the Python language, compile the Python interpreter from source code, and participate in the development of CPython. Guido van Rossum, the creator of Python, says: “I can recommend CPython Internals to anyone who wants to get going with hacking on CPython” →
ANTHONY SHAW sponsor

SSH Scripting With Fabric and Python

Reading and writing files is a basic task that most software applications need to do, but what if you need to do that on remote machines? This tutorial introduces you to Fabric and how to connect over SSH in Python.
MIKE DRISCOLL

Python 3.14.0 Alpha 1 Released

CPYTHON DEV BLOG

Quiz: Structural Pattern Matching

REAL PYTHON

Quiz: Iterators and Iterables in Python

REAL PYTHON

Quiz: Python import: Advanced Techniques and Tips

REAL PYTHON

Articles & Tutorials Mind Your Image Metadata

Most devices record a variety of metadata when generating images. While some of that information may be innocuous, you could end up exposing the GPS coordinates to your home if you aren’t careful. In this article, Stefanie provides a brief introduction to image metadata, and then shows you how to remove it with exif-stripper.
STEFANIEMOLIN.COM • Shared by Stefanie Molin

Open Source in Python and JavaScript 2024

Python vs. JavaScript: Which open-source community is leading the way? This analysis of 36,000 GitHub repositories explores the evolution of Python and JavaScript ecosystems, highlighting key trends and popular topics. Discover how open-source communities of Python and JavaScript have shaped the tech landscape.
PYCHALLENGER.COM • Shared by Erik Nogueira Kückelheim

Accelerate Edge Devices with High-Performance AI Power

Experience the power of Edge AI—delivering lightning-fast, real-time processing where it matters. Optimize your applications to push performance and accuracy beyond limits with Intel’s OpenVINO toolkit.
INTEL CORPORATION sponsor

Using Type Hints for Multiple Return Types in Python

In this video course, you’ll learn how to define multiple return types using type hints in Python. This course covers working with single or multiple pieces of data, defining type aliases, and performing type checking using a third-party static type checker tool.
REAL PYTHON course

Narwhals: Expanding DataFrame Compatibility

How does a Python tool support all types of DataFrames and their various features? Could a lightweight library be used to add compatibility for newer formats like Polars or PyArrow? This week on the show, we speak with Marco Gorelli about his project, Narwhals.
REAL PYTHON podcast

Syntactic Sugar: Why Python Is Sweet and Pythonic

In this tutorial, you’ll learn what syntactic sugar is and how Python uses it to help you create more readable, descriptive, clean, and Pythonic code. You’ll also learn how to replace a given piece of syntactic sugar with another syntax construct.
REAL PYTHON

Entering Text in the Terminal Is Complicated

Julia asked some folks on Mastodon what they found confusing about working in a terminal. It turns out that entering text in the terminal is complicated. This post talks about why that is and how to understand it better.
JULIA EVANS

4 Lessons From Small Teams That Ship Fast

Software engineering provides a lot of leverage and small teams can do a large amount of work. This post talks about several common examples in the industry where a small group created a big product.
LEONARDO CREED

Perks of Being a Python Core Developer

Mariatta has been a Python Core Developer since 2017. If you want to know just what that means, this post talks about all the things she gets to do.
MARIATTA

Custom Dictionary Types in Pydantic

Pydantic lets you create custom types. This post talks about how to create a custom dictionary type using root models and Enums.
BRYAN ANTHONIO

How to Use Lambda Functions in Python

This article looks at some examples and best practices when using Lambda functions in Python.
FEDERICO TROTTA • Shared by AppSignal

Projects & Code ryp: R Inside Python

GITHUB.COM/WAINBERG

pyglove: Symbolic OO for Python

GITHUB.COM/GOOGLE

pipreqs: Generate requirements.txt Based on Imports

GITHUB.COM/BNDR

Bowler: Safe Code Refactoring for Modern Python

GITHUB.COM/FACEBOOKINCUBATOR

nanodjango: Full Django in a Single File

GITHUB.COM/RADIAC

Events Weekly Real Python Office Hours Q&A (Virtual)

October 23, 2024
REALPYTHON.COM

October Oslo Python Meetup

October 24, 2024
MEETUP.COM

PyCon APAC 2024

October 25 to October 27, 2024
PYCON.ID

PyCon Korea 2024

October 25 to October 28, 2024
PYCON.KR

PythonHo Conference 2024

October 26 to October 28, 2024
PYTHONHO.COM

PythOnRio Meetup

October 26, 2024
PYTHON.ORG.BR

Django Girls Aba

October 27, 2024
DJANGOGIRLS.ORG

PyCon FR 2024

October 31 to November 3, 2024
PYCON.FR

PyCon Zimbabwe

October 31 to November 3, 2024
PYCON.ORG

Happy Pythoning!
This was PyCoder’s Weekly Issue #652.
View in Browser »

[ Subscribe to 🐍 PyCoder’s Weekly 💌 – Get the best Python news, articles, and tutorials delivered to your inbox once a week >> Click here to learn more ]

The Python Global Interpreter Lock or GIL, in simple words, is a mutex (or a lock) that allows only one thread to hold the control of the Python interpreter.

This means that only one thread can be in a state of execution at any point in time. The impact of the GIL isn’t visible to developers who execute single-threaded programs, but it can be a performance bottleneck in CPU-bound and multi-threaded code.

Since the GIL allows only one thread to execute at a time even in a multi-threaded architecture with more than one CPU core, the GIL has gained a reputation as an “infamous” feature of Python.

In this video course you’ll learn how the GIL affects the performance of your Python programs, and how you can mitigate the impact it might have on your code.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

In this quiz, you’ll test your understanding of how to define your own Python function.

You’ll revisit theoretical knowledge about passing values to functions, when to divide your program into separate user-defined functions, and all the tools you’ll need to define complex and powerful functions in Python.

[ Improve Your Python With 🐍 Python Tricks 💌 – Get a short & sweet Python Trick delivered to your inbox every couple of days. >> Click here to learn more and see examples ]

With PHP attributes, your Drupal development process can be simplified, more performant, and easier to read. Learn the difference between PHP Attributes and Annotations and discover why attributes are the future of efficient coding.

Takeaway: With Drupal 7's end-of-life date set for January 5, 2025, public sector institutions face critical decisions about their web infrastructure. While migration to Drupal 10 offers clear advantages, organizations unable to migrate immediately need robust security measures to protect their sites.

PythonAnywhere has been around for over 10 years, and as our platform continues to grow with thousands of users, we’re committed to keeping it in top shape. Part of this involves upgrading some of the older parts of our infrastructure, with a special focus on our file storage servers—some of the oldest systems we have.