FLOSS Project Planets

Here’s my weekly report for week 38 of 2017. This week has not been a great week as I saw my primary development machine die in a spectacular reboot loop. Thanks to the wonderful community around Debian and free software (that if you’re reading this, you’re probably part of), I should be back up to speed soon. A replacement workstation is currently moving towards me and I’ve received a number of smaller donations that will go towards video converters and upgrades to get me back to full productivity.

While the Drupalcon webseite has a good few pointers to the well-known major tourist attractions, as locals we'd like to share our knowledge about some of our favourite places with you! So here a few recommendations:

Viennese Wine and Heurige

If you stay for the weekend after the Con, you can join the Vienna Wine Hiking day, which I can highly recommend. There are 3 possible easy hikes through the vineyards with lots of options to stop for tasting gorgeous wine directly from the producers. Furthermore you may enjoy great views of the city even if the wheather is not that great!

If you stay long enough, don't miss it! You can find details and options at https://www.wien.info/en/shopping-wining-dining/wine/wine-trail

If you cannot join the wine hiking day, be sure to visit some Viennese "Heurige" (wine taverns). Good options would be the Schreiberhaus or a little bit closer to the city-center Sissy-Huber.

Otto Wagner Buildings

The famous Viennese Jugendstil architect Otto Wagner (and friends) has left lots of traces back in the city. Apart from some of the subway stations (you won't be able to miss them) we'd recommend looking at the following buildings at least from the outside:

• Just next to street: https://www.google.at/maps/place/Linke+Wienzeile+40,+1060+Wien/@48.1974274,16.3569926,17z/data=!3m1!4b1!4m5!3m4!1s0x476d0785e63c4893:0x9a9933c97d25bef3!8m2!3d48.1974274!4d16.3591813
• The Secession
• Stadtbahn-Pavillons
• Postsparkasse
• If you like Otto Wagner, we can highly recommend checking out the church at Steinhof as well: https://www.google.at/maps/place/Kirche+am+Steinhof/@48.2106261,16.2765585,17z/data=!3m1!4b1!4m5!3m4!1s0x476d082f7c463e0f:0xe1260d0ac6ac1779!8m2!3d48.2106225!4d16.2787525

Cafés & Restaurants

Kaffee Alt Wien: An interesting mixture between a traditional Vienese Cafe and a "Beisl" (pub). The food can be recommended too, simple but authentice Viennese dishes, like Gulasch, Schnitzel and a variety of sausages. Although the Kaffee Alt Wien is mentioned in travel guides, it has not lost its athmosphere and is visited by tourists and locals alike.

Flatchers: Great steaks for a reasonable price. There are two restaurants in the same street: A French bistro with georgous French athmosphere and a larger one in American style.

Brunnenmarkt: A local market in one of the lesser known districts, lots of immigrants of south-eastern Europe and Turkey run market booths and Cafés around a nice plaza. You'll find great athmosphere and good food options: Kent, Cafe Ando, Cay Cafe am Yppenplatz

Barfly's: A cuban style cocktail bar with authentic athmosphere and music!

 

In the summer 2017 edition of 2600 magazine there is a brilliant article on running onion services as part of a series on censorship resistant services. Onion services provide privacy and security for readers above that which is possible through the use of HTTPS. Since moving my website to Netlify, my onion service died as Netlify doesn’t provide automatic onion services (although they do offer automated Let’s Encrypt certificate provisioning). If anyone from Netlify is reading this, please consider adding a one-click onion service button next to the Let’s Encrypt button.

Over the years Drupal distributions, or distros as they're more affectionately known, have evolved a lot. We started off passing around database dumps. Eventually we moved onto using installations profiles and features to share par-baked sites.

There are some signs that distros aren't working for people using them. Agencies often hack a distro to meet client requirements. This happens because it is often difficult to cleanly extend a distro. A content type might need extra fields or the logic in an alter hook may not be desired. This makes it difficult to maintain sites built on distros. Other times maintainers abandon their distributions. This leaves site owners with an unexpected maintenance burden.

We should recognise how people are using distros and try to cater to them better. My observations suggest there are 2 types of Drupal distributions; starter kits and targeted products.

Targeted products are easier to deal with. Increasingly monetising targeted distro products is done through a SaaS offering. The revenue can funds the ongoing development of the product. This can help ensure the project remains sustainable. There are signs that this is a viable way of building Drupal 8 based products. We should be encouraging companies to embrace a strategy built around open SaaS. Open Social is a great example of this approach. Releasing the distros demonstrates a commitment to the business model. Often the secret sauce isn't in the code, it is the team and services built around the product.

Many Drupal 7 based distros struggled to articulate their use case. It was difficult to know if they were a product, a demo or a community project that you extend. Open Atrium and Commerce Kickstart are examples of distros with an identity crisis. We need to reconceptualise most distros as "starter kits" or as I like to call them "puppies".

Why puppies? Once you take a puppy home it becomes your responsibility. Starter kits should be the same. You should never assume that a starter kit will offer an upgrade path from one release to the next. When you install a starter kit you are responsible for updating the modules yourself. You need to keep track of security releases. If your puppy leaves a mess on the carpet, no one else will clean it up.

Sites build on top of a starter kit should diverge from the original version. This shouldn't only be an expectation, it should be encouraged. Installing a starter kit is the starting point of building a unique fork.

Project pages should clearly state that users are buying a puppy. Prospective puppy owners should know if they're about to take home a little lap dog or one that will grow to the size of a pony that needs daily exercise. Puppy breeders (developers) should not feel compelled to do anything once releasing the puppy. That said, most users would like some documentation.

I know of several agencies and large organisations that are making use of starter kits. Let's support people who are adopting this approach. As a community we should acknowledge that distros aren't working. We should start working out how best to manage the transition to puppies.

We are moving forward with the Camel in Action 2nd edition book. Manning just informed us that they have moved the book into production phase.

All chapters and appendixes except for Ch21 have now completed the copyedit and review cycle and have moved into the proofreading and indexing stages of production.

The proofing is done by a professional, whom has not seen the material before (new fresh set of eyes). Once the proofreading and indexing stages are complete, the final Word manuscript chapters will be sent on to the typesetter for layout, along with prepped graphics from the illustrator.

When we react this stage Jonathan and myself have chances to review PDFs of the formatted book pages, to ensure all the code examples are formatted correctly and so on.

At present time Jonathan and I are now working on the front matter, and writing our preface and what else. We are also reaching out to the foreword writers to ensure they hand in their material (Gregor Hohpe and James Strachan are yet to return their forewords).

And when Henryk Konsek has provided his updates, then chapter 21 can move along and also be part of the book.

So it starts to look really good and we can start to see the finish line. Its been a long run this time which has taken us almost double the time to complete the 2nd edition than what we did for the 1st edition. But that is possible expected, we are getting older, and the book is also almost twice the size.

To celebrate this, Manning has the book as deal of the day. This means you get 50% discount if you order the book on September 24th using the couple code: dotd092417au

The book can be ordered from Manning website at: https://www.manning.com/books/camel-in-action-second-edition

A little more than a month ago I wrote how to observe the SIM card ID (aka IMSI number) of mobile phones talking to nearby mobile phone base stations using Debian GNU/Linux and a cheap USB software defined radio, and thus being able to pinpoint the location of people and equipment (like cars and trains) with an accuracy of a few kilometer. Since then we have worked to make the procedure even simpler, and it is now possible to do this without any manual frequency tuning and without building your own packages.

The gr-gsm package is now included in Debian testing and unstable, and the IMSI-catcher code no longer require root access to fetch and decode the GSM data collected using gr-gsm.

Here is an updated recipe, using packages built by Debian and a git clone of two python scripts:

1. Start with a Debian machine running the Buster version (aka testing).
2. Run 'apt install gr-gsm python-numpy python-scipy python-scapy' as root to install required packages.
3. Fetch the code decoding GSM packages using 'git clone github.com/Oros42/IMSI-catcher.git'.
4. Insert USB software defined radio supported by GNU Radio.
5. Enter the IMSI-catcher directory and run 'python scan-and-livemon' to locate the frequency of nearby base stations and start listening for GSM packages on one of them.
6. Enter the IMSI-catcher directory and run 'python simple_IMSI-catcher.py' to display the collected information.

Note, due to a bug somewhere the scan-and-livemon program (actually its underlying program grgsm_scanner) do not work with the HackRF radio. It do work with RTL 8232 and other similar USB radio receivers you can get very cheaply (for example from ebay), so for now the solution is to scan using the RTL radio and only use HackRF for fetching GSM data.

As far as I can tell, a cell phone only show up on one of the frequencies at the time, so if you are going to track and count every cell phone around you, you need to listen to all the frequencies used. To listen to several frequencies, use the --numrecv argument to scan-and-livemon to use several receivers. Further, I am not sure if phones using 3G or 4G will show as talking GSM to base stations, so this approach might not see all phones around you. I typically see 0-400 IMSI numbers an hour when looking around where I live.

I've tried to run the scanner on a Raspberry Pi 2 and 3 running Debian Buster, but the grgsm_livemon_headless process seem to be too CPU intensive to keep up. When GNU Radio print 'O' to stdout, I am told there it is caused by a buffer overflow between the radio and GNU Radio, caused by the program being unable to read the GSM data fast enough. If you see a stream of 'O's from the terminal where you started scan-and-livemon, you need a give the process more CPU power. Perhaps someone are able to optimize the code to a point where it become possible to set up RPi3 based GSM sniffers? I tried using Raspbian instead of Debian, but there seem to be something wrong with GNU Radio on raspbian, causing glibc to abort().

This work is supported by Anaconda Inc. and the Data Driven Discovery Initiative from the Moore Foundation.

I’m pleased to announce the release of Dask version 0.15.3. This release contains stability enhancements and bug fixes. This blogpost outlines notable changes since the 0.15.2 release on August 30th.

You can conda install Dask:

conda install -c conda-forge dask

or pip install from PyPI

pip install dask[complete] --upgrade

Conda packages are available both on conda-forge channels. They will be on defaults in a few days.

Full changelogs are available here:

• dask/dask
• dask/distributed

Some notable changes follow.

Masked Arrays

Dask.array now supports masked arrays similar to NumPy.

In [1]: import dask.array as da In [2]: x = da.arange(10, chunks=(5,)) In [3]: mask = x % 2 == 0 In [4]: m = da.ma.masked_array(x, mask) In [5]: m Out[5]: dask.array<masked_array, shape=(10,), dtype=int64, chunksize=(5,)> In [6]: m.compute() Out[6]: masked_array(data = [-- 1 -- 3 -- 5 -- 7 -- 9], mask = [ True False True False True False True False True False], fill_value = 999999)

This work was primarily done by Jim Crist and partially funded by the UK Met office in support of the Iris project.

Constants in atop

Dask.array experts will be familiar with the atop function, which powers a non-trivial amount of dask.array and is commonly used by people building custom algorithms. This function now supports constants when the index given is None.

atop(func, 'ijk', x, 'ik', y, 'kj', CONSTANT, None) Memory management for workers

Dask workers spill excess data to disk when they reach 60% of their alloted memory limit. Previously we only measured memory use by adding up the memory use of every piece of data produce by the worker. This could fail under a few situations

1. Our per-data estiamtes were faulty
2. User code consumed a large amount of memory without our tracking it

To compensate we now also periodically check the memory use of the worker using system utilities with the psutil module. We dump data to disk if the process rises about 70% use, stop running new tasks if it rises above 80%, and restart the worker if it rises above 95% (assuming that the worker has a nanny process).

Breaking Change: Previously the --memory-limit keyword to the dask-worker process specified the 60% “start pushing to disk” limit. So if you had 100GB of RAM then you previously might have started a dask-worker as follows:

dask-worker ... --memory-limit 60e9 # before specify 60% target

And the worker would start pushing to disk once it had 60GB of data in memory. However, now we are changing this meaning to be the full amount of memory given to the process.

dask-worker ... --memory-limit 100e9A # now specify 100% target

Of course, you don’t have to sepcify this limit (many don’t). It will be chosen for you automatically. If you’ve never cared about this then you shouldn’t start caring now.

More about memory management here: http://distributed.readthedocs.io/en/latest/worker.html?highlight=memory-limit#memory-management

Statistical Profiling

Workers now poll their worker threads every 10ms and keep a running count of which functions are being used. This information is available on the diagnostic dashboard as a new “Profile” page. It provides information that is orthogonal, and generally more detailed than the typical task-stream plot.

These plots are available on each worker, and an aggregated view is available on the scheduler. The timeseries on the bottom allows you to select time windows of your computation to restrict the parallel profile.

More information about diagnosing performance available here: http://distributed.readthedocs.io/en/latest/diagnosing-performance.html

Acknowledgements

The following people contributed to the dask/dask repository since the 0.15.2 release on August 30th

• Adonis
• Christopher Prohm
• Danilo Horta
• jakirkham
• Jim Crist
• Jon Mease
• jschendel
• Keisuke Fujii
• Martin Durant
• Matthew Rocklin
• Tom Augspurger
• Will Warner

The following people contributed to the dask/distributed repository since the 1.18.3 release on September 2nd:

• Casey Law
• Edrian Irizarry
• Matthew Rocklin
• rbubley
• Tom Augspurger
• ywangd

These are my notes on using some MicroPython specific tools in relation to a ESP32-DevKitC board.

There are many tutorials and youtube videos that constantly encourage users to install tools and packages into their system-level libraries. (If you need to use sudo when you pip install foo, you are installing it as a system level library.) Please, Please, Please take the time to learn the basics of virtual environment. If you are a developer/hacker/maker – save yourself lots of frustration by using virtual environments.

A virtual environment is an isolated Python environment it contains all the necessary executables to use the packages that a Python project would need. It allows you use the desired version of a specific library and isolates that library from other virtual environments and the system and user level libraries. It allows you to easily define what packages are required to reproduce your work.

A short primer based on Python3.

To Create and setup a Python3 virtualenv

# nav to where you want to create your project then ... $ python3 -m venv your-project-name $ cd your-project-name $ source bin/activate (your-project-name) $ pip install --upgrade pip

Now just use pip and skip the sudo and kludging up your system level install.

To Deactivate a virtual environment

(your-project-name) $ deactivate $

to activate an existing virtual environment

$ source bin/activate (your-project-name) $

Mandatory link to the documentation.

Now you can list the packages you have installed for THIS project with pip:

(your-project-name) $ pip freeze pkg-resources==0.0.0 (your-project-name) $ pip list pip (9.0.1) pkg-resources (0.0.0) setuptools (20.7.0)

May the Zen of Python be with you.

These are the notes of a training course on systemd I gave as part of my work with Truelite.

Writing .unit files

For reference, the global index with all .unit file directives is at man systemd.directives.

All unit files have a [Unit] section with documentation and dependencies. See man systemd.unit for documentation.

It is worth having a look at existing units to see what they are like. Use systemctl --all -t unittype for a list, and systemctl cat unitname to see its content wherever it is installed.

For example: systemctl cat graphical.target. Note that systemctl cat adds a line of comment at the top so one can see where the unit file is installed.

Most unit files also have an [Install] section (also documented in man systemd.unit) that controls what happens when enabling or disabling the unit.

See also:

• Systemd unit dependencies and order
• Understanding systemd units and unit files

.target units

.target units only contain [Unit] and [Install] sections, and can be used to give a name to a given set of dependencies.

For example, one could create a remote-maintenance.target unit, that when brought up activates, via dependencies, a set of services, mounts, network sockets, and so on.

See man systemd.target

See systemctl --all -t target for examples.

special units

man systemd.special has a list of units names that have a standard use associated to them.

For example, ctrl-alt-del.target is a unit that is started whenever Control+Alt+Del is pressed on the console. By default it is symlinked to reboot.target, and you can provide your own version in /etc/systemd/system/ to perform another action when Control+Alt+Del is pressed.

User units

systemd can also be used to manage services on a user session, starting them at login and stopping them at logout.

Add --user to the normal systemd commands to have them work with the current user's session instead of the general system.

See systemd/User in the Arch Wiki for a good description of what it can do.

We live in a hobby-rich world. There is no shortage of pastimes to grow a passion for. There is a shortage of one thing: time to indulge those passions. If you're someone who pours your heart into that one thing that makes your life worthwhile, that's a great deal. But, what if you've got no shortage of interests that draw your attention and you realize you will never have the time for all of them?

If I look at all the things I'd love to do with my life as a rose bush I'm tending, I realize that careful pruning is essential for the best outcome. This is a hard lesson to learn, because it can mean cutting beautiful flowers and watching the petals fall to the ground to wither. It has to be done.

I have a full time job that takes a lot of my mental energy. I have a wife and a son and family time is very important in my house. I try to read more, and I want to keep up with new developments in my career, and I'm trying to make time for simple, intentional relaxing to lower my anxiety and stress. That doesn't leave a lot of room to pursue any of these hobbies.

I used to play the guitar, if only a bit.

I've always had my eye on becoming a writer.

Software development began as a passion hobby, and now that it is a carrier I still feel that draw to it outside of work.

A lot of my life was spent under the assumption that I would end up in some career as an artist, and I was even on a trajectory towards art school in my teens.

But there aren't enough days in the year, or hours in any of those days, to scratch 100% of those itches.

So, I'm committing to saying "No" to myself more often. When I'm looking for a small app or tool and can't find just the right thing, I'm going to say "No" to building my own, instead of making the best option work. When NaNoWriMo rolls around next year, I'm not going to cause myself anxiety over the "Will I? Won't I?" leading up, and I'm going to admit that it just doesn't work for me. When I end my work day, I'm going to leave the web development at work.

I will be saying "No" to myself on all these interests so I can direct my "Yes" whole heartedly to one: my blossoming foray into game development. And this is a really deliberate choice! Game development is what got me into computers and into programming. But, its also something multi-faceted in a way that few other pursuits are. By throwing myself fully into my game projects, I'll be able to spend time created art, to code outside of work and learn new techniques and paradigms, and to tell stories.

I'm putting down a lot of interests, and shelving a lot of personal projects. I have dozens of bits of code that'll only collect dust from now on, even though I think of them often and constantly feel the pull to hack on them in the evening or weekends. But, I have convinced myself this is for the best. I'm working on making 2017 a big year for me, and I can't do that when I'm pulled in a thousand directions.

Learning to give up just may be the ticket to finally succeeded.

These are the ten most rated questions at Stack Overflow last week.
Between brackets: [question score / answers count]
Build date: 2017-09-23 19:35:45 GMT

1. Why can't I swap two items in a list in one line? - [9/4]
2. How to remove all the values in a string except for the chosen ones - [8/6]
3. Python Sort a list based on a given order - [8/6]
4. python - iterating list of dictionaries and unpacking - [7/4]
5. Why is the dtype shown (even if it's the native one) when using floor division with NumPy? - [7/2]
6. How to assign count of unique values to the records in a data frame in python - [6/5]
7. Trouble converting string to float in python - [6/3]
8. How to test that a custom excepthook is installed correctly? - [6/2]
9. How to make a list of integers that is the sum of all the integers from a set of lists in a dict? - [6/2]
10. python rstrip or remove end of string by a pattern of charachters - [6/2]

A new version of the RcppCNPy package arrived on CRAN yesterday.

RcppCNPy provides R with read and write access to NumPy files thanks to the cnpy library by Carl Rogers.

This version updates internals for function registration, but otherwise mostly switches the vignette over to the shiny new pinp two-page template and package.

Changes in version 0.2.7 (2017-09-22)

• Vignette updated to Rmd and use of pinp package

• File src/init.c added for dynamic registration

CRANberries also provides a diffstat report for the latest release. As always, feedback is welcome and the best place to start a discussion may be the GitHub issue tickets page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

A bug-fix release RcppClassic 0.9.8 for the very recent 0.9.7 release which fixes a build issue on macOS introduced in 0.9.7. No other changes.

Courtesy of CRANberries, there are changes relative to the previous release.

Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

I gave a talk at Boston Python the other night. It started as an exposition of the point matching algorithm I've previously written about on this blog. But as I thought about my side project more, I was interested to talk about the challenges I faced while building it. Not because the specifics were so interesting, but because they were typical problems that all software projects face.

And in particular, I wanted to underscore this point: software is hard, even for experts. Experts have the same struggles that beginners do.

I used this tweet as an illustration:

I love the raw emotion on the two boys' faces. They perfectly illustrate both the frustration and exhilaration of writing software.

But here's what beginners might not understand: beginners think beginners feel the frustration, and experts feel the exhilaration. As any expert will tell you, experts feel plenty of frustration. They feel like that left-hand kid a lot.

The difference between beginners and experts is that experts are familiar with that frustration. They encounter it all the time, as they deal with new unfamiliar technologies, or a thorny bug, or just when they are too tired to tackle the problem before them. They know the frustration is because they are facing a challenging problem. They know it isn't a reflection of their self-worth or abilities. Experts know the feeling will pass, and they have techniques for dealing with the frustration.

When beginners get frustrated, they can start to worry that they are not cut out for software, or they are dumb, or that everyone else gets it and they don't.

The good news for beginners is: this isn't about you. Software is difficult. We build layer upon layer of leaky abstractions, of higher and higher realms of virtualization and complexity. There's no natural limit to how high our towers of complexity can go, so we are forced to try to understand them, or tame them, and it's very hard. Our languages and tools are obscure and different and new ones are invented all the time. Frustration is inevitable.

The bad news for beginners is: this feeling won't stop. If you do your software career right, then you will always be a newb at something. Sure, you can master a specialty, and stick with it, but that can get boring. And that specialty might dwindle away leaving you stranded.

You will be learning new things forever. That feeling of frustration is you learning a new thing. Get used to it.

This article describes the MU* development system Evennia using pictures!  

This article was originally written for Optional Realities. Since it is no longer available to read on OR, I'm reposting it in full here.
Figure 1: The parts of the Evennia library

Evennia is a game development library. What you see in Figure 1 is the part you download from us. This will not run on its own, we will soon initialize the missing “jigsaw puzzle” piece on the left. But first let’s look at what we’ve got.
Looking at Figure 1 you will notice that Evennia internally has two components, the Portal and the Server. These will run as separate processes.
The Portal tracks all connections to the outside world and understands Telnet protocols, websockets, SSH and so on. It knows nothing about the database or the game state. Data sent between the Portal and the Server is protocol-agnostic, meaning the Server sends/receives the same data regardless of how the user is connected. Hiding behind the Portal also means that the Server can be completely rebooted without anyone getting disconnected.
The Server is the main “mud driver” and handles everything related to the game world and its database. It's asynchronous and uses Twisted. In the same process of the Server is also the Evennia web server component that serves the game’s website. That the Server and webserver are accessing the database in the same process allows for a consistent game state without any concerns for caching or race condition issues.
Now, let’s get a game going. We’ll call it mygame. Original, isn’t it?
Figure 2: The full setup for mygame
After installing evennia you will have the evennia command available. Using this you create a game directory - the darker grey piece in Figure 2 that was missing previously. This is where you will create your dream game!
During initialization, Evennia will create Python module templates in mygame/ and link up all configurations to make mygame a fully functioning, if empty, game, ready to start extending. Two more commands will create your database and then start the server. From this point on, mygame is up and running and you can connect to your new game with telnet on localhost:4000 or by pointing your browser to http://localhost:4001.
Now, our new mygame world needs Characters, locations, items and more! These we commonly refer to as game entities. Let’s see how Evennia handles those.
Figure 3: The Database Abstraction of Evennia entities
Evennia is fully persistent and abstracts its database in Python using Django. The database tables are few and generic, each represented by a single Python class. As seen in Figure 3, the example ObjectDB Python class represents one database table. The properties on the class are the columns (fields) of the table. Each row is an instance of the class (one entity in the game).
Among the example columns shown is the key (name) of the ObjectDB entity as well as a Foreign key-relationship for its current “location”. From the above we can see that Trigger is in the Dungeon, carrying his trusty crossbow Old Betsy!
The db_typeclass_path is an important field. This is a python-style path and tells Evennia which subclass of ObjectDB is actually representing this entity.
Figure 4: Inheriting classes to customize entities

In Figure 4 we see the (somewhat simplified) Python class inheritance tree that you as an Evennia developer will see, along with the three instanced entities.
ObjectDB represents stuff you will actually see in-game and its child classes implement all the handlers, helper code and the hook methods that Evennia makes use of. In your mygame/ folder you just import these and overload the things you want to modify. In this way, the Crossbow is modified to do the stuff only crossbows can do and CastleRoom adds whatever it is that is special about rooms in the castle.
When creating a new entity in-game, a new row will automatically be created in the database table and then “Trigger” will appear in-game! If we, in code, search the database for Trigger, we will get an instance of the Character class back - a Python object we can work with normally.
Looking at this you may think that you will be making a lot of classes for every different object in the game. Your exact layout is up to you but Evennia also offers other ways to customize each individual object, as exemplified by Figure 5.
Figure 5: Adding persistent Attributes to a game entity.

The Attribute is another class directly tied to the database behind the scenes. Each Attribute basically has a key, a value and a ForeignKey relation to another ObjectDB. An Attribute serializes Python constructs into the database, meaning you can store basically any valid Python, like the dictionary of skills seen in Figure 5. The “strength” and “skills” Attributes will henceforth be reachable directly from the Trigger object. This (and a few other resources) allow you to create individualized entities while only needing to create classes for those that really behave fundamentally different.

Figure 6: Sessions, Players and Objects

Trigger is most likely played by a human. This human connects to the game via one or more Sessions, one for each client they connect with. Their account on mygame is represented by a PlayerDB entity. The PlayerDB holds the password and other account info but has no existence in the game world. Through the PlayerDB entity, Sessions can control (“puppet”) one or more ObjectDB entities in-game.
In Figure 6, a user is connected to the game with three Sessions simultaneously. They are logged in to their Player account Richard. Through these Sessions they are simultaneously puppeting the in-game entities Trigger and Sir Hiss. Evennia can be configured to allow or disallow a range of different gaming styles like this.
Now, for users to be able to control their game entities and actually play the game, they need to be able to send Commands.
Figure 7: Commands are Python classes too

Commands represent anything a user can input actively to the game, such as the look command, get, quit, emote and so on.
Each Command handles both argument parsing and execution. Since each Command is described with a normal Python class, it means that you can implement parsing once and then just have the rest of your commands inherit the effect. In Figure 7, the DIKUCommand parent class implements parsing of all the syntax common for all DIKU-style commands so CmdLook and others won’t have to.
Figure 8: Commands are grouped together in sets and always associated with game entities.

Commands in Evennia are always joined together in Command Sets. These are containers that can hold many Command instances. A given Command class can contribute Command instances to any number of Command Sets. These sets are always associated with game entities. In Figure 8, Trigger has received a Command Set with a bunch of useful commands that he (and by extension his controlling Player) can now use.
Figure 9: Command Sets can affect those around them

Trigger’s Command Set is only available to himself. In Figure 8 we put a Command Set with three commands on the Dungeon room. The room itself has no use for commands but we configure this set to affect those inside it instead. Note that we let these be different versions of these commands (hence the different color)! We’ll explain why below.
Figure 10: The name Command “Set” is not just a name

Command Sets can be dynamically (and temporarily) merged together in a similar fashion as Set Theory, except the merge priority can be customized. In Figure 10 we see a Union-type merger where the Commands from Dungeon of the same name temporarily override the commands from Trigger. While in the Dungeon, Trigger will be using this version of those commands. When Trigger leaves, his own Command Set will be restored unharmed.
Why would we want to do this? Consider for example that the dungeon is in darkness. We can then let the Dungeon’s version of the look command only show the contents of the room if Trigger is carrying a light source. You might also not be able to easily get things in the room without light - you might even be fumbling randomly in your inventory!
Any number of Command Sets can be merged on the fly. This allows you to implement multiple overlapping states (like combat in a darkened room while intoxicated) without needing huge if statements for every possible combination. The merger is non-destructive, so you can remove cmdsets to get back previous states as needed.


… And that’s how many illustrations I have the stamina to draw at this time. Hopefully this quick illustrated dive into Evennia helps to clarify some of the basic features of the system!

As of today Evennia 0.7 is officially out! Evennia is a Python framework and -server for creating text-based multiplayer games (MU*).

A big thank you to all collaborators that have helped with code and testing along the way!

Here is the hefty forum post that details how you migrate to Evennia 0.7.

Evennia 0.7 comes with a range of changes and updates (these are just the ones merged from the latest devel branch, a lot more has happened since 0.6 that were already in master): 
 

• Evennia separates Player objects from Character objects in that the former is an OOC entity that can puppet one or more of the latter. The name Player was a source of some confusion since this is used differently in other code bases. Player has now been renamed to Account to make its function clearer.
• Evennia's in-built website now uses our own theme and bootstrap under the hood to make it easier to modify as well as rescale it better on mobile devices (webclient is not updated at this point).
• Shared logins between webclient and website, with the ability to log out of each independently of the other if so desired.
• Prefix-ignoring - All default commands are now renamed without their @-prefixes. Both @examine, +examine or examine will now all point to the same command. You can customize which prefixes Evennia simply ignores when searching for a command. The mechanic is clever though - if you create a command with a specific key "+foo", then that will still work and won't clash with another command named just "foo".
• Easy pause mechanisms using yield statements directly in Command code (just do yield 10 in your command code to have it pause ten seconds without blocking anyone else). You can also do retval = yield "Do you want to accept?" and have the command pause (non-blocking) for the player's input before continuing.

New contribs (optional plugins) (selection, new since 0.6 but many have been available on master branch for a while):

• The optional In-Game-Python contrib by Vincent le Geoff allows for coding and scripting in-game using full-fledged Python to make events and triggers. It's not a safe softcode-style language (you have the full power of Python which is not good for untrusted users) but is intended for trusted builders to make complex scripting from the command line.
• Wilderness/maps - Creation of dynamic or static maps based on ascii-maps for dynamically creating rooms and to show when moving around. (titeuf87, Cloud Keeper)
• Full turn-based combat system, meant to expand for a desired system (battlejenkins)
• Alternative Unix-style command base for Evennia, for those that prefer to enter commands like you do on the unix command line (with --flags etc) (Vincent le Geoff)
• Multidescer, which together with Evennia's nick replacement system can be heavily customized in-game (me).
• Mail - a @brandymail-style in-game mail-system (grungies1138)
• Clothing system, for layered outfits adding to the wearer's desc when worn (battlejenkins).

A more technical list from the main announcement post: 

• EvMenu formatting functions now part of class - EvMenu no longer accepts formatting functions as inputs, these are now part of the EvMenu class. To override the formatting of EvMenu nodes you should now override EvMenu and replace the format method you want. This brings EvMenu more in line with other services in Evennia, all of which are built around overriding. 
• Scripts are now valid message senders - Also Scripts can now act as "sender" of a Msg, for example to a Channel or a player.
• Due to the new prefix-ignoring above, @desc (builder-level description-setting) was renamed to setdesc to make it clearly separate from desc (used by a player setting their own description). This was actually the only clash we had to resolve this way in the default commands. 
• Permission Hierarchy names change - To make Evennia's permission hierarchy better reflect how Evennia actually works, the old Players, Player Helpers, Builders, Wizards, Immortals hierarchy has changed to Player, Helper, Builder, Admin, Developer. Singular/Plural form is now ignored so Builder and Builders will both work (this was a common source of simple errors) Old permissions will be renamed as part of the migration process. The distribution of responsibilities has not changed: Wizards (which in some other systems was the highest level, which confused some) always had the power to affect player accounts (like an admin) whereas Immortals had server-level access such as @py - that is, they were developers. The new names hopefully makes this distinction clearer.  
• All manager methods now return querysets - Some of these used to return lists which was a throwback to an older version of Typeclasses. With manager methods returning querysets one can chain queries onto their results like you can with any django query.
• PrettyTable was removed from the codebase. You can still fetch it for your game if you prefer (it's on pypi). But EvTable has more functionality as well as color-support. 
• Add **kwargs support to all object at_* hooks - All object at_ hooks, such as at_look, at_give etc now has an additional **kwarg argument. This allows developers to send arbitrary data to those hooks so they can expand on them without changing the API.  
• Remove {-color tags - The use of {r, {n etc was deprecated for years and have now been completely removed from Evennia's core in favor of only one form, namely |r, |n etc. However, the color parser is now pluggable and the {-style as well as the %c style color tags etc can be re-added to your game since they are now in a contrib.  
• Updated hooks for say/whisper commands - There are now at_before/after_say sub-hooks to allow say to be more easily customized. There is still ongoing discussion on the best way to handle this (see e.g. this PR).  
• More compact distribution of ports - Before, Evennia distributed its ports widely, such as using 4000, 4001, 8000, 8001, 5001, 8022, some of which appeared close but actually was unrelated in functionality. They were also scattered throughout the settings file. The port definitions have now all moved more closer together, at the top of the settings file. Evennia will now use ports 4000-4006 by default (fewer depending on which protocols you want to start). This should make it easier to assign port ranges when wanting to run multiple Evennia instances on the same machine.  
• Change how auto-login works - It used to be that once you logged into the website, every time you went to the webclient you would auto-login, even if you manually quite the webclient before. Only way to really log out (to change account, say) would be to first log out of the website. Now this will work more intuitively - you will still auto-login but if you log out of the webclient you "decouple" from the login state of the website and need to re-login to the webclient next time, even if you are still logged into the website.  
• Better idle disconnects - This is a long-running pet peeve for many. Idle timeouts will now honor a particular lock. Setting this lock allows selected entities (for example staff and bots) to avoid getting timed out like everyone else.
• A slew of bug fixes and other tweaks. See here for the list.

Last night I installed updates on my FreeNAS box and rebooted it. As expected my network died, but then it never came back, which I hadn’t expected. My FreeNAS box provides backup storage space, a local Debian mirror and a mirror of talks from recent conferences. It also runs a couple of virtual machines and one of these provides my local DNS resolver. I hooked up the VNC console to the virtual machine and the problem looked to be that it was booting from the Debian installer CD.

Less than two weeks after the release of KStars 2.8.3 comes another minor bugfix release. Download KStars 2.8.4 for Windows, MacOS, and Linux.

Major highlights:

• Robert Barlow submitted a patch to add elevation information for all cities. The information was built from Google data. It closes Bug #382462. The elevation data is also now sent to the INDI drivers.

• Stars and Deep Sky Objects labels are now zoom-dependent so they appear larger when zoomed in which improves usability.
• Rotators are now fully tested and supported with Ekos capture and align module. In Ekos align module, use Load & Slew to go to any arbitrary target, and then have your rotator exactly match the orientation of the target in the image!

• Fixed several issues with internationalization of some strings discovered by Khalid AlAjaji. Khalid also submitted significant translations for KStars in Arabic!
• Jérôme Sonrier submitted a fresh new updated Daylight Saving rules. He is also working on getting rid of the static TZ rules in KStars and using the ones provided by the system.
• Added PSF (Point Spread Function) convoluted star detection algorithm adapted from PHD2 to improve auto-selection of stars during guide module calibration phase.
• Fixed bug in processing Load&Slew data when some keywords are missing.
• Fix layout issues for RTL languages.

This release is dedicated to Juli, my lovely German Shepard companion for the last 7 years. She is accompanied here by Tommy when he was just a small puppy back then. Long live and prosper my good girl!

Cython 0.27 is freshly released and comes with several great improvements. It finally implements all major features that were added in CPython 3.6, such as async generators and variable annotations. The long list of new features and resolved issues can be found in the changelog, or in the list of resolved tickets.

Probably the biggest new feature is the support for asynchronous generators and asynchronous comprehensions, as specified in PEP 525 and PEP 530 respectively. They allow using yield inside of async coroutines and await inside of comprehensions, so that the following becomes possible:

async def generate_results(source): async for i in source: yield i ** 2 ... d = {i: await result for i, result in enumerate(async_results)} ... l = [s for c in more_async_results for s in await c]

As usual, this feature is available in Cython compiled modules across all supported Python versions, starting with Python 2.6. However, using async cleanup in generators, e.g. in a finally-block, requires CPython 3.6 in order to remember which I/O-loop the generator must use. Async comprehensions do not suffer from this.

The next big and long awaited feature is support for PEP 484 compatible typing. Both signature annotations (PEP 484) and variable annotations (PEP 526) are now parsed for Python types and cython.* types like list or cython.int. Complex types like List[int] are not currently evaluated as the semantics are less clear in the context of static compilation. This will be added in a future release.

One special twist here is exception handling, which tries to mimic Python more closely than the defaults in Cython code. Thus, it is no longer necessary to explicitly declare an exception return value in code like this:

@cython.cfunc def add_1(x: cython.int) -> cython.int: if x < 0: raise ValueError("...") return x + 1

Cython will automatically return -1 (the default exception value for C integer types) when an exception is raised and check for exceptions after calling it. This is identical to the Cython signature declaration except? -1.

In cases where annotations are not meant as static type declarations during compilation, the extraction can be disabled with the compiler directive annotation_typing=False.

The new release brings another long awaited feature: automatic ``__richcmp__()`` generation. Previously, extension types required a major difference to Python classes with respect to the special methods for comparison, __eq__, __lt__ etc. Users had to implement their own special __richcmp__() method which implemented all comparisons at once. Now, Cython can automatically generate an efficient __richcmp__() method from the normal comparison methods, including inherited base type implementations. This brings Python classes and extension types another huge step closer.

To bring extension modules also closer to Python modules, Cython now implements the new extension module initialisation process of PEP 489 in CPython 3.5 and later. This makes the special global names like __file__ and __path__ correctly available to module level code and improves the support for module-level relative imports. As with most internal features, existing Cython code will benefit from this by simple recompilation.

As a last feature worth mentioning, the IPython/Jupyter magic integration gained a new option %%cython --pgo for easy profile guided optimisation. This allows the C compiler to take better decisions during its optimisation phase based on a (hopefully) realistic runtime profile. The option compiles the cell with PGO settings for the C compiler, executes it to generate the runtime profile, and then compiles it again using that profile for C compiler optimisation. This is currently only tested with gcc. Support for other compilers can easily be added to the IPythonMagic.py module and pull requests are welcome, as usual.

By design, the Jupyter cell itself is responsible for generating a suitable profile. This can be done by implementing the functions that should be optimised via PGO, and then calling them directly in the same cell on some realistic training data like this:

%%cython --pgo def critical_function(data): for item in data: ... # execute function several times to build profile from somewhere import some_typical_data for _ in range(100): critical_function(some_typical_data)

Together with the improved module initialisation in Python 3.5 and later, you can now also distinguish between the profile and non-profile runs as follows:

if "_pgo_" in __name__: ... # execute critical code here

MBox is the original and ancient format for storing mail on Unix systems, it consists of a single file per user under /var/spool/mail that has messages concatenated. Obviously performance is very poor when deleting messages from a large mail store as the entire file has to be rewritten. Maildir was invented for Qmail by Dan Bernstein and has a single message per file giving fast deletes among other performance benefits. An ongoing issue over the last 20 years has been converting Mbox systems to Maildir. The various ways of getting IMAP to work with Mbox only made this more complex.

The Dovecot Wiki has a good page about converting Mbox to Maildir [1]. If you want to keep the same message UIDs and the same path separation characters then it will be a complex task. But if you just want to copy a small number of Mbox accounts to an existing server then it’s a bit simpler.

Dovecot has a mb2md.pl script to convert folders [2].

cd /var/spool/mail mkdir -p /mailstore/example.com for U in * ; do ~/mb2md.pl -s $(pwd)/$U -d /mailstore/example.com/$U done

To convert the inboxes shell code like the above is needed. If the users don’t have IMAP folders (EG they are just POP users or use local Unix MUAs) then that’s all you need to do.

cd /home for DIR in */mail ; do U=$(echo $DIR| cut -f1 -d/) cd /home/$DIR for FOLDER in * ; do ~/mb2md.pl -s $(pwd)/$FOLDER -d /mailstore/example.com/$U/.$FOLDER done cp .subscriptions /mailstore/example.com/$U/ subscriptions done

Some shell code like the above will convert the IMAP folders to Maildir format. The end result is that the users will have to download all the mail again as their MUA will think that every message had been deleted and replaced. But as all servers with significant amounts of mail or important mail were probably converted to Maildir a decade ago this shouldn’t be a problem.

• [1] https://wiki1.dovecot.org/Migration/MailFormat
• [2] http://dovecot.org/tools/mb2md.pl

Related posts:

1. Why Cyrus Sucks I’m in the middle of migrating a mail server away...
2. Moving a Mail Server Nowadays it seems that most serious mail servers (IE mail...
3. Mail Server Training Today I ran a hands-on training session on configuring a...