FLOSS Project Planets

A. Jesse Jiryu Davis: It Seemed Like A Good Idea At The Time: PyMongo's "copy_database"

Planet Python - Tue, 2014-12-09 10:11

The road to hell is paved with good intentions.

I'm writing eulogies for four regrettable decisions we made when we designed PyMongo, the standard Python driver for MongoDB. Each of them made maintaining PyMongo painful, and confused our users. This winter, as I undo these regrettable designs in preparation for PyMongo 3.0, I carve for each a sad epitaph.

Today we reach the third regrettable decision: "copy_database".

Date: Fri Sep 12 15:00:20 2008 -0400 copydb - not yet done commit e783239b3f9284d0dfe0161b8f8effc41d33aa57 Author: dwight Date: Sun Sep 14 22:49:30 2008 -0400 copydb work commit 379a7562629ff0803cfb30e0abfcddcbee046a19 Author: Dwight Date: Mon Sep 15 15:30:53 2008 -0400 copydb first commit commit 7ed81cdf6bc3af668273983c8dd890e545bcdaa4 Author: Aaron Date: Tue Feb 16 15:20:35 2010 -0800 SERVER-579 support copyDatabase from source running with security commit 2213f813cf8c81ffa719adb46cfdbdf375bc8fae Author: Mike Dirolf Date: Tue Mar 9 13:06:00 2010 -0500 Adding copy_database --> The Beginning

In the beginning, MongoDB had a "copydb" command. Well, not the beginning, but it was an early feature: MongoDB was less than a year old when Dwight Merriman implemented "copydb" in September 2008.

The initial protocol was simple. The client told MongoDB the source and target database names, and MongoDB made a copy:

You could give the target server a "fromhost" option and it would clone from a remote server, similar to how a replica set member does an initial sync:

This is a really useful feature for sysadmins who occasionally copy a database using the mongo shell, but of course it's not a likely use case for application developers. So the mongo shell has a "db.copyDatabase" helper function, but at the time none of our drivers did.

A year later, in January 2010, a user wanted to do "copydb" from a remote server with authentication. Aaron Staple came up with a secure protocol: as long as the client knows the password for the source server, it can instruct the target server to authenticate, without revealing its password to the target server. The client tells the target to call "getnonce" on the source, and the source responds with a nonce, which the target forwards to the client:

Then the client hashes its password with the nonce, and gives the hashed password back to the target server, allowing the target to authenticate against the source once:

There's one important detail (imagine foreboding music now): "copydbgetnonce" and "copydb" must be sent on the same socket. This will be important later.

In any case, Aaron added support for this new protocol to MongoDB and to the mongo shell, so sysadmins could copy a database from a password-protected remote server. So far so good. But in a moment, we would make a regrettable decision.

PyMongo and "copy_database"

Nicolas Clairon, author of the PyMongo wrapper library MongoKit, asked us to add a feature to PyMongo. He wanted PyMongo to have a special helper method for copydb so "every third party lib can use this method". PyMongo's author, Mike Dirolf, leapt to it: just two days later, he'd implemented a "copy_database" method in PyMongo, including support for authentication.

I understand why this seemed like a good idea at the time. Let's avoid duplication! Put "copy_database" in PyMongo, so every third party lib can use it! No one asked whether any users actually executed "copy_database" in Python. Mike just went ahead and implemented it. How could he know he was setting a course to hell?

Requests, Again

Remember how I said copydbgetnonce and copydb must be sent on the same socket? Well, that wasn't a problem for Mike: at this time PyMongo always reserved a socket for each thread, and you couldn't turn this "feature" off. So if one thread called copydbgetnonce and then copydb, the two commands were sent on the same socket automatically.

But, as I described in my "start_request" story, after Mike had left and I joined the company, I made major connection pooling improvements. This included the ability for threads to freely share sockets in the connection pool. For real applications this dramatically increased PyMongo's efficiency. But it was bad news for "copy_database" with auth: now we needed a special way to ensure that the two commands were executed on the same socket. So I had to update "copy_database": Before calling copydbgetnonce, it checked if the current thread had a socket reserved. If not, it reserved one. Then it called the two commands in a row. Finally, it returned the socket, but only if the socket had been specially reserved for the sake of "copy_database".

There were already two code paths for "copy_database": one with authentication and one without. Now there were four: with and without authentication, with and without a socket already reserved for the current thread. Since concurrency bugs were a greater threat, I bloated the test suite with a half-dozen tests, probing for logic bugs and race conditions.

Motor

Six months after I'd made these changes to PyMongo's connection pool and its "copy_database" method, I first announced Motor, my asynchronous driver for Tornado and MongoDB. Motor wraps PyMongo and makes it asynchronous, allowing I/O concurrency on a single thread, using Tornado's event loop.

Tricking PyMongo into executing concurrently on one thread was straightforward, actually, except for one method: "copy_database". It wants to reserve a socket for the current thread, but in Motor, many I/O operations are in flight at once for the main thread. So I had to reimplement "copy_database" from scratch just for Motor. I also reimplemented all PyMongo's "copy_database" tests, and distorted Motor's design so it could reserve sockets for asynchronous tasks, purely to support "copy_database".

I made a horrible mistake, too: I introduced a bug in Motor's "copy_database" that leaked a socket on every single call, but no one ever complained. The method clearly was risky, and unused.

What the hell was I thinking when I added "copy_database" to Motor? Why would anyone need it? We'd seen no signs, after all, that anyone was even using "copy_database" in PyMongo. And compared to PyMongo, Motor is optimized for web applications with lots of small concurrent operations. It's not intended for rare, lengthy tasks like "copy_database". But I was new at the company and I was excited about making Motor feature-complete: it would include every PyMongo feature, no matter how silly.

SCRAM-SHA-1

The breaking point for the PyMongo team came this fall, when MongoDB 2.8 introduced a new authentication mechanism, SCRAM-SHA-1. When 2.8 is released, SCRAM-SHA-1 will be the new default. And it's not just a better way of hashing passwords: it requires a different authentication protocol than the old MongoDB Challenge-Response mechanism. The old "copydbgetnonce" trick doesn't work with SCRAM-SHA-1.

Our senior security engineer Andreas Nilsson devised a new protocol for copydb with SCRAM-SHA-1, using a SASL conversation. It's more complex than the old protocol:

This accomplishes the same goal as the old copydbgetnonce protocol: it allows the target server to log in to the source server once, without the client revealing its password to either server. But instead of two round trips, three are now required. Andreas added the new protocol to the mongo shell. Bernie had already implemented PyMongo's support for authentication with SCRAM-SHA-1, and he asked me to add it to our "copy_database" helper, too.

I've worked at MongoDB over three years, but I'm still prone to rushing headlong into new features. "I know!" I thought, "I won't just add SCRAM-SHA-1 to copy_database, I'll add LDAP and Kerberos, too!" It took Bernie and Andreas some effort to talk me down. I scaled the work to reasonable proportions. My final patch is tight.

But still, PyMongo's "copy_database" is silly. It now has eight code paths. It can run without authentication, or use SCRAM-SHA-1, or use the old authentication mechanism, or it can try to guess which mechanism to use. And it's implemented both for MongoClient and for MongoReplicaSetClient. The code is hard to follow and the test footprint is like a sasquatch's.

I began to contemplate adding SCRAM-SHA-1 to Motor's "copy_database", too, when suddenly I had a thought: what if we could stop the pain? What if we could just...delete "copy_database"?

Redemption

This year the company has created a dedicated Product Management team whose job is to know what users want, or to find out. Before, each of us at MongoDB had our various contacts with users—Bernie and I knew what Python programmers asked about, salespeople knew what customers asked for, the support team knew what questions people called with—but like any startup we were flying by the seat of our pants when we made decisions about what features to add and maintain.

Now we have a group of professionals gathering and sorting this data. This group can answer my question, "Hey, does anyone care about PyMongo's copy_database method, or is the mongo shell's method the only thing people use?" They researched for a few days and replied,

Consensus from the field is that copydb comes up very little, whether across hosts or not. They are generally OK with not supporting it in the drivers as it is a more administrative task anyway, but would want it supported by the shell.

Things were looking up. Maybe we could just delete the damn method. We polled the drivers team and found that of all the eleven supported MongoDB drivers, only PyMongo, Motor, and the Ruby Driver have a "copy_database" method. And the Ruby team plans to remove the method in version 2.0 of their driver. So we'll remove it from PyMongo in the next major release, and Motor too. Not only will we delete risky code, we'll be more consistent with the other drivers.

Bright Future

In PyMongo's next release, version 2.8, "copy_database" still works; in fact it gains the ability to do SCRAM-SHA-1 authentication against the source server. But it's also deprecated. In PyMongo 3.0 "copy_database" will be gone, and good riddance: there's no evidence that anyone copies databases using Python, and it's one of the most difficult features to maintain. It'll be gone from Motor 0.4 as well.

The lesson I learned is similar to last week's: gather requirements. But this time, we didn't just make up a useless feature: someone actually asked us for it. Even so, we should have turned him down. "Innovation is saying no to a thousand things," according to Steve Jobs.

Features are like children. They're conceived in a moment of passion, but you must support them for years. Think!

Categories: FLOSS Project Planets

Hideki Yamane: ThinkPad X121e with UFEI boot

Planet Debian - Tue, 2014-12-09 09:09
I have ThinkPad X121e and recenly exchanged its HDD to SSD, then I've tried to boot from UEFI but I couldn't. And I considered its something wrong with this old BIOS verion but new one can improve the situation, tried to update it. Steps are below.
  1. get iso image file from Lenovo (Japanese site) (release note)
  2. put iso image into /boot
  3. add custom grub file as /etc/grub.d/99_bios (note: I don't separate /boot partition, maybe you should specify path for file if you don't do so). $ sudo sh -c "touch /etc/grub.d/99_bios; chmod +x /etc/grub.d/99_bios"and edit /etc/grub.d/99_bio file. #! /bin/sh
    menuentry "BIOS Update" {
    linux16 memdisk iso
    initrd16 xxxxxxxxxx.iso
    }
  4. update grub menu with and check /boot/grub/grub.cfg file $ sudo update-grub
    $ tail /boot/grub/grub.cfg
  5. make sure memdisk command is installed $ sudo apt-get install syslinux
  6. just reboot and select bios update menu
Looks okay, its firmware update was success but I cannot boot it (installation was okay). Hmm...

As Matthew Garrett blogged before, probably ThinkPad X121e's firmware doesn't allow to boot from any entries in UEFI except "Windows Boot Manager" :-(

 ...So I have to back to legacy BIOS. *sigh*
Categories: FLOSS Project Planets

Steve Holden: Python Training Day Feedback

Planet Python - Tue, 2014-12-09 07:35
After a day at The Church House I am happy to say that the Intermediate Python repository is in much better shape to move forwards.

Work on the Repository
About eight people turned up throughout the day, with the stalwarts being Kevin Dwyer and João Miguel Neves. With their help I investigated some thorny problems I had created for myself in the area of git filtering. It turns out that there is a subtle bug in Python's JSON generation which has existed for quite a long time.

We eventually worked around it by adding import simplejson as json for now, though I suspect a better fix might be to explicitly add a separator specification. Anyway, with that change the filtering then started to work predictably, and we could move forward with a review of the notebook contents to ensure that changes I made during the teaching were incorporated into the main code base.

With three of us hacking away that work didn't take very long, but before it could commence there was a certain amount of git-wrangling that I confess I probably wouldn't have managed anywhere near as quickly without Kevin's and João's help.

Training Market Discussions
All in all quite a success, and we also spent time discussing the UK training market for Python. I've a few more ideas now about how to approach that market, but if you, dear reader, have any ideas I would be happy to here them either in the comments below or through my web site's contact page.

Thank You
Thanks to everyone who turned up or merely wished the enterprise well. I am really looking forward to spending more time in the UK and helping to encourage even more Python use. Thanks also to the staff at The Church House for their excellent attention. We couldn't have been better taken care of.
Categories: FLOSS Project Planets

Interview with Sylvia Ritter

Planet KDE - Tue, 2014-12-09 07:00

Would you like to tell us something about yourself?

I’m a digital painter, illustrator, concept artist and game developer, currently living in Dresden, Germany.

In January 2008 I started drawing and painting regularly. I went from fineliner drawings to vector art, pencil drawings, sculpting, papercuts, linocuts, fabric design, oil/acrylic paintings, fabric design, typography until I finally found my preferred medium. In 2010 my apprenticeship at a photographer agency ended and I started to work on my art full-time. The same year my husband and I also successfully crowdfunded the creative commons music album “paniq – Beyond Good and Evil”. I created the cover and booklet art and my husband composed the music. In May 2012 we founded our company “duangle” and started another crowdfunding campaign for our game in development “NOWHERE”, a hardcore alien life simulator.

Do you paint professionally or as a hobby artist?

Both, sporadically. I’m not comfortable with either label. Art happens.

When and how did you end up trying digital painting for the first time?

I wanted to get more out of my monochromatic fineliner drawings. I scanned and vectorized these drawings with Inkscape and added some color with MyPaint. That’s how it started. Since 2013 I enjoy painting mostly digitally.

What is it that makes you choose digital over traditional painting?

Almost no costs, more freedom.

No need for a studio to paint. No storage problems. Sharing new artworks is easier and you don’t have to scan the pictures first, which really pays off once the paintings get bigger. Nevertheless I hope that someday I will be able to afford to work with oils. I miss a messy working space with paint everywhere. Sadly we are living in a rented apartment and I can’t do what I want with it.

How did you first find out about open source communities? What is your opinion about them?

My husband is a musician and programmer. He licensed all his work under creative commons. I started to love open source projects, too. I’m very thankful that there are people who work, live for the community and people who support them.

Have you worked for any FOSS project or contributed in some way?

We, duangle, are currently developing an open source game engine “Liminal” for no-static-assets procedural games, using C and LuaJIT:https://bitbucket.org/duangle/liminal_lua/src. My husband is doing the programming and I’m an alpha tester.

How did you find out about Krita?

A friend mentioned Krita in 2010, claiming that it was the best open source alternative to Photoshop. Had to try it and never stopped working with it. On a sidenode I’ve never worked with commercial art software before, but I’m not eager to either.

What was the first thing you made with it?

I had a pencil sketch in my sketchbook, scanned it and added some color. The result is on my deviantArt page – “Study No.80”: http://faith303.deviantart.com/art/study-no-80-361827667.

What do you love about Krita?

I love to work with a tool that is continuously improving. New features are always such a joy. They give you new ways to explore art. Krita is fun. I can’t put it into better words. Everyone should give it a try

What do you think needs improvement in Krita? Also, anything that you really hate?

The usual problem with software  – I hate it when the program crashes and I forgot to save the current progress. Krita has an auto-save mode but it doesn’t always seem to work. A reminder to everyone: save more often! Then, sometimes I duplicate a layer and can’t move it to the place I want, so I have to delete the duplicated layer and try again. That’s all, I’m happy otherwise.

In your opinion, what sets Krita apart from the other tools that you use?

More features and possibilities.

If you had to pick one favourite of all your work done in Krita so far, what would it be?

Not an easy task. I’m the worst critic. One of my favorites is “Keeper of Secrets”.

What is it that you like about it? What brushes did you use in it?

I love wildcats. It was fun to paint the fluffy fur and the lynx eyes.

Can’t remember all the brushes, definitely the block and hard oval opacity detail brush. I try to experiment with all brushes, patterns, textures that can be used in Krita.

Would you like to share it with our site visitors?

Sure! You can find it on my deviantArt page: http://faith303.deviantart.com/art/Keeper-of-Secrets-484685409.

Anything else you’d like to share?

If you want to see and know more about my art please visit the following links:

Animal Illustrations Calendar 2015: https://www.etsy.com/listing/203986448/animal-illustrations-by-sylvia-ritter

Website: www.sylvia-ritter.com

Twitter: https://twitter.com/sylvia_ritter

Facebook: https://www.facebook.com/pages/Sylvia-Ritter-Artist/242134975862592

Patreon: http://www.patreon.com/sylviaritter

More information about our indie game “NOWHERE” on http://www.duangle.com/nowhere.

Categories: FLOSS Project Planets

Thorsten Glaser: The colon in the shell: corrigenda

Planet Debian - Tue, 2014-12-09 06:40

Bernhard’s article on Plänet Debian about the “colon” command in the shell could use a clarification and a security-relevant correcture.

There is, indeed, no difference between the : and true built-in commands.

Stéphane Chazelas points out that writing : ${VARNAME:=default} is bad, : "${VARNAME:=default}" is correct. Reason: someone could preset $VARNAME with, for example, /*/*/*/*/../../../../*/*/*/*/../../../../*/*/*/* which will exhaust during globbing.

Besides that, the article is good. Thanks Bernhard for posting it!

PS: I sometimes use the colon as comment leader in the last line of a script or function, because it, unlike the octothorpe, sets $? to 0, which can be useful.

Update: As jilles pointed out in IRC, “colon” (‘:’) is a POSIX special built-in (most importantly, it keeps assignments), whereas “true” is a regular built-in utility.

Categories: FLOSS Project Planets

Ludvig Ericson: Good news, Everyone! Python 3.x Support for Newly-released Pylibmc 1.4.0

Planet Python - Tue, 2014-12-09 06:13

Brace yourself, Python 3.x support has come!

Thanks to everybody involved in this project; this release involves less authors but a lot more work per person. Thanks especially to Harvey Falcic for the work he put in, without which there wouldn't be any Python 3.x support. Also thanks to Sergey Pashinin for the initial stab at the problem.

Other than that, we had miscellaneous bug fixes, testing improvements, and documentation updates.

Get it on PyPI or check out the GitHub repository.

Last but not least I would like to ask for your support in this project, either by helping out with development, testing, documentation or anything at all; or simply by donating some magic internet money to the project's Bitcoin address 12dveKhqiJWCY8zXT4kaHdHELXPeGAUo9h.

Categories: FLOSS Project Planets

Steve Holden: UK Python Training Open Day and Lunch

Planet Python - Tue, 2014-12-09 05:33
Regular readers (both?) know that I live in the USA but hail from the UK, where I still return regularly to stay in touch with family and friends. So to dip a toe in the UK training waters I booked a room (rather a nice one, as those who have attended other events I organized will not be surprised to learn) and put out an announcement that I was running a Python Programming Skills Lab. You could have booked a place here. Of course if this were the movies, the spirit of “if you build it they will come” would rule and I could welcome those wishing to drink at the Python fount. But it isn't.

So
a) I am not a marketing genius;
b) this is not the movies.Result: I am now planning to spend the day in said reserved room, working on my current open source project, whose sadly neglected repository can be found in a weed-infested corner of Github. This is the code I used as a script for the videos which (if you are reading my blog) you should see advertised up at top right. The code is presented as iPython Notebooks, and only some of them are properly documented in interspersed Markdown cells, so if I end up spending the day alone then at least some sensible purpose will have been served.

With that in mind the Skills Lab is now metamorphosing into an open Python Training Day. I understand this is very short notice (I'm not very good at operations either, go figure), but if you have any interest in either receiving Python training or having such training presented to a target group I would be delighted to talk with you on Tuesday December 9, 2014 at any time between 10:00 am and 4:30 pm at the Church House Conference Centre in Westminster.

If you would like to join me for lunch then please sign up (there are 12 places for lunch) and arrive no later than 12:30. If you simply want to drop in and say hello (or, better still, help work on the codebase) then feel free to arrive at any time during the advertised hours and stay as long as you like (but please do sign up for lunch if you plan to eat). It would be helpful if you reserved a drop-in place even if you aren't coming to eat. Those with an interest in beer might want to arrive towards the end of the proceedings so a post-event excursion to a licensed hostelry can be entertained if appropriate.

The room reserved for this event is absolutely delightful, with a view across the Dean's Yard. I welcome all Python friends and anyone who would like to get to know me professionally to visit at some convenient time during that day. But do remember to reserve a place if you want lunch!

I have had very little time to work on the Github repository since the videos were shot, and yearn for an army of Internet interns to help me improve what I believe are useful open-source training materials. In the absence of visitors that day, any spare time I have on that day will go to improve the Notebooks, so at the very least the curated Notebook collection will become more useful.
Categories: FLOSS Project Planets

Michael McCandless: Scoring tennis using finite-state automata

Planet Apache - Tue, 2014-12-09 05:31
For some reason having to do with the medieval French, the scoring system for tennis is very strange.

In actuality, the game is easy to explain: to win, you must score at least 4 points and win by at least 2. Yet in practice, you are supposed to use strange labels like "love" (0 points), "15" (1 point), "30" (2 points), "40" (3 points), "deuce" (3 or more points each, and the players are tied), "all" (players are tied) instead of simply tracking points as numbers, as other sports do.

This is of course wildly confusing to newcomers. Fortunately, the convoluted logic is easy to express as a finite-state automaton (FSA):



The game begins in the left-most (unlabeled) state, and then each time either player 1 (red) or player 2 (blue) scores, you advance to the corresponding state to know how to say the score properly in tennis-speak. In each state, player 1's score is first followed by player 2's; for example "40 30" means player 1 has scored 3 points and player 2 has scored 2 and "15 all" means both players have scored once. "adv 2" means player 2 is ahead by 1 point and will win if s/he scores again.

There are only 20 states, and there are cycles which means a tennis game can in fact go on indefinitely, if the players pass back and forth through the "deuce" (translation: game is tied) state.

This FSA is correct, and if you watch a Wimbledon match, for example, you'll see the game advance through precisely these states.

Minimization

Yet for an FSA, merely being correct is not good enough!

It should also strive to be minimal, and surprisingly this FSA is not: if you build this Automaton in Luceneand minimize it, you'll discover that there are some wasted states! This means 20 states is overkill when deciding who won the game.

Specifically, there is no difference between the "30 all" and "deuce" states, nor between the "30 40" and "adv 2" states, nor between the "40 30" and "adv 1" states. From either state in each of these pairs, there is no sequence of player 1 / player 2 scoring that will result in a different final outcome (this is in principle how the minimization process identifies indistinguishable states).

Therefore, there's no point in keeping those states, and you can safely use this smaller 17-state FSA (15% smaller!) to score your tennis games instead:



For example, from "15 30", if player 1 scores, you go straight to "deuce" and don't bother with the redundant "30 30" state.

Another (simpler?) way to understand why these states are wasted is to recognize that the finite state machine is tracking two different pieces of information: first, how many points ahead player 1 is (since a player must win by 2 points) and second, how many points have been scored (since a player must score at least 4 points to win).

Once enough points (4 or more) have been scored by either player, their absolute scores no longer matter. All that matters is the relative score: whether player 1 is ahead by 1, equal, or behind by 1. For example, we don't care if the score is 197 to 196 or 6 to 5: they are the same thing.

Yet, early on, the FSA must also track the absolute scores, to ensure at least 4 points were scored by the winner. With the original 20-state FSA, the crossover between these two phases was what would have been "40 40" (each player scored 3 points). But in the minimal machine, the crossover became "30 30" (each player scored 2 points), which is safe since each player must still "win by 2" so if player 1 scores 2 points from "30 30", that means player 1 scored 4 points overall.

FSA minimization saved only 3 states for the game of tennis, resulting in a 15% smaller automaton, and maybe this simplifies keeping track of scores in your games by a bit, but in other FSA applications in Lucene, such as the analyzing suggester, MemoryPostingsFormatand the terms index, minimization is vital since it saves substantial disk and RAM for Lucene applications!
Categories: FLOSS Project Planets

Andre Roberge: Reloadable user-defined library with Brython/Python

Planet Python - Tue, 2014-12-09 04:48

[Note: this post is a more detailed explanation of something that is briefly described in a previous post on supporting multiple human languages for Reeborg's World.]

In Reeborg's World, I want to have programmers (read: Python beginners who have never programmed before) to be able to learn about using libraries in Python.  As usual, I was looking at the simplest way to introduce the idea of libraries.  Since almost all of the programs that programmers write make use of their own definition for turning right:

def turn_right():
turn_left()
turn_left()
turn_left()
it makes sense to have this "reusable" function be put in a library. So, instead of a single code editor, the code editor has two tabs: one for the basic program and one for the library. Initially, I only had a Javascript version of Reeborg's World, but I still wanted to introduce the concept of using libraries. So, when I first introduced support for using a library with Javascript, I cheated.  And I continued to cheat when I added Python support.  If the programmer wanted to use the functions (or anything else) defined in their library, I required them to insert the following statement in their program.

from my_lib import *
Before running the program using Brython's exec(), I would scan the code in the editor's program tab. If I found this general import statement, I would replace the line by the entire content of the editor's library tab and execute this modified source instead.   [Note: I have since removed the idea of using a library in the Javascript version since there is no natural syntax for importing a library using Javascript.]

However, this approach had two problems:
  1. It did not support the other ways to use an import statement in Python (see below for an example).
  2. It encouraged a bad practice of including everything, polluting the program's namespace.  I already had shown the idiom "from some_lib import *" when explaining how to have Reeborg understand instruction using other human languages (such as from reeborg_fr import * for the French version, or from reeborg_es import * for the Spanish version; other translations welcome! ;-)
I wanted to encourage good programming practice, such as using

from my_lib import turn_right
One problem I had is that Brython's import mechanism is based on retrieving files on the server using ajax calls.  This by itself might not be a problem ... except that I do not want to store anything created by users on the server: Reeborg's World is meant to be run entirely inside the browser, with no login required. (The content of the editor and library tabs are saved in the browser local storage so that they are available when the user comes back to the site using the same browser with local storage enabled.)

Another problem I had is that, once a module is imported, future import statements for that module make use of the cached version.  If the programmer modifies the code in their library (tab), the corresponding module needs to be reloaded.  I need for this to be done automatically, without the programmer having to do anything special.

One solution to these problems might have been to create a special importer class that could import code directly from the library tab and add it to sys.meta_path.  Then, after a program has been run, remove all traces of the imported module (user's library) so that the next time it is executed, the import takes place all over again.

I decided instead on a different approach.  I created a simple module, called my_lib.py  (and another one, biblio.py, for the French version) and put it in Brython's site package directory.  The content of that module is simply:

from reeborg_en import *
which ensures that all normal robot commands can be used in that module.  When Reeborg's World is first loaded, I import that module so that it is cached.  Then, whenever the programmer's code needs to be executed, instead of simply having exec(src) called, the following is called instead:

def generic_translate_python(src, lib, lang_import, highlight):
''' Translate Python code into Javascript and execute

src: source code in editor
lib: language specific lib (e.g. my_lib in English, biblio in French)
already imported in html file
lang_import: something like "from reeborg_en import *"
'''
# save initial state of lib
initial_lib_dict = {}
for key in lib.__dict__:
initial_lib_dict[key] = lib.__dict__[key]

exec(library.getValue(), lib.__dict__)
exec(lang_import)
if highlight:
src = insert_highlight_info(src)
exec(src)

# remove added definitions
new_keys = []
for key in lib.__dict__:
if key not in initial_lib_dict:
new_keys.append(key)
else:
lib.__dict__[key] = initial_lib_dict[key]

for key in new_keys:
del lib.__dict__[key]


In the above, highlight refers to some pre-processing of the code which allow to show which line of the code is executed as illustrated in two previous blog posts.  library.getValue() is a method that returns the content of the library tab.
Categories: FLOSS Project Planets
Syndicate content