Snippets in Kate 5

Planet KDE - Sun, 2014-09-14 20:49

Recently I spent some time to port and clean up the Snippets plugin and the underlying template interface for Kate 5.  It's now fully working again and more powerful than ever. The template code was originally written by Joseph Wenniger and most of what I show here is still working like originally implemented by him. Still, there were some improvements I would like to show; also, I'm sure many readers might not be aware of this great feature at all.

Classical snippets use case: insert a for loop witout having to type the iterator variable three times.The template interface, which is part of the long-time stable KTextEditor API, was heavily cleaned up and now just consists of a single function
bool insertTemplate(const KTextEditor::Cursor& insertPosition,
const QString& templateString,
const QString& script = QString());which inserts a template into a view at the given position. It's very easy to use and still powerful -- if you write an application which uses KTextEditor, it might be worth to spend a moment thinking about how you might be able to make use of it.
I also heavily refactored the implementation of the interface. More than 1000 lines of code were removed while effectively enhancing functionality. 
Core functionality changesI changed the language of the snippets a bit to make it more clear and easy to use. In the following, I want to give a short overview of how it works now.

The heart of the templates (or snippets) are editable fields (shown in green). They are created in the template string by writing ${fieldname}. They can have a default value, which can be any JavaScript expression. Pressing Tab jumps between the fields of a template. Whenever such a field is changed, all so-called dependent fields are updated. Those can simply be mirror fields (created by having a second field with the same name), or can do something which depends on the contents of the other fields in the template, such as perform replacements or concatenations. Again, you can have arbitrary JavaScript expressions doing that.
An example snippet (not very useful in practice) which has three editable fields (find, replace and sample_text) with a default value for each. Changing the values will update the result in the red "dependent" field in real-time.Noticeable improvements over the previous functionality (from KDE 4 times) is that you can have fields with arbitrarily complicated default values which are still editable, and that the dependent fields can use all other fields as input (not just one like in KDE 4). It is now also possible to have inline JavaScript doing simple operations in the template.

The Shortcuts feature for the snippets now actually works in Kate.

Snippets now also have proper undo; in KDE 4, only a single character typed could be undone at once while editing a snippet. Now, undo grouping works like it always does.
User interface improvementsFor easy testing of your snippets, the "Edit Snippet" dialog has a "Test snippet" button now, which lets you test your snippet on-the-fly.
The user interface was simplified by removing unneeded options, and an inline quick-help feature was added which introduces the user to the most important features of the snippet language. Just click the "More" button.
Inline documentation on how snippets workAn example: C++ Header guardsAs an example for how this feature works, let's look at how to create a snippet to generate a C++ header guard. First, create a repository for your C++ snippets:
Open the Snippets toolview and click "Add Repository".Then, enter a name and specify that you want this only for C++ files:
Create your new repository.Then, add a snippet:
Add a snippet. Easy.
You can retrieve the document's file name from the editor, make it upper-case and replace dots by underscores automatically to get a nice header-guard-suitable format by using code like this:
Example code for how you can create C++ header guards fully automatically.If you do not want the guard field to be editable, just create a function which does the upper(fileName...) stuff, and have three fields which call the function (like ${func()}) instead of the two mirror fields and one default-valued editable field. If you do that, the template handler will immediately exit and not present any editable fields.
The ${cursor} variable can be used to place the cursor after all fields were filled. When you type something there, the handler will exit.

Click Ok. Now, to use your snippet, either press the shortcut you defined (if any), click it in the snippets toolview, or use code completion:
Snippets appear in code completion.Result after executing our new header guard script. A sensible default value was selected automatically. Pressing Escape or Alt+Enter will exit the template handler and place the cursor at the point marked with ${cursor} in the template.That should hopefully equip you with most of the knowledge you need to write your own snippets. If you like, you can use the full kate scripting API to write snippet code -- it for example allows you to retrieve the text in the current selection and similar useful things.
Some more examples on what you can do Here's a few snippets demonstrating the features of the engine while partly being of debatable practical relevance. I'm sure you can come up with better use cases for some of those things though.
Write a clean regular expression in a comment and have the snippet mirror it with added extra-backslashes and removed spaces in a QRegularExpression variable. Makes regular expressions even more write-only than they already are.Get the file encoding from the editor and use it as the coding of a python file header.
Some base64 in the selection ...
... decoded by a snippet which takes the selection and inserts the base64-decoded result.Next stepsMy next step will be to make this plugin loadable in KDevelop as well -- which should be quite easily possible due to the awesome work done in kate to make the plugin infrastructure more generic. If you have further ideas on how to improve the snippets, let me know :)

Categories: FLOSS Project Planets

Justin Mason: Links for 2014-09-14

Planet Apache - Sun, 2014-09-14 19:58
  • Platform Game

    I’m ambivalent about Microsoft acquiring Mojang. Will they Embrace and Extend Minecraft as they’ve done with other categories? Let’s hope not. On the other hand, some adult supervision and a Plugin API would be welcome. Mojang have the financial resources but lack the will and focus needed to publish and support a Plugin API. Perhaps Mojang themselves don’t realise just how important their little game has become.

    (tags: minecraft platforms games plugins mojang microsoft)

Categories: FLOSS Project Planets

Károly Négyesi: Drupal 8 progress from my / MongoDB perspective: update #29

Planet Drupal - Sun, 2014-09-14 18:47

Perhaps the most important development is the final naming of what was field/field instance in Drupal 7: in Drupal 8 these are configuration entities. The machine names are field_storage_config and field_config. There has been several renames but we have settled on these finally (although the field_config rename is not yet in). It does reflect the most important difference between them: field storage contains everything pertaining to the storage of the field. The things that do not change the storage of it go into the instance. However, some confusion might come from already existing Drupal 8 documentation and other materials where field_config was the name for what ended up as field_storage_config.

In other rename news, entity storage/list/form/render "controllers" are now handlers. This probably decreases confusion as controllers are used by the router system to deliver the page content.

Migrate now tracks the whether a migration has been run or not. Previously we were guessing from the id mappings -- but if a source returned zero rows we couldn't really say whether the migration ran or not. Now we could clarify for real what are hard dependencies and what are optional. In the Drupal 6 migration, there are few optional ones which is just as it should be.

There's a Views migration in the works. As expected it's really complicated with lots of moving parts but it's coming along nicely.

Only book storage needs entity field query conversion after the epic comment query conversion issue finally got resolved.

One of the child issues of the last beta blocker removes the special casing for SQL storage -- hurray! Instead of requesting the schema of entity tables during module install (and other places) and then creating the tables there are now events on entity type create, update and delete. This makes it possible to create mongodb indexes cleanly.

Categories: FLOSS Project Planets

Gregor Herrmann: RC bugs 2014/34-37

Planet Debian - Sun, 2014-09-14 17:13

the perl 5.20 transition is over, debconf14 is over, so I should have more time for RC bugs? yes & no: I fixed some, but only in "our" (as in: pkg-perl) packages:

  • #711418 – src:libanyevent-dbi-perl: "libanyevent-dbi-perl: FTBFS: Failed test 'Using an unknown function results in error'"
    add patch for newer SQLite (pkg-perl)
  • #756566 – libxml-dt-perl: "libxml-dt-perl: Insecure use of temporary files (CVE-2014-5260)"
    upload new upstream release (pkg-perl)
  • #759838 – src:padre: "padre: FTBFS: Failed test 'no warnings'"
    fix 2/3 of the test failures in git, last one fixed by dod (pkg-perl)
  • #759942 – src:cpanminus: "cpanminus: FTBFS: Can't write to cpanm home '/sbuild-nonexistent/.cpanm': You should fix it with chown/chmod first."
    set HOME once more in debian/rules (pkg-perl)
  • #759964 – src:libhtml-formhandler-model-dbic-perl: "libhtml-formhandler-model-dbic-perl: FTBFS: dh_auto_test: make -j1 test returned exit code 2"
    upload new upstream release (pkg-perl)
  • #761312 – libdbix-class-resultset-recursiveupdate-perl: "libdbix-class-resultset-recursiveupdate-perl: missing dependency on liblist-moreutils-perl"
    add missing dependency (pkg-perl)
  • #761313 – libgeo-google-mapobject-perl: "libgeo-google-mapobject-perl: missing dependency on libjson-perl"
    add missing dependency (pkg-perl)
  • #761315 – libfile-read-perl: "libfile-read-perl: missing dependency on libfile-slurp-perl"
    add missing dependency (pkg-perl)
  • #761319 – libpod-wordlist-hanekomu-perl: "libpod-wordlist-hanekomu-perl: missing dependency on libtest-spelling-perl"
    add missing dependency (pkg-perl)
  • #761526 – src:liblocale-maketext-gettext-perl: "liblocale-maketext-gettext-perl: FTBFS: Tests failures"
    skip test which needs internet and a writable $HOME (pkg-perl)
  • #761558 – src:libposix-strptime-perl: "libposix-strptime-perl: FTBFS: Tests failures"
    skip test which needs internet and a writable $HOME (pkg-perl)
Categories: FLOSS Project Planets

d7One: Step by step guide to installing Commerce from scratch

Planet Drupal - Sun, 2014-09-14 14:50
The following guide was used in a presentation at DrupalCamp Montreal 2014. The presentation focused on how to install Drupal Commerce from scratch. FIRST FLOOR
  1. https://www.drupal.org/project/commerce
  2. Install required modules
    • drush dl ctools views entity rules addressfield commerce
    • drush en ctools views entity rules addressfield commerce -y
Categories: FLOSS Project Planets

Steve Purkiss: Case Study - NickJr. Colour CMS & API - Feedback sought!

Planet Drupal - Sun, 2014-09-14 14:00
Sunday, 14th September 2014Case Study - NickJr. Colour CMS & API - Feedback sought!

It's been an embarrassing amount of time since I last blogged or did anything to my site so I've started to create some content, but before I submit it as a case study on Drupal's site I'd love to have some feedback, it's been years since I wrote case studies so a little rusty. Is it of interest? Does it help? Does it answer the right questions? Are there any questions you'd like me to answer?

Brief overview

Viacom’s Nick Jr. is a digital television channel aimed younger children, home to popular shows including Dora the Explorer, Angelina Ballerina, and PAW Patrol. 'Colour' is a section of the Nick Jr. website where, daily, tens of thousands of users interact with predefined line-art drawings using paint brushes and stickers. They can then, if they wish to, send pictures they create to their friends via email. The mostly Flash-based front-end currently connects to a legacy back-end system which is being replaced with Drupal.

Completed Drupal site or project URL


Why Drupal was chosen

Drupal's core and contributed modules along with customisable web administration interface provided all the functionality required out-of-the-box and is already used widely throughout the organisation.

Describe the project (goals, requirements and outcome) Project Goals

The goal of this project was to provide a bespoke back-end CMS based on Drupal to replace the legacy system, along with introducing new functionality enabling Send to Gallery and Pic of the Week features. The CMS needed to as simple as possible for content managers to use, and the whole system had to be able to cope with the tens of thousands of users using the API on a daily basis.


The site was to be delivered as a Drupal install profile so it can be adapted and re-used – an initial step towards creating internal Nickelodeon-specific distributions of Drupal, further lowering costs and time-to-market for future projects with similar requirements. Purkiss built a similar system a few months prior for Universal Artists uView Augmented Reality App and have extensive experience integrating Drupal with 3rd party services so were ideally placed to deliver this type of Drupal project.


First we set up the install profile with Shiny, the administration theme which comes with Drupal Commerce Kickstart, to be the main theme to the site. Shiny is a great theme for this kind of system where there's really only one main display the content managers interact with – it's easy on the eye with good typography and spacing.

Legacy Migration

We then re-created what functionality was required from the legacy system using Drupal Services – the calls for saving images, retrieving images for a show, etc. There was a major difference here as the legacy system stored all the data in flat XML files with no user data linked. Drupal is a relational database system so we have user accounts, content types and taxonomies – this proved to be an exercise in both education of the client as to the benefits of hooking into the Drupal API, along with re-architecting of the current content model and workflow.


We used a Features-based approach to development with each of these features, along with the install profile, having their own private Git code repository in the cloud in order for issue tracking and maintenance to be easier.

The features we delivered were as follows:

  • Content Types
    • NickJr Colour Image – stores the user’s image along with a Canvas ID which refers to the line art used, a taxonomy field to link to the associated show, and two flags – one to show if the user has sent the image in to Nickelodeon for consideration to be included in the showcase, and one to denote whether it is in the associated show’s showcase.
  • Fields
    • User Profile Fields – extra profile fields to store the user’s name and age.
  • Rules
    • Redirect after login – redirects to the front page view after logging in.
    • Pic of the Week – Views Bulk Operations Component – one rule to set and one to unset an image as Pic of Week
    • Showcase – one to set and one to unset an image as to be in a Show's Showcase
  • Services
    • NickJr. Colour API – the Services settings for the custom API endpoints.
  • Taxonomy
    • Shows Taxonomy – with Pic of the Week field
  • Variables
    • Always harmonize views filters – Allows views to use both contextual and exposed filters for views
  • Views
    • Front Page – Displays all images, defaults to only show images Sent to Nick
    • Showcase – Displays images in showcase, all if no show_id sent through
    • User Profile – Displays users images on their profile page

Project timescale and budget was for an incredibly tight 10 days which we delivered in, however were subsequently hired for a further 7 days to re-factor the API and provide a few final additions to the administration interface.

Out-of-the-box, Drupal Services module provide a standard set of API calls so you can do things like create a user, a node, etc. so all the functionality required was there however required a number of calls to the API to achieve the desired workflow. It was decided to spend extra time creating custom API endpoints which would hide this functionality, reduce the number of API calls required, and make it easier for future users of the API.

The downside to this approach is the technical debt introduced when diverting away from the *.drupal.org infrastructure of support, i.e. core and contributed modules. In this instance the code was simply wrapping existing API calls and the client has access to technical resources so the decision was to take on the technical debt in order to produce a more tailored outcome.


We used the extra time to create a RESTful API which abstracted away from Drupal Services out-of-the-box API endpoints along with coding an auto account creation system to link up user data from the front-end without users having to enter their email address, a requirement of Drupal’s user account creation. We documented the API using Drupal’s inbuilt help module and passed the system over to the client, helping them through building the system from the Drush make file.


The resulting system is a clean, simple-to-use CMS with RESTful API handling thousands of users on a daily basis. Content managers have a separate role with very restricted access so can only do what they need to do on the system.

We really enjoyed delivering this project, however alert anyone thinking of building similar to allow extra time, especially if it is their first Drupal experience - no matter how good specifications are, issues usually arise in one form or another. Also often once people do start using Drupal and see the possibilities available, requirements change and it is good to allow for those to be taken into account.

About Purkiss

Purkiss helps organisations onboard Drupal through consultancy, development, training and support. For more information about Purkiss services please visit http://purkiss.com



Why these modules/theme/distribution were chosen
  • boolean_formatter – makes the user interface a little nicer by providing icons for boolean fields such as ticks and crosses
  • CORS – required for connecting to the front-end
  • EVA - we used an Entity Views Attachment to display users pictures on their profiles
  • features - features allows you to export configuration settings to code
  • filter_harmonizer - Views Filter Harmonizer fixes an issue when using multiple filters on views
  • mimemail - used to send images via email
  • rules - as described above, used for redirecting user on login and for custom views bulk operations
  • services - used for the RESTful API
  • shiny - an administration theme we are using for the entire site
  • smtp - for sending email
  • strongarm - for storing variables in features
  • views_bulk_operations - to enable easy workflow, such as adding an image to a showcase
  • views - so we can display the data we want easily
Community contributions

As the API is specific to the client and used mostly existing functionality we had no code to contribute. The only custom code was for automatic account creation and there are existing modules which provide similar functionality which we used the code from but is too specific to this install to re-use externally.

This is however the second time we’ve been asked for this sort of system, and with the growth of mobile apps we are currently working out whether it would be possible to create a distribution which will help get most of the way there - at least have relevant modules and perhaps an example scenario based on the simple image requirements for this project.

tags: case studiesDrupal PlanetPlanet Drupal
Categories: FLOSS Project Planets

How to launch .jar files using nautilus or nemo

LinuxPlanet - Sun, 2014-09-14 11:33

This is not nautilus specific issue and will work in more tools (like other file manager, xdg-open in cli etc)

Create a run-jar.desktop in your ~/.local/share/applications/ directory with the following content:

[DesktopEntry] Encoding=UTF-8 Type=Application Exec=java -jar %f Icon=java Name=run-jar Name[zh_CN]=run-jar Comment=Run the jar file Comment[zh_CN]=运行 JAR 文件

Now when you open the file’s property dialog and go to open with tab, you can see run-jar mentioned in ‘show more app’.

To make run-jar the default action, use nautilus ‘set default’ button or the type the following command in a terminal:

xdg-mime query default application/x-java-archive

The mime type can be found with the command:

xdg-mime query filetype my_shiny_app.jar


There is other ways to do it too, like creating a nautilus script. But this feels like a better way though.

Categories: FLOSS Project Planets

How To Create Application Folders in GNOME

LinuxPlanet - Sun, 2014-09-14 06:51

You might want to add some more folders into GNOME, but seems impossible from a default system. So let’s show you how to do it.

1 – Open Terminal / Install ‘dconf’


sudo conary install dconf-editor 2 – Open ‘dconf’ and edit

Open dconf and navigate to org >> gnome >> shell. You’ll see a text box say “app-folder-categories.” You’ll see only two things listed there, maybe one. Double-click it and paste the following:

['Utilities', 'Games', 'Sundry', 'Office', 'Network', 'Internet', 'Graphics', 'Multimedia', 'System', 'Development', 'Accessories', 'System Settings', 'wine-wine', 'Crossover', 'Crossover Games', 'Science', 'Education', 'Universal Access', 'chrome-apps', 'Other']

Now you have alot more folders.


Categories: FLOSS Project Planets

Hernan Grecco: PyVISA 1.6 brings comprehensive resources classes, thread-safety and more

Planet Python - Sun, 2014-09-14 06:39
PyVISA is a Python frontend for the VISA library that enables controlling all kinds of measurement equipment through GPIB, RS232, USB, Ethernet among others interfaces.

PyVISA 1.6 release is close and it brings several new nice things. One of the most visible improvements is the addition of a comprehensive set of resources classes, each mapping to a one of the 13 types of resources. Each class implements the methods and attributes that the specific session can handle, providing a Pythonic way to interact with the device (See here). This has allowed us to implement higher level functions such as the group execute trigger in the GPIB Interface.

PyVISA 1.6 brings a much better way to query a device for values, providing a comprehensive API to convert back and from ASCII and binary blocks. An API to write values to a device has been added (See here).

PyVISA 1.6 is thread-safe. While the VISA library has always been thread-safe, PyVISA was not. We have refactored the code removing several global variables that were used to handle state and tweaked the API accordingly.

PyVISA 1.6 continues on the road started with 1.5 to improve debugging, usability and reporting. The quality and density of logging messages have been improved as well as the information provided with the error messages. Python 3.4 Enums are now used to handle constants (the backported packaged enum34 is installed in older Python version).

There is one more change under the hood. PyVISA 1.6 can discover and use other backends to talk to devices (See here). This feature is allowing us to prototype some new packages about which I am going to blog next week.

All these things are only possible as we have dropped some backwards compatibility requirements. In particular, we have removed the legacy package, the string manipulation functions and all the singletons. Most programs will run without major changes but in any case, all API modifications are documented here to help with the transition.

All this would not have been possible without the support of a great community that has been helping with the development and testing different development versions. But we really need your help before uploading this to PyPI as an stable release. PyVISA is all about interacting with your operating system (finding and locating the C library) and you instruments. With the variety of platforms around, it is very difficult to test. We are asking all users to install PyVISA from the github repository and test it on your programs. You can install PyVISA or upgrade to the development version using pip:

$ pip install -U https://github.com/hgrecco/pyvisa/zipball/master
Again, this release is only possible thanks to the contribution of a lot of people that contributed bug reports, testing and code. Thanks to everybody!

Submit your bug reports, comments and suggestions in the Issue Tracker. We will address them promptly.

Read the development docs: https://pyvisa.readthedocs.org/en/master/
or fork the code: https:/https://github.com/hgrecco/pyvisa/
Categories: FLOSS Project Planets

Takeaway from Akademy

Planet KDE - Sun, 2014-09-14 02:48

[[ Way back in 2008 or so, at Akademy in Mechelen, most attendees stayed in a hostel with 4 bunks to a room, which meant that after a long day hacking we ended up talking. It may have been Kevin who first asked "Ade, tell us a story." This year, in 2014, at my first Akademy in four years, I received the same request. I've written down the story that I told this year. It was invented on the tram in Brno between Česká and Technologiky Park at 23:20 on the 9th of September 2014. When I got home I illustrated it. Text and illustrations licensed under CC-BY. If Timothéé feels like drawing it up better, by all means. This is a work of fiction. Any resemblance to people or rabbits, living or dead, is purely coincidental or a weak attempt at humor. ]]

The tale of Glenda the Plan 9 Rabbit

Once upon a time, Glenda the Plan 9 Rabbit was hopping through the forest. She was very happy, because she was free and hopping through the forest. The sun was shining and the birds were singing in the trees. Glenda was free because she had escaped from Bell Labs with a BSD license. That’s not quite as good as a license to kill, but you hardly need one of those as a member of the lagomorphidae, do you? Glenda hopped amongst the trees and she hopped through the bushes and she was very happy. There was a forest path and she ignored the crossing lights and just hopped straight to the other side. She nibbled some grass and hopped onwards, through the trees, under the bushes and over the grass. 

But then, suddenly, Glenda was stuck! She could not move her hoppy feet. She looked down, and there was a big black pool of tar. She tried to pick up her front left foot, but the tar was stuck to that. She tried to pick up her right front foot, but the tar was stuck to that too. Struggle as she might, her feet would not let go. Her left back foot sunk a little deeper into the tar. And then, with a glurpy sound, the tar made a great big tentacle that reached up and swayed in the crisp morning air. Glenda could still hear the birds singing as the tentacle squeezed her. With growing dread, Glenda realized that she would soon be stuck in a compressed tar archive. “That’s it!” thought Glenda, and she pulled out the “j” she still had left over from jay-walking that morning. But it was BSD tar, and it doesn’t understand the “j” option. Desperate now, Glenda the narcoleptic Plan 9 Rabbit (this is actually a useful power saving feature in all modern kernels) quickly fell asleep and with the help of those “z”s extracted herself from the terrible tar archive. She cleaned her feet on some chestnut leaves and hopped on through the forest. She hopped through the trees and under the bushes and over the grass. The sun was shining and the birds were still twittering. Glenda hopped over the grass again.

But suddenly, MROWR! A cat(1) ran out from the bushes towards her, with great terrible pointed teeth and sharp claws and Glenda jumped a foot in the air and turned and ran and ran and ran away from the cat(1) until she reached the edge of the sea and she ran across the beach and jumped into the water! Splash! The cat(1) could not get at her now, and it stayed on dry land, hissing and spitting. She swam and swam and there was a big red sailboat out on the sea. She swam until she got close and climbed up the ladder and there was a big man with a broad smile. 

It was Larry O’Liason, the Irish gourmand who had grown fat by eating all the animals in the sea. “Welcome to my boat, O’Coral. An Irish name of the sea, I’m sure you’ll see. Be welcome! Have something to eat!” And Glenda was very happy. There were no birds singing here, but gulls that screeched, and there was no grass to hop on, only 108 feet of teak from bow to stern. Still, Glenda was happy on board the O’Coral, and happier still because Larry gave her succulent grass to eat, and fresh crisp carrots. Soon Glenda was gaining weight, perhaps even becoming a little bloated. Her features, once sleek like a healthy forest rabbit, grew puffy. And Larry, she noticed, was watching her. Watching, while he sharpened a knife. But still she got her crispy carrots and succulent grass, and she hopped less on the teak deck. Until one day, Larry, with a grim smile, said “tonight, I dine on water-rabbit! I’ve not had that before.” 

And he pulled out his knife and gave out a cry and jumped at Glenda who shot a foot in the air and turned and ran and ran across the slippery teak deck and she ran to the bow and she ran back to the stern and she could hear Larry laughing behind her and he was gaining and she ran back to the bow and Larry was still gaining when THUMP! Another ship hove alongside! It was a crusty old galleon, and it had hoisted the black flag. The skull and crossbones! Larry gripped his knife between his teeth and turned to face this adversary. At the bow of the galleon stood a man with a long tangled beard and a wild look in his eyes. He held a scimitar in one hand and cried “O ho ho ho! I, Richard Boatsman, have come to thwart your terrible deeds, Larry O’Liason! And the railing rang as he leaped across and his scimitar was shining in the sun. It didn’t flash, though Richard did gnash his teeth. He bore down on Larry and the terrified Glenda, who regarded him with great big bunny brown eyes full of awe and relief. Richard faced the opposite captain, glanced at Glenda, and said “oh, hang on, you’re BSD licensed, aren’t you. Well, carry on then.” He turned and sailed off in his boat and Larry had Glenda for dinner and lived happily ever after.


Categories: FLOSS Project Planets

Vasudev Ram: Read a DBF file's metadata and data with Python and xtopdf

Planet Python - Sun, 2014-09-14 01:42
By Vasudev Ram

DBF files (a.k.a. XBASE files), were one of the most widely used data formats for storing structured relational data on PCs, due to the fact that the original products that used DBF files, dBase II and III, were among the most successful database products of their time (the early personal computer era). DBF is probably still very widely used as a format in small and medium-sized desktop-based applications.

DBFReader.py, a program I wrote as part of my xtopdf toolkit for PDF creation using Python, can be used to read the contents of a DBF file, including both the metadata (file and field header information) and the data records, and display them on the screen.

Here is an example invocation of DBFReader.py:
python DBFReader.py test3.dbf | more
This command will read the metadata and data of the specified DBF file and display them on your screen. Here is the output from the above command:
File header :

key: rec_len value: 30
key: ver value: 245
key: hdr_len value: 193
key: last_update value: 02/11/04
key: num_flds value: 5
key: num_recs value: 4

Field headers :

num_flds = 5
Name | Type | Length | Decimals
FIELD1 | C| 5| 0
FIELD2 | N| 5| 0
FIELD3 | L| 1| 0
FIELD4 | D| 8| 0
FIELD5 | M| 10| 0

Data records:

(' ', ['AAAAA', '11111', 'F', '19010101', ' '])
(' ', ['BBBBB', '22222', 'T', '19020202', ' '])
(' ', ['CCCCC', '33333', 'F', '19030303', ' '])
(' ', ['DDDDD', '44444', 'T', '19040404', ' '])

- Vasudev Ram - Python consulting and training - Dancing Bison Enterprises

Contact Page

Share |
Vasudev Ram
Categories: FLOSS Project Planets

Tim Knapp: Surprise features that you didn't ask for - Mike Haworth

Planet Python - Sun, 2014-09-14 01:18

Server Side Request Forgery

  • Could access services on localhost
  • Access other hosts in DMZ
  • Bypass host-based authentication systems

Vulnerable systems:

  • memcached
  • couchdb

Urllib: accepts file://

Pycurl accepts gopher://

Memcached: gopher://

Preventing: blacklist is error prone.

XML External Entity (XXE):

Safer: http://pypi.python.org/pypi/diffusedxml

Signed cookies - session data is visible to user.

Also unless it’s got discount applied, just reapply discount. Can have a unique number that is issued each time we give discount.

Don’t use pickle as could execute shellcode but instead use JSON.

Pentesting challenges: http://tinyurl.com/h4ckit

Categories: FLOSS Project Planets

Tim Knapp: Automated Deployment with Ansible - Juergen Brendel

Planet Python - Sun, 2014-09-14 00:29

Configuration - system packages, settings, users, apps

CM Tools - describe the desired state. E.g. ensure all system packages are updated.

Puppet, chef - feel bit like working with Java.

Salt, Ansible - simple.

Ansible - orchestration engine. Written in Python. Uses YAML to write “playbooks”. Can use it to configure Windows.

No central configuration server/client (on target).

Just needs SSH access and Python on target machine.

Modules - 100s of them. They know how to do stuff.

Deletes module code from target after running it on target.

Can define groups to operate against.

group_vars directory/ all, groupname1

Ansible ‘cloud’ modules: AWS, Docker.

Modules: all output to stdout + JSON.


Categories: FLOSS Project Planets

Tim Knapp: Lightning Talks

Planet Python - Sun, 2014-09-14 00:00

LCA: 12-16 January 2015

Program online in next year or 2.

Singapore also put in proposal for future LCA.

6-700 attendees hoped.

Heritage Preserve #WeWantJam

Grabbed 30TBs of .co.nz internet.

2 days of catering, hacking.

If you’d like to do it again tweet #WeWantJam

Less Boilerplate

import argparse

Command line


Command line programs for busy developers.

Show us the world

rbenv - isolate different

pyenv shell 3.4.1

pyenv virtualenv name

pyenv local dirname

MySQL performance schema

Debugging schema for the database.

Can configure at runtime.

5.5 or 6 dbs.


Very pythonic. Stolen lots of good ideas from Python.

Compiled just in time. Timing a file that does nothing it takes 2.25s.

The Python Promotion Pamphlet

International Python Promotion Pamphlet. Glossy & stylised.

Creating NZ-version. Will have high-quality printable version and web-version.

Make your company more visible to Python programmers rather than pay recruitment agents.





When you subscribe to the mailing list (607 members on mailing list) you aren’t a member of the incorporated society.

Jessica McKellar’s Kiwi PyCon73,000 views. Far more than Guido McV’s keynote at PyCon US.

Thomi is commandeering LT and announcing my leaving committee and asking if someone can join NZPUG committee. See committee members if would like to join.

Docker for Python

Will do to IT industry what container box did to shipping industry.

LXC (Linux Lightweight Containers) + UnionFS.

Docker file is essentially like a bash script file of commands.

You can inherit from other docker files.

Fig - Vagrant for Docker.

Ship deltas not images.

Shipyard - 3rd party ecosystem.


  • Use docker-osx, PyCharm
Object Factories

Sole object is to create objects for testing purposes.

Don’t care about what object is. Just assign random attributes to objects.

Code Obfuscation

Weird bugs: object() > object()

If doing tests against time it will fail at midnight depending on your timezone.

False = True

Arduino + Thermal Printer + Python

Arduino sketch, Python script, Git


Dynamically creating Python tutorials and presentations

Massey Computer Science moved to Python in 2011.

Emacs: an operating system disguised as an editor.


Org-babel - generate code.


Bad-ass Postgres Tricks


  1. Template Databases - clone dbs
  2. Estimated Counts
  3. Hstore - values have to be strings
  4. Smart Indexes
  5. Schema Change Locks
  6. PL/Python
"One button" test and deploy on AWS

Use Vagrant+Ansible

Scripted creation of AMI and deployment.

Python Antipatterns

Better: list comprehension

Better: raise AssertionError

Better: if pattern in input_str

Don’t pass empty list as default are.

if 5 <= i < 10: print “something”

Better: return NotImplementedError

Map reduction using Python scripts

Script to create ArcMap maps.

Making weird maps with Python

Pulled data from census + meshblock boundaries (Koordinates.com) + map box (tile mill) + GDAL.

Should have used Fiona + Shapely.

Categories: FLOSS Project Planets

Tim Knapp: Multimedia programming using Gstreamer (and, of course, Python) - Douglas Bagnall

Planet Python - Sat, 2014-09-13 21:58

Pipelines consist of linked elements.


  • Link
  • Source
  • Sink (file, net, screen, etc)
  • Demux, tee

$ gst-launch-1.0 filesrc location=video.ogv ! oggdemux ! theoradec ! autovideosink

playbin - just play file = mplayer

Probably about 100 media players in world that are based on gstreamer

Muxer: combining 2 streams into 1 file.

Really good example - video wall.

Decode bin -> split into 4 streams (tee) -> queues (often freezes as all running in 1 thread and bump into eachother) -> crop (I.e. Remove 3/4s of video). And also play audio.

Can just use gstreamer cli syntax when calling gi calls in Python.

github: douglasbagnall


Categories: FLOSS Project Planets

Tim Knapp: Understanding human language with Python - Alyona Medelyan

Planet Python - Sat, 2014-09-13 21:30

Android Auto: hands-free operation via voice commands.

WordLense: “augmented reality translation”

"Her" movie in reality: "Siri will you marry me? Siri doesn’t seem to be available."

Segmentation complexities.

Disambiguation complexities. E.g. Flying planes can be dangerous.

Categories - taxonomy terms, entities - biological/chemical, sentiment.

NKTK - Python toolkit for NLP.

How to get to the core words? Remove stopwords.

from nltk.corpus import stopwords from nltk.tokenizie import word_tokenize

Keyword scoring: TFxIDF


from gensim import corpora, models

When ranking words can use score to discard them.

Text Categorisation with NLTK

Sentiment Analysis:

from textblob import TextBlob







Categories: FLOSS Project Planets

Tim Knapp: PyPy.js: What? How? Why? - Ryan Kelly

Planet Python - Sat, 2014-09-13 19:51


Mozilla - protecting & promoting the web.

The Web? Technology + Culture (open - don’t need to submit to ‘web pool’, ubiquitous, secure, trustworthy)

"For Mozilla, anything that the Web can’t do, or anything that the Web is not faster and better at than native technologies, is a bug. We should file it in our Bugzilla system, so we can start writing a patch to fix it" - Andreas Gal (Mozilla CEO)

var vm = new PyPyJS() import js js.eval()

An experiment in building a fast, compliant, in-browser Python environment.

PyPy.js is … Not so Fast. Incompatibilities between the JIT compilers.

Is … Humongous.

How? PyPy + Emscripten.

Emscripten: an LLVM backend that generates JavaScript.

"It’s awful all the way down"

Alts: http://GitHub.com/PythonJS

"Please, file bugs against the Web"



Categories: FLOSS Project Planets

Bruce Snyder: Discovering a New Normal :: Bruce Snyder's Status

Planet Apache - Sat, 2014-09-13 19:36
I know it has been a while since my last blog post, but I've been rather busy lately trying to find my way again. I know I have said this before, but having such a traumatic injury has really changed my life. It's not only the physical changes of being in a wheelchair, but there are many other things have been affected about my life and my family. Because I have not blogged in a while (shame on me), this post is full of a variety of topic swirling around in my head right now. Family Leaving Since the accident back in April, my family was here in our house helping us with everything. My parents were here most of the summer and Janene's mom Susan was here for four months straight. We have had numerous other family members visit, but these were the mainstays. As of a couple of weeks ago, everyone has left including Susan. At first, I kinda felt like my safety net had been removed. She was always here for me since the accident, willing to help me with anything. But I have quickly discovered that although I do need help with tasks, if I take my time I can do many things myself. For example, yesterday on my way home I ran a bunch of errands and had to make five trips in and out of the house to get everything out of the car. Prior to the accident, I could have gotten it all in a couple of trips. It's just another adjustment. Air Travel Back in August, I was able to take my first trip involving air travel. I only went for one night to my company's office in Chicago. Everything went very well, but it was not easy to sit on the plane for two hours without moving much. Unfortunately I wound up with a middle seat so I wasn't able to easily get up and out of my seat, but I did shift around as much as space would allow and I made it successfully. It really made me realize how much I move around during the day what with standing, shifting around in my wheelchair, sitting in other chairs, etc. This was a good test as a first trip involving air travel. I'm certainly not ready to travel to my company's headquarters in Munich, Germany yet, but I will eventually need to build back up to making this trip. Driving Although Janene and I have always vowed not to buy our girls a car when they each turn 16, almost dying in a tragic accident can really change your perspective on life. While I was in the hospital recovering, Janene told me that I was going to need to buy a new car because our oldest daughter Jade was going to be driving my car now. My response was essentially, 'But what about what we talked about and agreed upon?'. To which she responded, 'Bruce, you were nearly killed in this accident. Life is too fucking short to hold back on such things.' This definitely gave me pause and has really shaped how I look at life now. This is not to say that we have thrown out all reasoned arguments and discussions, more that we believe as parents we should be developing an environment of trust for our kids rather than holding things back from them. Also, just to put this into perspective, my old car is a 2001 Nissan Maxima with ~150k miles on it. So we're not talking about a care that is even close to being new. 
Anyway, one thing that has really changed is the way I drive a car now. Being that I'm no longer driving my car and I still need to get a new one, in the meantime, we decided to have hand controls installed in Janene's car. When I was in the hospital I learned to drive with hand controls. After over 25 years of driving a car using the foot pedals for the accelerator and brakes, because I have no movement in my feet, I now have to use hand controls. It's a handle on the left side of the steering wheel and is easy to use, it just takes a bit to get your muscle memory to begin thinking in terms of the hand control vs. the foot pedals. Using the handle, you push to brake and pull down to accelerate -- it's really that simple. There's also a removable metal foot plate that keeps my feet from bumping the real accelerator and brake.

Since mid-July, I have been able to drive myself to work every day. The great thing about driving my wife's car is that it's Toyota Highlander, an SUV. This means that I can put my wheelchair in the back of the car without breaking it down (i.e., taking the wheels off of it, etc.). Because I'm lucky enough to be able to stand and I can now walk with the braces on my legs if I have something to lean on, I have taught myself to lift the wheelchair into the back of the car and then make my way along the driver's side by hanging onto the car and kinda shuffling along. It has been a liberating feeling to be capable of driving myself to work and running some errands around town.
The Fucking Health Insurance IndustryI know this is a bit off-topic, but it has been incredibly frustrating to navigate the insurance industry bullshit. In my life, I have been lucky enough to be a very healthy person and rarely ever needed to utilize my health insurance. Since this accident, I have been extremely grateful to have good health insurance. However, I have also discovered that there seems to be some super secret list of things that health insurance does not cover or that they are somehow required by some fucking insurance credo to give you a difficult time about. There's nothing quite like a hospital hounding you personally for a bill of nearly $400k because your health insurance keeps stalling. We had to chalk up the $2200 installation fees for the hand controls as an out-of-pocket expense -- and that's just one of the many costs that come out of own own pockets. I'm extremely fortunate to have a very good attorney to help me navigate the insurance industry. As he has explained to me, insurance companies get stuck between their own policies and the laws of different states. All I know is that without my attorney, I would be going mad right now.
Walking With Arm Crutches Several weeks ago, I got some new leg braces. This might not sound very exciting but for me it is because they allow me to balance so well that I can stand in place without holding on to anything. Taking off walking is still out of the question right now because I'm still weak from the muscle I lost from being stuck in a hospital bed and a wheelchair. So for now, I'm able to walk using leg braces and balance with arm crutches. I certainly won't win any races, but it is such a wonderful thing to be six feet tall again. I realized this before and even blogged about it, but because I'm six feet tall when I stand, I'm actually taller than many people. But having been in a wheelchair for a while now means that I have been much shorter than everyone around me. I can now be eye-to-eye with people again while speaking with them which is very liberating. I can also hug my girls and Janene again without the wheelchair being in the way. I'm not walking full-time yet but I'm pushing myself to do it more and more and have been standing up for longer and longer periods of time throughout the day. As nice as it is to be my full height again, it still sucks to rely on the arm crutches because I can't carry anything. Sure, I can wear a backpack and take, say, my laptop from room to room, but I can't just carry a glass of water or a dish of food to the dinner table. Hopefully one day.

At least now I can stand more easily and I'm getting stronger. This means that I can begin to cook some things again. Something that I have always prided myself on is cooking for my family. For years now, I have been the primary cook in our house just due to Janene's work schedule. I have really learned to love cooking for my family over the years and it is something that I have missed since my accident. I'm not able to cook a huge meal yet, but smaller things here and there. I'm hopeful that with increased strength, this will get easier. I'm also enlisting Bailey (11) and Jade (16) to help me with many things, especially cooking. Discovering a New Normal For those of you who have been following my blog posts, you already know that my spine is now crooked. To date, I have been able to alleviate much of this problem by putting a lift in my right shoe. But whenever I'm not wearing shoes I am crooked again which is rather frustrating because it throws me off balance entirely. I hope that as I continue to build strength that my body will learn to compensate more and more. But this is just one of many small things that I am learning to live with now.

I also have paralysis across my butt and down the outsides of my hamstrings. This paralysis affects all of the gluteal muscles so that I cannot even squeeze the muscles in my butt. The result is that I have no muscle tone in my butt and this makes it very difficult to sit on certain surfaces for very long. So I need to be very careful about the surfaces I sit on and I cannot sit on most chairs, even if they are soft. On my wheelchair, I have a special cushion that contains multiple chambers of air. I also have a thinner more portable cushion that I can use on standard chairs and in the car. Even if the movement of my feet does not return, I really wish that I could regain the feeling in my butt and the ability to squeeze the muscles so that I could build them back up again.

Another difficulty is the difference with my bladder and bowels. When you have a spinal injury, depending on the level of the injury, it's very common to have changes in the function of your bladder and bowels. My situation could be a lot worse, but still the nerve damage has affected both bladder and bowel control. Luckily I can feel when my bladder is full or when I need to go. This is great because many people with spinal cord injuries cannot. Unfortunately I cannot pee on my own, so I must use a catheter to drain my bladder. Furthermore, my bowels are just not the same as before the accident so I just need to be more conscious of when I need to go. So I have had to learn to manage this whole new set of challenges. 
Janene did a wonderful job of installing various ramps around the house so that I can get around on the main level of our home, but I'm still not able to easily climb stairs. So we had to move our bedroom to the main level until I recover enough to get upstairs. We have set a goal for end of the year for me to get back to our master bedroom and bathroom on the second level of our home. This goal is about gaining back enough muscle to be able to climb the stairs more easily than I do now. At the moment, it seems like a far off goal, but I have been surprised at the changes I have seen in the last three months.
To address the rebuilding of muscle, I'm also looking into some new rehab options now. I feel like I've been sitting and waiting, for what I don't know. So I feel that it's a good time to ramp up my physical therapy to focus on rebuilding the muscle mass that I have lost. Not only will more rigorous workouts encourage more healing of the nerves, but it will also prepare my body for the time when movement does return. Instead of having some movement return and still being extremely weak, my hope is to help my body rebuild itself to where it was prior to the accident and be better prepared for the movement to come back.

I think the most difficult thing for me has been the amount of help I need with so many things now. Previous to my injury, I enjoyed helping others with various tasks and I was extremely independent with anything I did. Now I'm the one who needs help. This has really been difficult because I went from such extreme independence to a fair amount of dependence. My family and friends are wonderful about providing me the help I need, but I have never been someone who needed help. I always enjoyed providing help to others. Now the shoe is on the other foot.

I still have some not so good moments when my emotions get the best of me. I guess I haven't been meditating as much as I did when I was hospitalized and I need to change this right away. I guess in the hospital I had the time to do so and now, being back to work I don't have so much time available. It has been amazing to me that having gone through such a life change you kinda go through a grieving process for the things you have lost. Anyone who knows me knows how physically active I tend to be -- cycling, running, hiking and general fitness was a big part of my life. Needless to say, this is all on hold right now until I determine if I will regain movement in my feet or not. It's really a wait-and-see situation because nobody can predict what will happen. Everyone's injuries are unique and everyone's body heals differently. I'm still very hopeful that I will regain movement.

There are many other small things involved in my new normal. Although my life is full of new challenges, I still feel very fortunate that my situation is not worse. I try to be as independent as possible and I continue working to rebuild the strength that I have lost. 
Categories: FLOSS Project Planets

Tim Knapp: Deploying a Django application using Juju - Tim Penhey

Planet Python - Sat, 2014-09-13 19:11

Or "How I used Juju to deploy my Django app up to AWS"

Used backbone.js

Juju is written in Go. Service orchestration in the cloud.

  • Encapsulate application configurations
  • Define how services get deployed
Existing Charms
  • PostgreSQL
  • Python-Django
  • Gunicorn

Juju can help with scaling later.

Simple repeatable deployments.

Writing a subordinate charm.

Hooks are just scripts. Used Install hooks and Everything else hooks.

Promoted charms are only ones that are officially available.

"As a Juju core developer it was interesting that if I didn’t know the value that Juju would give me later, I would’ve thrown it out ages ago."

Insufficient documentation around using the Python-Django charm, had to read the code.

Insufficient documentation around how to write a payload charm for Python-Django, very much work it out as you go.

Python-Django and gunicorn still needs nginx to be really useful.

Should have a full stack Django charm.

juju bootstrap|deploy|add-relation|deploy|expose

Categories: FLOSS Project Planets

Tim Knapp: 2nd Keynote: A Snake in Space - The rise of scientific Python in Astrodynamics and Astronomy - Francesco Biscani

Planet Python - Sat, 2014-09-13 18:37

Max Planck Institute for Astronomy, European Space Agency.

PaGMO and Astropy - will mainly talk about PaGMO as wrote this and more familiar with this.

PaGMO - Parallel Global Multiobjective Optimiser. Optimisation tool. Initially a trajectory optimisation tool, evolved as a general-purpose optimiser.

Focused on parallel and distributed computing. Can use via C++ or Python.

Astropy - community effort to develop a single core package for Astronomy.

Optimisation - a large area of applied mathematics. “The selection of a best element (with regard to some criteria) from some set of available alternatives.” E.g. Travelling salesman (TSP).

E.g. algorithms: gradient-based methods, evolutionary algorithms, stochastic algorithms.

Interplanetary trajectories - space mission trajectories are defined by sets of parameters: launch date, initial velocity vector, sequence of flybys, sequence of deep-space manoeuvres (DSM).

Usually we want to minimise fuel consumption.

The resulting optimisation problem is hard: multimodal objective function, highly nonlinear, highly dimensional.

Traditionally tackled by teams of human experts.

Bio-inspired algorithms:
  • genetic algorithms
  • differential evolution
  • ant-colony optimisation
  • artificial bee-colony optimisation

Island model - name inspired by Darwin’s trip to the Galápagos Islands.

History of PaGMO - pattern of scientific programming code: created as part of some research and then abandoned for many years; not useable by anyone else and then picked up later and made more consumable by others.

Initially created Python bindings to initially created C/C++ ‘research’ code (2008-2009) and followed ‘eat own dog food’ approach by using it a lot for internal research. Been through 2 GSoCs and now ‘fully-fledged’ open source project.

Pros of Scientific Computing
  • Emphasis on correctness & reproducibility
  • Powerful driver for innovations in HPC
Cons of Scientific Computing
  • Wheel re-invention
  • Code is often written with a one-paper-horizon mindset
  • Most scientists are not trained in software engineering
PaGMO framework:
  • The abstract island class includes a problem, an algorithm and a population of candidate solutions, and a virtual evolve() method that dispatches the optimisation (to a thread or a process on another machine)

Implemented via Boost.Python.

Challenging, non-trivial issues:
  • serialisation across language boundaries involving virtual classes, base pointers, etc

  • extension from Python of C++ base classes

  • working around some of Python’s limitation wrt parallel programming (GIL)

Leveraging Python’s strengths:
  • Scientific Python stack: NumPy (crunching results), SciPy (optimisation algorithms), matplotlib, IPython, etc

PaGMO uses: evolution of neural networks for autonomous Martian rovers, selection of Near Earth Asteroids for future human missions.

Astropy - “a community package for Astronomers”:
  • Handle practical needs of astronomers: units, coordinates, FITS files, cosmological calculations

  • "One of best community packages ever seen"

  • Not research package like PaGMO

  • Heavy reliance on NumPy

Categories: FLOSS Project Planets
Syndicate content