Reinout van Rees: Pygrunn: IPython and MongoDB as big data scratchpads - Jens de Smit

Planet Python - Fri, 2015-05-22 04:34

(One of the summaries of the 2015 Pygrunn conference )

A show of hand: about half the people in the room have used mongodb and half used ipython notebooks. There's not a lot of overlap.

Jens de Smit works for optiver, a financial company. A "high-frequency trader", so they use a lot of data and they do a lot of calculations. They do a lot of financial transactions and they need to monitor if they made the right trades.

Trading is now almost exclusively done electronically. Waving hands and shouting on the trading floor at a stock exchange is mostly a thing of the past. Match-making between supply and demand is done centrally. It started 15 years ago. The volume of transactions really exploded. Interesting fact: the response time has gone from 300ms to just 1ms!

So... being fast is important in electronic trading. If you're slow, you trade at the wrong prices. Trading at the wrong prices means losing money. So speed is important. Just as making the right choices.

What he had to do is to figure out how fast an order was made and wether it was a good order. Non-intrusively. So: what market event did we react to? What was the automatic trade decision (done by an algorithm)? Was it a good one? How long did it all take?

So he monitors data going in and out of their system. He couldn't change the base system, so: log files, network data and an accounting database. Most of the data is poorly indexed. And a very low signal-to-noise ratio. And of course the logfiles aren't all consistent. And documentation is bad.

Oh, and the data size is of course also to big to fit in memory :-)

He used mongodb. A schemaless json (well, bson, binary version of json) store. Great for messy data. Easy to use. Just put in a python dictionary, basically. The data is persisted to disk, but as long as you have enough RAM, it'll keep it in memory. Very fast that way. You get indexes and speedups by default.

After he managed to get everything into mongodb, he had to make sense of things. So: correlate decision logs to network data. This is easy for humans to spot, but hard for computers. Computers are good at exact matches, humans are better at inexact pattern matches.

He used ipython notebook, a nice interactive python shell with a browser interface. Including matplotlib integration for easy graphs. Syntax highlighting; you can render html inside the shell; you can save your work at the end of the day (which you can't with a regular python shell!); inline editing.

Nice: since last week, rendering such notebooks is supported by github. (I guess he means this announcement ).

Now mongodb. It is very simple to create a directory and start mongodb. If you stop mongo and delete the directory, it is gone as if it was never there. Easy. And with pymongo it is just a few lines of python code and you're set. Including a handy query language.

He showed a couple of code examples. Looked pretty handy.

Creating an index is a oneliner. If you know beforehand what kinds of queries you want to do, you can quickly create an index for it, which speeds up your queries a lot. You can make complex indexes, but in his experience, simple single-field indexes are often enough.

Something to watch out for: mongo does never return disk space to the OS. If you delete lots of objects, the OS doesn't get it back unless you shut mongodb down and "repair" the database. What he does is simply delete the database at the end of the day!

He showed one of the outputs: a graph with response times which immediately showed that several responses were too slow. Good, useful information. One year ago he wouldn't have dreamt of being able to do this sort of analysis.

Mongo is very useful for this kind of work. You use mongodb's strengths and you aren't bothered by many of the drawbacks, like missing transactions.

Categories: FLOSS Project Planets

gxmessage @ Savannah: gxmessage 2.20.2 for GTK2 has been released

GNU Planet! - Fri, 2015-05-22 04:33

gxmessage is a GTK based alternative to xmessage.

This is a maintenance and minor bug-fix release of the GTK2 based version of gxmessage.

Also, I'm pleased to announce a new mailing list for gxmessage users:


Categories: FLOSS Project Planets

Michal Čihař: Weblate 2.3

Planet Debian - Fri, 2015-05-22 04:00

Weblate 2.3 has been released today. It comes with better features for project owners, better file formats support and more configuration options for users.

Full list of changes for 2.3:

  • Dropped support for Django 1.6 and South migrations.
  • Support for adding new translations when using Java Property files
  • Allow to accept suggestion without editing.
  • Improved support for Google OAuth2.
  • Added support for Microsoft .resx files.
  • Tuned default robots.txt to disallow big crawling of translations.
  • Simplified workflow for accepting suggestions.
  • Added project owners who always receive important notifications.
  • Allow to disable editing of monolingual template.
  • More detailed repository status view.
  • Direct link for editing template when changing translation.
  • Allow to add more permissions to project owners.
  • Allow to show secondary language in zen mode.
  • Support for hiding source string in favor of secondary language.

You can find more information about Weblate on http://weblate.org, the code is hosted on Github. If you are curious how it looks, you can try it out on demo server. You can login there with demo account using demo password or register your own user.

Weblate is also being used https://hosted.weblate.org/ as official translating service for phpMyAdmin, Gammu, Weblate itself and other projects.

If you are free software project which would like to use Weblate, I'm happy to help you with set up or even host Weblate for you.

Further development of Weblate would not be possible without people providing donations, thanks to everybody who have helped so far!

PS: The roadmap for next release is just being prepared, you can influence this by expressing support for individual issues either by comments or by providing bounty for them.

Filed under: English phpMyAdmin SUSE Weblate | 0 comments

Categories: FLOSS Project Planets

Reinout van Rees: Pygrunn: Leveraging procedural knowledge - K Rain Leander

Planet Python - Fri, 2015-05-22 03:45

(One of the summaries of the 2015 Pygrunn conference )

K Rain Leander works at Red Hat and yes, she wore a bright red hat :-) She's a python and django newbie. She knows how it is to be a newbie: there is so much in linux that there are always areas where you're a complete newbie. So everyone is helpful there.

"Amsterdam is the capital of the netherlands" is declarative knowledge. Procedural knowledge is things like learning to ride a bike or a knew language. So: What versus How. You might know declaratively how to swim, but procedurally you might still drown: you need to practice and try.

Some background: she was a dancer in the USA. Unless you're famous, you barely scrape by financially. So she started teaching herself new languages. Both real-life languages and computer languages. Css, html for starters. And she kept learning.

She got a job at Red Hat. You have to pass a RHCE certification test within 90 days of starting work there - or you're fired. She made it. She

She has military background. In bootcamp, the purpose is not the pushups and the long runs. The goal is to break you down so that you jump when they say "jump".

In the Red Hat bootcamp, the goal is not making the test. The goal is to figure out if you're able to drink from the firehose. Which means if you get a support request, you say "I'll figure it out for you" and you just dive in and try to figure it out. You have to be able to dive into a whole lot of new information without panicking. That's drinking from the firehose.

She re-used existing knowledge and previous skills to learn everything. The important part was not being afraid to dive in.

She moved towards programming. Python, django. She was new to it. One of the first steps? "Set up a virtualenv and....". It can frighten you, but it is just a question of RTFM. Just read the manual. Just read it and then start doing it.

She went to a Django Girls Workshop. (One of the results: http://leanderthalblog.herokuapp.com/). Django girls does a really good job of providing material and documentation. She had some problems installing it, but continued (and succeeded) anyway.

... and then someone challenged her to deploy it on openshift. http://django-leanderthal.rhcloud.com/ It hasn't succeeded completely yet. But she'll persevere and get it working.

She recommends http://learnpythonthehardway.org/ to learn python.

What's next: she'll practice, practice, practice. And she'll contribute to the community. Probably build one or two apps. And she'll be a coach at the upcoming Groningen django girls workshop ("as a coach. No, I'm not worried....")

So: re-use your existing knowledge and build from there. Don't be afraid. Just do it.

Categories: FLOSS Project Planets

Second stretchgoal reached and new builds!

Planet KDE - Fri, 2015-05-22 03:38

We’ve got our second stretchgoal through both Kickstarter and the Paypal donations! We hope we can get many more so that you, our users, get to choose more ways for us to improve Krita. And we have got half a third stretch goal actually implemented: modifier keys for selections!

Oh — and check out Wolthera’s updated brush packs! There are brush packs for inking, painting, filters (with a new heal brush!), washes, flow-normal maps, doodle brushes, experimental brushes and the awesome lace brush in the SFX brush pack!

We’ve had a really busy week. We already gave you an idea of our latest test-build on Monday, but we had to hold back because of the revived crash file recovery wizard on windows… that liked to crash. But it’s fixed now, and we’ve got new builds for you!

So what is exactly new in this build? Especially interesting are all the improvements to PSD import/export support. Yesterday we learned that Katarzyna uses PSD as her working format when working with Krita – we still don’t recommend that, but it’s easier now!

Check the pass-through switch in the group layer entry in the layerbox!

  • Dmitry implemented Pass-Through mode for group layers. Note: filter, transform and transparency masks and pass-through mode don’t work together yet, but loading and saving groups from and to PSD now does! Pass-through is not a fake blending mode as in Photoshop: it is a switch on the group layer. See the screenshot!
  • We now can load and save layerstyles, with patterns from PSD files! Get out your dusty PSDs for testing!
  • Use the right Krita blending mode when a PSD image contains Color Burn.
  • Add Lighter Color and Darker Color blending modes and load them from PSD.
  • When using Krita with a translation active on windows, the delay on starting a stroke is a bit less, but we’re still working on eliminating that delay completely.
  • The color picker cursor now shows the currently picked and previous color.
  • Layer styles can now be used with inherit-alpha
  • Fix some issues with finding templates.
  • Work around an issue in the oxygen widget style on Linux that would crash the OpenGL-based canvas due to double initialization
  • Don’t toggle the layer options when right-clicking on a layer icon to get the context menu (patch by Victor Wåhlström)
  • Update the Window menu when a subwindow closes
  • Load newer Photoshop-generated JPG files correctly by reading the resolution information from the TIFF tags as well. (Yes, JPG resolution is marked in the exiv metadata using TFF tags if you save from Photoshop…)
  • Show the image name in the window menu if it hasn’t been saved yet.
  • Don’t crash when trying to apply isolate-layer on a transform mask
  • Add webp support (at least on Linux, untested on Windows)
  • Add a shortcut to edit/paste into a new image. Patch by Tiffany!
  • Fix the autosave recovery dialog on Windows for unnamed autosaves!
  • Added a warning for intel users who may still be dealing with the broken driver. If Krita works fine for you, just click okay. If not, update your drivers!

New builds for Linux are being created at the moment and will be available through the usual channels.

Linux: Windows:

From Vista and up, Windows 7 and up is recommended. There is no Windows XP build. If you have a 64 bits version of Windows, don’t use the 32 bits build! The zip files do not need installing, just unpacking, but do not come with the Visual Studio C runtime that is included in the msi installer.


(Please keep in mind that these builds are unstable and experimental. Stuff is expected not to work. We make them so we know we’re not introducting build problems and to invite hackers to help us with Krita on OSX.)

Categories: FLOSS Project Planets

Chris Mitchell: Minimizing render times of shared Django forms

Planet Python - Fri, 2015-05-22 03:24
A common situation with Django sites is the need to render a given form across all pages, such as a login-form that is embedded in the header. There is a recipe I came upon, probably from stackoverflow, that has some derivation of the following pattern:

# as a context_processor
from .forms import SomeLoginForm

def loginFormProcessor(request):
ctx = {}
if not request.user.is_authenticated():
ctx['login_form'] = SomeLoginForm
return ctx

# your template
{% if not request.user.is_authenticated %}
{% crispy login_form %}
{% endif %}

I was using this pattern for a rather complicated form without thinking about the overhead incurred. However, when new-relic revealed this was taking ~600 ms per render, I knew it had to be fixed.

 The simplest solution is template caching, making our template look like so:

# your template
{% load cache %}
{% if not request.user.is_authenticated %}
{% cache 99999 login_form_cache %}
{% crispy login_form %}
{% endcache %}
{% endif %}

The problem with this is we still incur the overhead in our context processor. We can avoid this by doing all our work within the cache tag. First, we need to move the logic of generating the form out of the context processor and into a template_tag.

# our template_tag.py file
def get_login_forms():
from ..forms import StepOne, StepTwo, StepThree
ctx = {}
ctx['first'] = StepOne
ctx['second'] = StepTwo
ctx['third'] = StepThree
return Context(ctx)

Now, we need to integrate this tag into our text, so our final template looks like the following (this is also more related to my particular example where I have a multi-stepped form):

# our template file
{% load cache our_tags %}
{% if not request.user.is_authenticated %}
{% cache 99999 login_form_cache %}
{% get_login_forms as modal_login_forms %}
{% crispy modal_login_forms.first %}
{% crispy modal_login_forms.second %}
{% crispy modal_login_forms.third %}
{% endcache %}
{% endif %}

This alone made the server response time come from ~2-3  seconds down to 0.69 seconds.  Not too shabby.

Note: This code should run but I didn't test it as it isn't exactly my code copy & pasted, but an example.
Categories: FLOSS Project Planets

LevelTen Interactive: How To Fine Tune Drupal's Editorial Controls with Workbench Access

Planet Drupal - Fri, 2015-05-22 01:00

If you're building a Drupal website with a lot of content for a community of users, chances are you'll need to set up some editorial controls. Starting with the Workbench and Workbench Moderation modules, you can create editorial workflows for content types. Nodes pass through different 'States', like Draft, Needs Review, and Published. Different User Roles control the flow of nodes through these different states.... Read more

Categories: FLOSS Project Planets

Dries Buytaert: Why WooMattic is big news for small businesses

Planet Drupal - Thu, 2015-05-21 23:38

Earlier this week Matt Mullenweg, founder and CEO of Automattic, parent company of WordPress.com, announced the acquisition of WooCommerce. This is a very interesting move that I think cements the SMB/enterprise positioning between WordPress and Drupal.

As Matt points out a huge percentage of the digital experiences on the web are now powered by open source solutions: WordPress, Joomla and Drupal. Yet one question the acquisition may evoke is: "How will open source platforms drive ecommerce innovation in the future?".

Larger retailers with complex requirements usually rely on bespoke commerce engines or built their online stores on solutions such as Demandware, Hybris and Magento. Small businesses access essential functions such as secure transaction processing, product information management, shipping and tax calculations, and PCI compliance from third-party solutions such as Shopify, Amazon's merchant services and increasingly, solutions from Squarespace and Wix.

I believe the WooCommerce acquisition by Automattic puts WordPress in a better position to compete against the slickly marketed offerings from Squarespace and Wix, and defend WordPress's popular position among small businesses. WooCommerce brings to WordPress a commerce toolkit with essential functions such as payments processing, inventory management, cart checkout and tax calculations.

Drupal has a rich library of commerce solutions ranging from Drupal Commerce -- a library of modules offered by Commerce Guys -- to connectors offered by Acquia for Demandware and other ecommerce engines. Brands such as LUSH Cosmetics handle all of their ecommerce operations with Drupal, others, such as Puma, use a Drupal-Demandware integration to combine the best elements of content and commerce to deliver stunning shopping experiences that break down the old division between brand marketing experiences and the shopping process. Companies such as Tesla Motors have created their own custom commerce engine and rely on Drupal to deliver the front-end customer experience across multiple digital channels from traditional websites to mobile devices, in-store kiosks and more.

To me, this further accentuates the division of the CMS market with WordPress dominating the small business segment and Drupal further solidifying its position with larger organizations with more complex requirements. I'm looking forward to seeing what the next few years will bring for the open source commerce world, and I'd love to hear your opinion in the comments.

Categories: FLOSS Project Planets

Python Sweetness: Block Range (BRIN) Indexes in PostgreSQL 9.5

Planet Python - Thu, 2015-05-21 21:56

After reading that PostgreSQL 9.5 will support BRIN indexes, it took me a couple of re-reads of the attached documentation to understand exactly what this index technique is about. Actually, it’s really simple, but for people like me who prefer things to be spelled out, here are some hopefully useful (and at least somewhat accurate) notes.


As a quick recap, table rows in PostgreSQL are stored into an on-disk structure known as the heap. The heap is an array that is logically partitioned into 8kb “pages”, with each page containing one or more “tuples” (rows). To ease management, as the heap grows it is additionally split into 1GB-sized files on disk, however the overall structure is still essentially just one big logical array.

When you ask PostgreSQL to insert a row into a table, it uses an auxilliary structure known as the free space map to locate the first available heap page for your relation (“table”) that has sufficient space to store the data for your row. If your row is larger than a pre-set limit (2kb), large columns are split out of the row’s data and stored in a series of rows in an internal table (the so-called TOAST tables).

The net result is that each data row exists entirely within one page, and that row lives at a particular logical index (the “item ID”) within its page. If PostgreSQL must refer to a row, it can uniquely identify it using just its page number, and its index within the page. The combination of this pair of numbers is known as the row’s ctid, or its tuple ID. Tuple IDs can thus be used as a small, efficient, unique locator for every row in a database, and they exist regardless of your schema design.

[Side note: that’s not entirely true! If a row has been updated since the database was last VACUUMed, multiple versions will exist, chained together using some special fields in each version’s on-disk data. For simplicity let’s just assume only one version exists.]

In the current PostgreSQL implementation, 32 bits are used for the page number, and 16 bits for the item number (placing an absolute upper bound on a single database table to somewhere around 32 PiB), allowing the ctid to fit comfortably in 64 bits.

Using just the name of a relation and a ctid, PG can first split the page number from the ctid and use that to efficiently locate the physical database file and offset where the page lives:

page_size = 8KiB pages_per_segment = 1GiB / page_size segment, index = divmod(page_number, pages_per_segment) page_offset = page_size * index

Finally to locate the tuple within the page, a small, constant-sized lookup table exists at the start of each page that maps its item IDs to byte offsets within the page:

item_offset = page.lookup_table[item_id]


Without further help, answering a query such as SELECT * FROM person WHERE age BETWEEN 18 AND 23 would require PG to visit every page in the heap, decoding each row in turn, and comparing its age column to the WHERE predicate. Naturally for larger tables, we prefer to avoid that, and an index is necessary to allow PostgreSQL to avoid scanning the full table.

Btree Indexes

The most common index type in PG is the btree, which maintains an efficient map from column value to ctid. Given the imaginary table:

Person table heap layout Page Number Item ID ctid Name Age Creation Date 1 1 (1, 1) John 10 1998-01 2 (1, 2) Jack 99 1998-02 3 (1, 3) Jill 70 1998-03 4 (1, 4) Jemma 19 1998-04 2 1 (2, 1) George 60 1998-05 2 (2, 2) James 44 1998-05 3 (2, 3) Jocelyn 55 1998-06 4 (2, 4) Jemima 22 1998-07 3 1 (3, 1) Jerry 60 1999-01 2 (3, 2) Jarvis 44 1999-02 3 (3, 3) Jasper 55 1999-03 4 (3, 4) Josh 24 1999-04 4 1 (4, 1) Jacob 60 2000-01 2 (4, 2) Jesse 44 2000-02 3 (4, 3) Janet 55 2000-03 4 (4, 4) Justine 24 2000-04

A btree index created using CREATE INDEX person_age ON person(age) might resemble:

person(age) btree index layout Age ctid 10 (1, 1) 19 (1, 4) 22 (2, 4) 24 (3, 4) 24 (4, 4) 44 (3, 2) 44 (4, 2) 44 (2, 2) 55 (3, 3) 55 (4, 3) 55 (2, 3) 60 (3, 1) 60 (4, 1) 60 (2, 1) 80 (1, 3) 99 (1, 2)

This is getting too long already, so skipping to the chase we can see that PG can now efficiently locate an exact row given its associated indexed column value, and that value in turn is stored in a data structure that permits fast lookup.

For our SELECT query from above, PG can jump to btree key 18 and scan out ctids until it reaches a key with an entry larger than 23. In the demo table, this means PG must only visit 2 rows from our set of 16, and prior to accessing the row data, it already knows the row definitely matches the predicate.

For some other queries, such as SELECT COUNT(*) FROM person WHERE age = 22, PG may not even need to visit the row data itself, since it can infer from index entries how many data rows exist. [Another MVCC caveat! This is not entirely true, since index entries may exist pointing to deleted rows, or rows created in later transactions]

The crucial point to note, though, is that one exact index entry is produced for every row, which usually doesn’t amount to much, maybe no more than 5-15% overhead relative to the source table, however for a large table, that overhead may be the difference between a dataset that fits in RAM, and one in which common queries end up hitting disk, or IO is doubled due to index access, since the dataset was already vastly larger than available RAM. It’s easy to imagine indexes quickly adding up, such that perhaps half of an application’s storage is wasted on them.

BRIN Indexes

Finally enough verbiage is spilled so that we can reach the point: BRIN indexes introduce a cool tradeoff where instead of covering individual rows, index entries cover one or more heap pages:

person(age) BRIN index with group size 1 Page Number Has NULL values? Lowest Age Highest Age 1 No 10 99 2 No 22 60 3 No 24 60 4 No 24 60

The structure is used like so: given a query such as SELECT * FROM person WHERE age BETWEEN 10 AND 15, PG will visit every index entry in turn, comparing its minimum/maximum values against the query predicate. If the index entry indicates that a range of pages contains at least one record matching the query, those pages will be scanned for matching rows. For this query, only one page contains rows whose age fields overlap the desired region, and so PG can avoid visiting 75% of the table.

Notice that in order to find just one row, PG must now scan a full page and compare each of its 4 rows against the query predicates. While index size is reduced, query time has increased! There is also little pattern in our age column: in fact, it is quite lucky that our index described only a single page covering the range 10..15. Had users signed up in a slightly different order, the distribution of ages across physical storage pages may have resulted in PG having to scan many more pages.

[Another side note: unlike our dummy table above, a typical PG heap page may contain over 200 rows, depending on how many columns are present, and how many of those are strings. Our dummy BRIN index above looks as if it contains just as much information as the original btree index, but that’s just because my example only has 16 rows instead of 800].

BRIN also permits configuring how many heap pages contribute to an index entry. For example, we can halve the size of our first index while also halving its precision:

person(age) BRIN index with group size 2 Page Number Has NULL values? Lowest Age Highest Age 1-2 No 10 99 3-4 No 24 60

Due to the work-increasing factor, and also since every index entry must be visited (resulting in a potentially high fixed cost for any query), BRIN is probably never useful for “hot” queries against a table, or even much use at all in a typical “hot” database, however for auxilliary queries, such as producing once-per-month reports or bulk queries against archival data, where reduced runtime or IO is desired without the storage costs of an exact index, BRIN may be just the tool.

Finally, notice in the original table how as new records were inserted, their creation date roughly tracked which database page they ended up on. This is quite a natural outcome since as the table grows, newer items will occupy later pages in the array, and so there is quite a reliable correlation between page number and the creation date column value. A BRIN index over this column would work very well.

This was supposed to be a 5 minute “hey, that’s cool!” post, but somehow I suck at keeping these things short. I found the source code documentation the best explanation for how this stuff works; the public wiki is pretty vague. If you have corrections, pump the Ask Me Anything link to the right of this page.

Categories: FLOSS Project Planets

Vasudev Ram: Talk Python To Me podcast; Australia mandates text-based programming

Planet Python - Thu, 2015-05-21 20:52
By Vasudev Ram

Today, for a change, a different kind of Python post, but one that I think will be interesting to my readers:

From the horse's, er, snake's mouth :)

Talk Python to Me "is a weekly podcast hosted by Michael Kennedy. The show covers a wide array of Python topics as well as many related topics (e.g. MongoDB, AngularJS, DevOps)."

The format is a casual 30 minute conversation with industry experts.

I just came across it today, and am checking out the site a bit. The site itself looks good, visually, I mean.

Some of the podcasts also have text transcripts.

I'm reading one of the transcripts, Transcript for Episode #8, of the conversation with Dr. James Curran:

Teaching Python at Grok Learning and Classrooms


"James Curran is an associate professor in computer science at the University of Sidney and co-founder of Grok Learning, which you can find at groklearning.com. James has been teaching computer science to students and teachers for over a decade. In 2010 he was named one of Sidney magazines top 100 influential people for his work in computer science education."

(I guess the 'Sidney' spelling is due to an error in the automated or manual transcription of the podcast.)

Anyway, the transcript is interesting, since is about Python and/in education / training, which are both among my interests.

An interesting point mentioned in the talk, is that Australia is mandating text-based computer programming in its schools, as opposed to only having visual programming with tools like Scratch. Methinks that is a good idea, since only being able to work with GUI's and no skill with text-based tools (such as text editors and the command line), is not a good thing, IMO. I've come across, and once interviewed (for a client), some Java "programmers" who could not write a simple Java program without reaching for Eclipse. What the hell. Not knocking Java. I'm sure there must be people like that for other languages too. [ Dons latest flame shield made of modern composites in advance ... :) ]

Dr. Curran on the topic:

"So we have done a lot of work with teaching students in seventh and eighth grade, and I think that that is the ideal zone in fact the Australian curriculum that I was just involved in writing has mandated that the kids will learn a text based programming language as opposed to something like visual language, like Scratch. So text based programming is mandated for year seventh and eighth, and Python I think is the ideal language to be teaching there."

- Vasudev Ram - Online Python training and programming

Dancing Bison Enterprises

Signup to hear about new products or services that I create.

Posts about Python  Posts about xtopdf

Contact Page

Share |

Vasudev Ram
Categories: FLOSS Project Planets

DrupalCon News: The PM Track: What it’s about and how to get your session picked +Bonus 40 ideas for sessions you can steal!

Planet Drupal - Thu, 2015-05-21 20:06

If you’re anything like me, right now you’re thinking: Finally! It’s a very exciting moment for those in our field who have craved ways to collaborate, learn from experiences and refine our craft. The Drupalcon team has heard our request loud and clear, and we can now enjoy the very first Project Management Track!

Categories: FLOSS Project Planets

Pixelite: Top 10 DrupalCon LA sessions

Planet Drupal - Thu, 2015-05-21 20:00
DrupalCon LA

So I did not make it along to DrupalCon Los Angeles, but I did spend some time reading twitter, and watching the sessions online. Here are some of the sessions I found entertaining and insightful and would recommend to others.

Driesnote Keynote

Dries, as always, sets the lay of the land with Drupal. He also goes into the early days of Drupal, and how some key people he was involved with and have now gone on to form organisations that centre around Drupal.

Best quote:

Obstacles don’t block the path, they are the path


Larry Garfield gives an interesting talk on why sometimes it is best to say NO in order to give focus to the things that actually matter.

Best quote:

Case and point, the new Macbook Airs, they say NO TO EVERYTHING.

PHP Containers at Scale: 5K Containers per Server

David Strauss explains the history of web hosting, and how this is now far more complex. David is CTO of Pantheon, and they now run 100,000+ websites, all with dev + test + production environments. Pantheon run 150+ containers on a 30GB box (205MB each on average). Really interesting talk on how to run large amounts of sites efficiently.

Decoupled Drupal: When, Why, and How

Amitai Burstein and Josh Koenig give a really entertaining presentation on monolithical architectures and some developer frustrations. And then introduce REST web services in Drupal 8, and how this can be used to provide better consumer interfaces for other frameworks.

Features for Drupal 8

Mike Potter goes through what role features played in Drupal 7, and how features will adapt in Drupal 8 now that CMI is in. Features in Drupal 8 will be going back to it’s roots and provide ‘bundles’ of configuration for re-use.

Meet Commerce 2.x

Ryan and Bojan go through 1.x on Drupal 7, and how they have chosen to develop Commerce 2.x on Drupal 8. This is a complete rewrite. The hierarchical product model is really exciting.

How, When and Why to Patch a Module

Joshua Turton goes over what a patch is, when you should patch contributed modules, and how to keep track of these with Drush make.

My colleague Josh also wrote a blog post on how to use Drush make.

CI for CSS: Creating a Visual Regression Testing Workflow

I topic that I am passionate about is visual regressions, here Kate Kligman goes through some tools that can help you test your site for visual changes. Tools covered include PhantomJS, SlimerJS, Selenium, Wraith.

Speeding up Drupal 8 development using Drupal Console

Eduardo and Jesus give us an introduction to your new best friend in Drupal 8. Drupal console is a Symfony CLI application to help you write boilerplate code, e.g. to create a new module. Personally, I am excited for the form API generator, and the ability to create a new entity with a single command.

For more information see drupalconsole.com.

Q&A with Dries

As Drupal heads down from 130 critical issues down to 22 currently, what are some key concerns by people. The questions are answered by dries, xjm, webchick and alexpott.

Where can I find more videos

Don’t worry there are plenty more videos on the Drupal Association Youtube page.


If you have any awesome sessions that I have missed let me know in the comments.

Categories: FLOSS Project Planets

Updates on Kate's Rust plugin, syntax highlighting and the Rust source MIME type

Planet KDE - Thu, 2015-05-21 19:17
KDE Project:

The other day I introduced a new Rust code completion plugin for Kate, powered by Phil Dawes' nifty Racer. Since then there's been a whole bunch of additional developments!

New location

Originally in a scratch repo of mine, the plugin has now moved into the Kate repository. That means the next Kate release will come with a Rust code completion plugin out of the box! (Though you'll still need to grab Racer yourself, at least until it finds its way into distributions.)

For now the plugin still works fine with the stable release of Kate, so if you don't want to build all of Kate from git, it's enough to run make install in addons/rustcompletion in your Kate build directory.

This also means the plugin is now on bugs.kde.org - product kate, component plugin-rustcompletion (handy pre-filled form link). And you can submit patches via ReviewBoard now.

New feature: Go to Definition

In addition to code completion popups, the plugin now also installs Go to Definition action (in the Edit menu, the context menu, and you can configure a keyboard shortcut for it as well). It will open the document containing the definition if needed, activate its view and place the cursor at the start of the definition.

Rust syntax highlighting now bundled with Frameworks

After brainstorming with upstream, we decided together that it's best for Rust and Kate users to deprecate the old rust-lang/kate-config repository and move the syntax highlighting file into KDE's KTextEditor library (the foundation of Kate, KDevelop and several other apps) for good, where it now resides among the many other rules files. With 1.0 out the door, Rust is now stable enough that delivering the highlighting rules via distro packages becomes feasible and compelling, and moving the development location avoids having to sync multiple copies of the file.

The full contribution history of the original repo has been replayed into ktexteditor.git, preserving the record of the Rust community's work. The license remains unchanged (MIT), and external contributions remain easy via ReviewBoard or bugs.kde.org.

KTextEditor is a part of KDE's Frameworks library set. The Frameworks do monthly maintenance releases, so keeping up with the Rust release cadence will be easy, should the rules need to be amended.

It's a MIME type: text/rust

Kate plugins and syntax highlighting files preferably establish document identity by MIME type, as do many other Linux desktop applications. The desktop community therefore maintains a common database in the shared-mine-info project. With the inclusion of a patch of mine on May 18th, shared-mime-info now recognizes the text/rust type for files matching a *.rs glob pattern.

If you're searching, opening or serving Rust source files, you should be using text/rust from now on.

That's it for today! I still have a bunch of improvements to the plugin planned, so stay tuned for future updates.

Categories: FLOSS Project Planets

Dirk Eddelbuettel: BH release 1.58.0-1

Planet Debian - Thu, 2015-05-21 18:50

A new released of BH is now on CRAN. BH provides a large part of the Boost C++ libraries as a set of template headers for use by R and Rcpp.

This release both upgrades the version of Boost to the current release, and adds a new library: Boost MultiPrecision .

A brief summary of changes from the NEWS file is below.

Changes in version 1.58.0-1 (2015-05-21)
  • Upgraded to Boost 1.58 installed directly from upstream source

  • Added Boost MultiPrecision as requested in GH ticket #12 based on rcpp-devel request by Jordi Molins Coronado

Courtesy of CRANberries, there is also a diffstat report for the most recent release.

Comments and suggestions are welcome via the mailing list or the issue tracker at the GitHubGitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

parallel @ Savannah: GNU Parallel 20150522 ('Nepal') released [stable]

GNU Planet! - Thu, 2015-05-21 18:48

GNU Parallel 20150522 ('Nepal') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/

No new functionality was introduced so this is a good candidate for a stable release.

Haiku of the month:

Many jobs to run.
Idle cores is not your thing.
Use GNU Parallel.
-- Ole Tange

New in this release:

  • Security: After further security analysis the issue fixed in 20150422 also fixed the problem for --tmux.
  • Bug fixes and man page updates.

GNU Parallel - For people who live life in the parallel lane.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job is can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with: (wget -O - pi.dk/3 || curl pi.dk/3/) | bash

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). Your commandline will love you for it.

When using programs that use GNU Parallel to process data for publication please cite:

O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.

If you like GNU Parallel:

  • Give a demo at your local user group/team/colleagues
  • Post the intro videos on Reddit/Diaspora*/forums/blogs/ Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
  • Get the merchandise https://www.gnu.org/s/parallel/merchandise.html
  • Request or write a review for your favourite blog or magazine
  • Request or build a package for your favourite distribution (if it is not already there)
  • Invite me for your next conference

If you use GNU Parallel for research:

  • Please cite GNU Parallel in you publications (use --bibtex)

If GNU Parallel saves you money:


GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

Categories: FLOSS Project Planets

Mediacurrent: Mediacurrent Dropcast: Episode 5

Planet Drupal - Thu, 2015-05-21 16:57

Our first foray into public during Drupalcon Los Angeles. Bob, Jason and Mark are live interviewing anyone who showed up to our BOF (Birds of a Feather) and gave away fancy Weekly Drop T-shirts. We also talked about our favorite sessions in this years North American ‘Con. Special thanks to Benztown Radio for the use of their equipment.

Categories: FLOSS Project Planets

Yves-Alexis Perez: Followup on Debian grsec kernels for Jessie

Planet Debian - Thu, 2015-05-21 16:36

So, following the previous post, I've indeed updated the way I'm making my grsec kernels.

I wanted to upgrade my server to Jessie, and didn't want to keep the 3.2 kernel indefinitely, so I had to update to at least 3.14, and find something to make my life (and maybe some others) easier.

In the end, like planned, I've switched to the make deb-pkg way, using some scripts here and there to simplify stuff.

The scripts and configs can be found in my debian-grsec-config repository. The repository layout is pretty much self-explaining:

The bin/ folder contains two scripts:

  • get-grsec.sh, which will pick the latest grsec patch (for each branch) and applies it to the correct Linux branch. This script should be run from a git clone of the linux-stable git repository;
  • kconfig.py is taken from the src:linux Debian package, and can be used to merge multiple KConfig files

The configs/ folder contains the various configuration bits:

  • config-* files are the Debian configuration files, taken from the linux-image binary packages (for amd64 and i386);
  • grsec* are the grsecurity specifics bits (obviously);
  • hardening* contain non-grsec stuff still useful for hardened kernels, for example KASLR (cargo-culting nonwidthstanding) or strong SSP (available since I'm building the kernels on a sid box, YMMV).

I'm currently building amd64 kernels for Jessie and i386 kernels will follow soon, using config-3.14 + hardening + grsec. I'm hosting them on my apt repository. You're obviously free to use them, but considering how easy it is to rebuild a kernel, you might want to use a personal configuration (instead of mine) and rebuild the kernel yourself, so you don't have to trust my binary packages.

Here's a very quick howto (adapt it to your needs):

mkdir linux-grsec && cd linux-grsec git clone git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git git clone git://anonscm.debian.org/users/corsac/grsec/debian-grsec-config.git mkdir build cd linux-stable ../debian-grsec-config/bin/get-grsec.sh stable2 # for 3.14 branch ../debian-grsec-config/bin/kconfig.py ../build/.config ../debian-grsec-config/configs/config-3.14-2-amd64 ../debian-grsec-config/configs/hardening ../debian-grsec-config/configs/grsec make KBUILD_OUTPUT=../build -j4 oldconfig make KBUILD_OUTPUT=../build -j4 deb-pkg

Then you can use the generated Debian binary packages. If you use the Debian config, it'll need a lot of disk space for compilation and generate a huge linux-image debug package, so you might want to unset CONFIG_DEBUG_INFO locally if you're not interested. Right now only the deb files are generated but I've submitted a patch to have a .changes file which can be then used to manipulate them more easily (for example for uploading them a local Debian repository).

Note that, obviously, this is not targeted for inclusion to the official Debian archive. This is still not possible for various reasons explained here and there, and I still don't have a solution for that.

I hope this (either the scripts and config or the generated binary packages) can be useful. Don't hesitate to drop me a mail if needed.

Categories: FLOSS Project Planets

Drupal @ Penn State: D7 Features secret Rolemageddon

Planet Drupal - Thu, 2015-05-21 16:36

There’s many dirty little secrets in Drupal 7 core’s API when it comes to inconsistencies and oversights. It’s a big part of why so much care is being placed in D8 and its taking so long, because people realize this is a platform that’s used for the long haul and core decisions today will have lasting impacts a decade from now.

Categories: FLOSS Project Planets

FSF Blogs: Friday Free Software Directory IRC meetup: May 15

GNU Planet! - Thu, 2015-05-21 16:11

Join the FSF and friends Friday, May 15, from 2pm to 5pm EDT (18:00 to 21:00 UTC) to help improve the Free Software Directory by adding new entries and updating existing ones. We will be on IRC in the #fsf channel on freenode.

Tens of thousands of people visit directory.fsf.org each month to discover free software. Each entry in the Directory contains a wealth of useful information, from basic category and descriptions, to providing detailed info about version control, IRC channels, documentation, and licensing info that has been carefully checked by FSF staff and trained volunteers.

While the Free Software Directory has been and continues to be a great resource to the world over the past decade, it has the potential of being a resource of even greater value. But it needs your help!

If you are eager to help and you can't wait or are simply unable to make it onto IRC on Friday, our participation guide will provide you with all the information you need to get started on helping the Directory today!

Categories: FLOSS Project Planets

Friday Free Software Directory IRC meetup: May 15

FSF Blogs - Thu, 2015-05-21 16:11

Join the FSF and friends Friday, May 15, from 2pm to 5pm EDT (18:00 to 21:00 UTC) to help improve the Free Software Directory by adding new entries and updating existing ones. We will be on IRC in the #fsf channel on freenode.

Tens of thousands of people visit directory.fsf.org each month to discover free software. Each entry in the Directory contains a wealth of useful information, from basic category and descriptions, to providing detailed info about version control, IRC channels, documentation, and licensing info that has been carefully checked by FSF staff and trained volunteers.

While the Free Software Directory has been and continues to be a great resource to the world over the past decade, it has the potential of being a resource of even greater value. But it needs your help!

If you are eager to help and you can't wait or are simply unable to make it onto IRC on Friday, our participation guide will provide you with all the information you need to get started on helping the Directory today!

Categories: FLOSS Project Planets
Syndicate content