Feeds

Background

A question came up on the Stack Exchange site Unix & Linux in which I wrote up a pretty good answer, that describes some of the mechanics of how a process deals with its user credentials, so I’m adding my writeup to the blog.

It really comes down to what makes up a process in Unix. A process can come into existence in one of 2 ways. Either via the fork() function or through one of the exec() functions in C.

fork()

fork() basically just makes a copy of the current process, but assigns it a new process ID (PID). It’s a child of the original process. You can see this relationship in the output of @ps@:

1 2 3 4 5 6 7 $ ps axjf PPID PID PGID SID TTY TPGID STAT UID TIME COMMAND 1 5255 1964 1964 ? -1 Sl 500 0:39 gnome-terminal 5255 5259 1964 1964 ? -1 S 500 0:00 \_ gnome-pty-helper 5255 18422 18422 18422 pts/1 18422 Ss+ 500 0:01 \_ bash 5255 30473 30473 30473 pts/4 30473 Ss+ 500 0:00 \_ bash 30473 782 782 30473 pts/4 30473 Sl 500 1:14 | \_ evince s.pdf

Here you can see that gnome-terminal is the parent process (PID = 5255) and that bash is it’s child (PID = 18422, PPID = 5255).

When a process forks from its parent, it “inherits” certain things, such as copies of all the file descriptors that the parent currently has for open files and the parent’s user and group IDs.

NOTE1: PPID = Parent Process ID.
NOTE2: The last 2 are what identify what file and group permissions this process will have when accessing the file system.

So if a process just inherits its user and group ID from its parent, then why isn’t everything just owned by root or a single user? This is where exec() comes in.

exec() Part #1

The exec() family of functions, specifically execve(), “replace” a current process image with a new process image. The terminology “process image” is really just a file, i.e. an executable on disk. So this is how a bash script can execute a program such as /usr/bin/time.

So what about the user ID and group ID? Well to understand that let’s first discuss the concept of “Persona”.

Persona

At any time, each process has an effective user ID, an effective group ID, and a set of supplementary group IDs. These IDs determine the privileges of the process. They are collectively called the [persona of the process]1, because they determine “who it is” for purposes of access control.

exec() Part #2

So in addition to being able to swap out the “process image”, exec() can also change the user & group IDs from the original “real” ones to “effective” ones.

An example

For this demonstration I’m going to show you what happens when we start out in a shell as our default UID/GID, and then spawn a child shell using one of my supplementary GIDs, making it the child shell’s effective GID.

To perform this I’m going to make use of the unix command newgrp. newgrp allows you to spawn a new shell passing it the supplementary group that I’d like to make my effective GID.

For starters:

1 2 $ id -a uid=500(saml) gid=501(saml) groups=501(saml),502(vboxusers),503(jupiter)

We can see that this shell is currently configured with my default UID/GID of saml & saml. Touching some files shows that this is the case as well:

1 2 3 4 5 6 $ touch afile1 $ touch afile2 $ ls -l total 0 -rw-rw-r-- 1 saml saml 0 May 21 23:47 afile1 -rw-rw-r-- 1 saml saml 0 May 21 23:47 afile2

Now we make our supplementary group jupiter the effective GID:

1 2 3 $ newgrp jupiter $ id -a uid=500(saml) gid=503(jupiter) groups=501(saml),502(vboxusers),503(jupiter)

Now if we touch some files:

1 2 3 4 5 6 7 8 $ touch afile3 $ touch afile4 $ ls -l total 0 -rw-rw-r-- 1 saml saml 0 May 21 23:47 afile1 -rw-rw-r-- 1 saml saml 0 May 21 23:47 afile2 -rw-r--r-- 1 saml jupiter 0 May 21 23:49 afile3 -rw-r--r-- 1 saml jupiter 0 May 21 23:49 afile4

We see that the shell’s effective GID is jupiter, so any interactions with the disk result in files being created with jupiter rather than my normal default group of saml.

References

• fork() man page
• exec() man page
• execve() man page
• credentials man page
• newgrp man page
• The GNU C Library
• The GNU C Library – 26.4 Creating a Process
• The GNU C Library – 26.5 Executing a File
• GID, current, primary, supplementary, effective and real group IDs?

• Hollywood Studios [attempt to censor] Pirate Bay Documentary

  Probably not deliberate, but pretty damn inept.

  Over the past weeks several movie studios have been trying to suppress the availability of TPB-AFK [the Pirate Bay documentary] by asking Google to remove links to the documentary from its search engine. The links are carefully hidden in standard DMCA takedown notices for popular movies and TV-shows. The silent attacks come from multiple Hollywood sources including Viacom, Paramount, Fox and Lionsgate and are being sent out by multiple anti-piracy outfits. Fox, with help from six-strikes monitoring company Dtecnet, asked Google to remove a link to TPB-AFK on Mechodownload. Paramount did the same with a link on the Warez.ag forums. Viacom sent at least two takedown requests targeting links to the Pirate Bay documentary on Mrworldpremiere and Rapidmoviez. Finally, Lionsgate jumped in by asking Google to remove a copy of TPB-AFK from a popular Pirate Bay proxy.

  (tags: funny inept hollywood lionsgate fox viacom paramount dtecnet tpb-afk piratebay piracy copyright movies google)

• Flashback: How Yahoo Killed Flickr and Lost the Internet

  This is about the best tech journalism I’ve ever read on Flickr. nice one Mat Honan

  (tags: gizmodo flickr acquisition mergers yahoo corporate-culture mat-honan tech journalism)

• Resisting the lure of the Freeman movement | Workers Solidarity Movement

  An anarchist critique of the Freeman movement from the WSM:

  This has been a very brief overview of the Freeman movement that has tried to capture with broad strokes its nature and possible responses. There is room for much more work, including a more in-depth analysis of the various flaws in the approach to the law. The greatest danger however is allowing a movement to develop within anarchist circles that ignores the principle of mutual aid and implicitly promotes private ownership of resources, that by granting absolute right to individuals gives them the ability to ignore their responsibilities to the wider community and ecology that sustains them. In more traditional terms, the movement is one all about negative freedoms, ignoring positive freedom as a concept.

  (tags: anarchism freeman-on-the-land politics ireland law wsm)

• The Reactionary ‘Freeman-?on-?the-?land’ and a Political Fracture

  Another leftie view on the Freeman movement

  (tags: freeman-on-the-land politics ireland left-wing anarchism law)

• memcached turns 10 years old

  Well, apparently tomorrow, but close enough. Happy birthday to bradfitz’ greatest creation and its wonderful slab allocator!

  (tags: birthdays code via:alex-popescu open-source history malloc memory caching memcached)

Debian GNU/Hurd 2013 released! Details.

It is with huge pleasure that the Debian GNU/Hurd team announces the release of Debian GNU/Hurd 2013. This is a snapshot of Debian "sid" at the time of the Debian "wheezy" release (May 2013), so it is mostly based on the same sources. It is not an official Debian release, but it is an official Debian GNU/Hurd port release.

The installation ISO images can be downloaded from Debian Ports in the usual three Debian flavors: NETINST, CD, DVD. Besides the friendly Debian installer, a pre-installed disk image is also available, making it even easier to try Debian GNU/Hurd.

Debian GNU/Hurd is currently available for the i386 architecture with more than 10.000 software packages available (more than 75% of the Debian archive, and more to come!).

Please make sure to read the configuration information, the FAQ, and the translator primer to get a grasp of the great features of GNU/Hurd.

Due to the very small number of developers, our progress of the project has not been as fast as other successful operating systems, but we believe to have reached a very decent state, even with our limited resources.

We would like to thank all the people who have worked on GNU/Hurd over the past decades. There were not many people at any given time (and still not many people today, please join!), but in the end a lot of people have contributed one way or another. Thanks everybody!

The GNU Hurd is the GNU project's replacement for the Unix kernel. It is a collection of servers that run on the Mach microkernel to implement file systems, network protocols, file access control, and other features that are implemented by the Unix kernel or similar kernels (such as Linux). More detailed.

GNU Mach is the microkernel upon which a GNU Hurd system is based. It provides an Inter Process Communication (IPC) mechanism that the Hurd uses to define interfaces for implementing in a distributed multi-server fashion the services a traditional operating system kernel provides. More detailed.

GNU Parallel 20130522 ('Rana Plaza') has been released. It is available for download at: http://ftp.gnu.org/gnu/parallel/

The changes for this release has been in the core of the code, so this release should be considered beta quality.

New in this release:

• --ctrlc is now default if --pipe is not used; thus making it possible to kill remotely started jobs simply by pressing Ctrl-C.

• --timeout 200% now means kill jobs that take more than twice the time of the median run time of a job.

• Jobs are now distributed round robin when having mulitiple --sshlogin instead of filling up one --sshlogin at a time.

• niceload: darwin version of --io --mem --swap. Thanks to Anders F Björklund.

• Bug fixes and man page updates.

About GNU Parallel

GNU Parallel is a shell tool for executing jobs in parallel using one or more computers. A job is can be a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables. A job can also be a command that reads from a pipe. GNU Parallel can then split the input and pipe it into commands in parallel.

If you use xargs and tee today you will find GNU Parallel very easy to use as GNU Parallel is written to have the same options as xargs. If you write loops in shell, you will find GNU Parallel may be able to replace most of the loops and make them run faster by running several jobs in parallel. GNU Parallel can even replace nested loops.

GNU Parallel makes sure output from the commands is the same output as you would get had you run the commands sequentially. This makes it possible to use output from GNU Parallel as input for other programs.

You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/

You can install GNU Parallel in just 10 seconds with: wget -O - pi.dk/3 | sh

Watch the intro video on http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 or at http://tinyogg.com/watch/TORaR/ and http://tinyogg.com/watch/hfxKj/

When using GNU Parallel for a publication please cite:

O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47.

About GNU SQL

GNU sql aims to give a simple, unified interface for accessing databases through all the different databases' command line clients. So far the focus has been on giving a common way to specify login information (protocol, username, password, hostname, and port number), size (database and table size), and running queries.

The database is addressed using a DBURL. If commands are left out you will get that database's interactive shell.

When using GNU SQL for a publication please cite:

O. Tange (2011): GNU SQL - A Command Line Tool for Accessing Different Databases Using DBURLs, ;login: The USENIX Magazine, April 2011:29-32.

About GNU Niceload

GNU niceload slows down a program when the computer load average (or other system activity) is above a certain limit. When the limit is reached the program will be suspended for some time. If the limit is a soft limit the program will be allowed to run for short amounts of time before being suspended again. If the limit is a hard limit the program will only be allowed to run when the system is below the limit.

Lucene's facet module has seen some great improvements recently: sizable (nearly 4X) speedups and new features like DrillSideways. The Jira issues search example showcases a number of facet features. Here I'll describe two recently committed facet features: sorted-set doc-values faceting, already available in 4.3, and dynamic range faceting, coming in the next (4.4) release.

To understand these features, and why they are important, we first need a little background. Lucene's facet module does most of its work at indexing time: for each indexed document, it examines every facet label, each of which may be hierarchical, and maps each unique label in the hierarchy to an integer id, and then encodes all ids into a binary doc values field. A separate taxonomy index stores this mapping, and ensures that, even across segments, the same label gets the same id.

At search time, faceting cost is minimal: for each matched document, we visit all integer ids and aggregate counts in an array, summarizing the results in the end, for example as top N facet labels by count.

This is in contrast to purely dynamic faceting implementations like ElasticSearch's and Solr's, which do all work at search time. Such approaches are more flexible: you need not do anything special during indexing, and for every query you can pick and choose exactly which facets to compute.

However, the price for that flexibility is slower searching, as each search must do more work for every matched document. Furthermore, the impact on near-real-time reopen latency can be horribly costly if top-level data-structures, such as Solr's UnInvertedField, must be rebuilt on every reopen. The taxonomy index used by the facet module means no extra work needs to be done on each near-real-time reopen.

Enough background, now on to our two new features!


Sorted-set doc-values faceting

These features bring two dynamic alternatives to the facet module, both computing facet counts from previously indexed doc-values fields. The first feature, sorted-set doc-values faceting (see LUCENE-4795), allows the application to index a normal sorted-set doc-values field, for example:
doc.add(new SortedSetDocValuesField("foo"));
doc.add(new SortedSetDocValuesField("bar"));
and then to compute facet counts at search time using SortedSetDocValuesAccumulatorand SortedSetDocValuesReaderState.

This feature does not use the taxonomy index, since all state is stored in the doc-values, but the tradeoff is that on each near-real-time reopen, a top-level data-structure is recomputed to map per-segment integer ordinals to global ordinals. The good news is this should be relatively low cost since it's just merge-sorting already sorted terms, and it doesn't need to visit the documents (unlike UnInvertedField).

At search time there is also a small performance hit (~25%, depending on the query) since each per-segment ord must be re-mapped to the global ord space. Likely this could be improved (no time was spend optimizing). Furthermore, this feature currently only works with non-hierarchical facet fields, though this should be fixable (patches welcome!).


Dynamic range faceting

The second new feature, dynamic range faceting, works on top of a numeric doc-values field (see LUCENE-4965), and implements dynamic faceting over numeric ranges. You create a RangeFacetRequest, providing custom ranges with their labels. Each matched document is checked against all ranges and the count is incremented when there is a match. The range-test is a naive simple linear search, which is probably OK since there are usually only a few ranges, but we could eventually upgrade this to an interval tree to get better performance (patches welcome!).

Likewise, this new feature does not use the taxonomy index, only a numeric doc-values field. This feature is especially useful with time-based fields. You can see it in action in the Jira issues search example in the Updated field.

Happy faceting!

• Flight arrived late Sunday night and took the Portland MAX light rail to the hotel. Grabbed something to eat and went to bed 4PM EST!
• Had a descent breakfast at the hotel
• Went on the "Secrets of Portlandia" free walking tour (http://secretsofportlandia.com). This was a great tour. I highly recommend it if you come to Portland.
• Lunch (frozen yogurt!)
• Took a walk across the river to the Burnside Skate Park that Brent Dunn wanted to checkout.
• We then made it over the the Oregon Convention Center to get our registrations
• Met with greggles to discuss keynoting DrupalCamp Charlotte and some Classic security related questions. including security audits and Guardr/Hardened Drupal
• Worked on the DrupalCamp Charlotte website
• Dinner at a local brewery
• Worked more on the DrupalCamp Charlotte website

Blog Category: Drupal var switchTo5x=false;stLight.options({publisher:'dr-4334ed90-d702-124d-3358-aea68cdffdd9'});

I have just merged to master a branch by Fabio D'Urso that lets you configure the review tools that appear in the review bar.



This way you can decide that by default you want your highlighter to be green instead of yellow. Or even have two highlighters in the review bar.

Please test and enjoy it :-)

We are happy to announce the seventh GNU Hackers Meeting, which will take place from August 22 to August 25 2013 in Paris, France.

We are happy to announce the seventh GNU Hackers Meeting, which will take place from August 22 to August 25 2013 in Paris, France.

The GNU Hackers Meetings are a friendly and informal venue to discuss technical, social and organizational issues concerning GNU and free software.

Following the success of the 2011 edition, we will be again generously hosted by IRILL (http://www.irill.org), the Center for Research and Innovation in Free Software. We have set up an official meeting page containing practical information at http://www.gnu.org/ghm/2013/paris/.

As we are still in the process of defining the program, we welcome your talk proposals. We are particularly interested in new GNU programs and recent developments in existing packages, plus any related topic. In our experience the audience tends to be technically competent, so feel free to propose very technical topics as well; we will try to schedule all such talks together in the same morning or afternoon session, for the public’s sake.

Registration is free of charge, but in case you want to contribute to the organization of future meetings, we are glad to accept a small donation.

The FSF has kindly set up a small fund to help finance (documented) travel expenses for GNU maintainers and contributors, past and present. In case you qualify and you need financial assistance to attend, please mention this fact in your registration e-mail, along with all the relevant details including the minimum amount that would permit you to attend the event, no later than June 24 2013.

You are welcome to join the public mailing list ghm-discuss@gnu.org (https://lists.gnu.org/mailman/listinfo/ghm-discuss). Talk proposals and registration e-mails should also be directed to the public list ghm-discuss@gnu.org, or directly to Luca Saiu's personal address positron@gnu.org.

We are happy to announce the seventh GNU Hackers Meeting, which will take place from August 22 to August 25 2013 in Paris, France.

We are happy to announce the seventh GNU Hackers Meeting, which will take place from August 22 to August 25 2013 in Paris, France.

The GNU Hackers Meetings are a friendly and informal venue to discuss technical, social and organizational issues concerning GNU and free software.

Following the success of the 2011 edition, we will be again generously hosted by IRILL (http://www.irill.org), the Center for Research and Innovation in Free Software. We have set up an official meeting page containing practical information at http://www.gnu.org/ghm/2013/paris/.

As we are still in the process of defining the program, we welcome your talk proposals. We are particularly interested in new GNU programs and recent developments in existing packages, plus any related topic. In our experience the audience tends to be technically competent, so feel free to propose very technical topics as well; we will try to schedule all such talks together in the same morning or afternoon session, for the public’s sake.

Registration is free of charge, but in case you want to contribute to the organization of future meetings, we are glad to accept a small donation.

The FSF has kindly set up a small fund to help finance (documented) travel expenses for GNU maintainers and contributors, past and present. In case you qualify and you need financial assistance to attend, please mention this fact in your registration e-mail, along with all the relevant details including the minimum amount that would permit you to attend the event, no later than June 24 2013.

You are welcome to join the public mailing list ghm-discuss@gnu.org (https://lists.gnu.org/mailman/listinfo/ghm-discuss). Talk proposals and registration e-mails should also be directed to the public list ghm-discuss@gnu.org, or directly to Luca Saiu's personal address positron@gnu.org.

I've been enjoying Erik Dietrich's series of articles on the Daedtech blog about a phenomenon he calls the "Expert Beginner".

Dietrich's observation is that, while learning a new skill (such as computer programming, say), you might accidentally think that you had become an expert when in fact you were actually doing it all wrong. You just hadn't noticed that you were doing it wrong, and nobody was around to point that out to you.

For whatever reason, software appears to be a field that is particularly prone to this sort of Expert Beginner situation, perhaps because we software types are often learning things on our own, rather than having the opportunity to be taught from an established body of knowledge.

Not only is the Expert Beginner full of bad habits, but even worse: should he at some point encounter somebody who has greater expertise, and can help educate the Expert Beginner toward improvement, this assistance is likely to be met by resistance, as the Expert Beginner will have to start by un-learning all of his bad habits, which he is naturally reluctant to do (since they have served him well so far).

Dietrich is an entertaining writer and I recommend the essays:

• How Developers Stop Learning: Rise of the Expert BeginnerThe Expert Beginner has nowhere to go because progression requires an understanding that he has a lot of work to do, and that is not a readily available conclusion. You’ll notice that the Expert Beginner is positioned slightly above Advanced Beginner but not on the level of Competence. This is because he is not competent enough to grasp the big picture and recognize the irony of his situation, but he is slightly more competent than the Advanced Beginner due mainly to, well, extensive practice being a Beginner.
• How Software Groups Rot: Legacy of the Expert BeginnerPerhaps it’s a lack of automated testing. Giant methods/classes. Lots of copy and paste coding. Use of outdated or poor tooling. Process. It can be any number of things, but the common thread is that you have a person or people in positions of authority that have the culturally lethal combination of not knowing much, not knowing what they don’t know, and assuming that, due to their own expertise, anything they don’t know isn’t worth knowing. This is a toxic professional culture in that it will force talented or ambitious people either to leave or to conform to mediocrity.
• How Stagnation is Justified: Language of the Expert BeginnerThe Expert Beginner believes that he and his ‘fellow’ Expert have a simple difference of opinion among ‘peers.’ While it may be true that one Expert speaks at conferences about source control best practices and the other one runs the IT for Bill’s Bait Shop and has never used source control, either opinion is just as valid.
• Up or Not: Ambition of the Expert BeginnerAn Expert Beginner’s entire career is built on a foundation of cognitive dissonance. Specifically, they believe that they are experts while outside observers (or empirical evidence) demonstrate that they are not. So an Expert Beginner is sentenced to a life of believing himself to be an expert while all evidence points to the contrary, punctuated by frequent and extremely unwelcome intrusions of that reality.
• Self-Correcting Organizations: Fall of the Expert BeginnerFollowing the career arc of Expert Beginners is really quite sad. In the early stages, one feels annoyed and a little indignant at advancement by luck instead of competence. As things progress, real damage is caused by poor implementation and wrong-headed approaches, resulting for a lot of people in stress, frustration, failure, and at times even lost jobs and failed ventures. And, in the end, the fate of the one that caused these things is probably poetically just, but hard to find happiness in. A person ill-suited for a role assumed it, caused problems, and then suffered personal hardship. It’s not a great story.

To my mind, what this boils down to, for a person who's concerned with continuing to develop their own skills indefinitely, is: in order to improve my skills, I need to find somebody who can coach me.

You can always get better; you can always learn more. It's just a matter of finding somebody who can help you improve, and putting in the effort to learn from that person.

Just don't get stuck being an Expert Beginner.

In general there is a tendency to set parallel implementations to being equal
to performant implementations. Except in the really naive case there is always
going to be some overhead due to scheduling work, managing memory sharing and
network communication overhead. Essentially that knowledge is reflected in
Amdahl’s law (the amount of serial work limits the benefit from running parts
of your implementation in parallel, http://en.wikipedia.org/wiki/Amdahl’s_law),
and Little’s law (http://en.wikipedia.org/wiki/Little’s_law) in case of queuing
problems.

When looking at current Java optimisations there is quite a bit going on to
support better parallelisation: Work is being done to provide for improving
lock contention situations, the GC adaptive sizing policy has been improved to
a usable state, there is added support for parallel arrays and lampbda’s
splitable interface.

When it comes to better locking optimisations what is most notable is work
towards coarsening locks at compile and JIT time (essentially moving locks from
the inside of a loop to the outside); eliminating locks if objects are being
used in a local, non-threaded context anyway; and support for biased locking
(that is forcing locks only when a second thread is trying to access an
object). All three taken together can lead to performance improvements that
will almost render StringBuffer and StringBuilder to exhibit equal performance
in a single threaded context.

For pieces of code that suffer from false sharing (two variables used in
separate threads independently that end up in the same CPU cacheline and as a
result are both flushed on update) there is a new annotation: Adding the
“@contended” annotation can help the compiler for which pieces of code to add
cacheline padding (or re-arrange entirely) to avoid that false sharing from
happening. One other way to avoid false sharing seems to be to look for class
cohesion - coherent classes where methods and variables are closely related
tend to suffer less from false sharing. If you would like to view the resulting
layout use the “-XX:PrintFieldLayout” option.

Java 8 will bring a few more notable improvements including changes to the
adaptive sizing GC policy, the introduction of parallel arrays that allow for
parallel execution of predicates on array entries, changes to the concurrency
libraries, internalised iterators.

Dries Buytaert, Drupal's founder and project lead, presents his six-monthly keynote at Drupalcon, and sets out where he thinks Drupal needs to go.

Dries starts by talking about a highly visible sign of growth in Drupal — 8 years ago he and half of the conference delegates were able to fit into one hotel room for drinks on the night before the event opened. This year, over 3,000 delegates took over and filled up most of the bars in Portland, Oregon.

Next he shows how the users of Drupal have grown too. He says that 3-5 years ago it was just a hope that one day the White House website, and major broadcasters like NBC would use Drupal. 

Dries then returns to a theme he has raised in previous keynotes — summed up by the phrase "Do Well, Do Good". He says that as a result of Drupal's success, and the success of the businesses and careers of those who work on it, the world is benefiting. Hundreds of thousands of charities and non-profits are able to download Drupal for free and adapt it for their needs, to better achieve their aims — raising money, organising volunteers and campaigning.

But also, there are now over a hundred countries whose governments use Drupal to help provide better public services online to citizens. 

The White House Case Study

To illustrate this, Dries introduces a video in which Macon Phillips, the Director of Digital Strategy at The White House, speaks about their use of Drupal.

Macon says that they launched WhiteHouse.gov on Drupal in October 2009. They did a lot of work to assess the best system for them to use, and whether they could adopt open source. They have had great success with their choice of Drupal. As a result they've been keen to give back to the community, and have contributed a number of modules, and hosted a 'hack day' at the White House.

He gives the example of the 'We The People' petitioning application. It allows anyone to start a petition, and anyone to sign it online. The executive has pledged to respond to petitions that get enough signatures. Since its launch in September 2011 over 8 milion people have signed a petition on the site. Something that would only be possible online. It's a new way to scale civic participation.

The open nature of Drupal means they can now extend this even further, and they have launched a Read API to allow other websites to pull in data about petitions. They are now also planning a Write API to allow other applications to help people create and sign petitions.

Context

Dries next moves to talking about context. He says it's the availability of these petitions online that enables more people to engage with them. What's true in government is also true in business. People need the right content, in the right place, at the right time. Context is key.

He gives the examples of Amazon and Netflix. Able to personalise their sites and give recommendations and related content to users. As a result they've been able to massively disrupt their markets.

However, most organisations are currently really bad at this.

Dries shows a google search for 'flight to London'. The first result is actually from Icelandair which does really well — when he clicks through to the site, it prefills the form with 'London' as the destination for him, shows images of London to get him excited, and even knows that he wants to get there from Boston where he is now. Meanwhile, the next result Google presents, Continental, just takes him to a blank search form when he clicks through.

Context can be about many things. Some examples are geography, past behaviour, intent, device used, the time of day, and many more.

By responding to the context, you provide a much better experience to the user — what is becoming known as Web Experience Management (WEM).

Web Experience Management

Owners of websites want to do this in three stages, each of which requires integration with other systems:

• Stage 1: Attract — by using SEO, mobile platforms, social networks
• Stage 2: Convert — using CRM systems, email, ecommerce, personalisation and marketing automation systems
• Stage 3: Analyse — using CRM, analytics and marketing automation

So the Content Management System needs to integrate with all these other systems.

But, Dries cites recent research that shows that marketers are really unhappy with the state of integrations between CMSs and other systems such as CRM, ecommerce, personalisation etc. Just 6% of those surveyed were able to describe the integration between their CMS and their CRM as excellent.

At the moment these kinds of sites are using big proprietary systems, and Dries says this level of disatisfaction presents a great opportunity for Open Source, and Drupal in particular.

How To Address WEM With Drupal

So how could we develop Drupal to serve the future needs of these organisations?

1. The Drupal community could build these kinds of CRM and marketing automation tools ourselves. That's what competitors are doing — creating proprietary product suites to try to handle it all.
2. Drupal could focus on being very good at integrating with other systems. There are many good integration modules already, but these could be extended and improved.

Dries says that research supports the second option. End users want 'best of breed' tools that work well together. So, if we don't go the route of building everything ourselves, what do we need to do?

What Clients Want from a CMS

The research, says Dries, shows that clients want three things from a CMS:

1. Integrate best of breed tools
2. Deliver a great mobile experience
3. Ease of Use & Content Authoring

Dries says that all of these are addressed in Drupal modules — but what's more they are all addressed in Drupal 8 core:

 

1. Integrate best of breed tools: Web services are now provided in core to facilitate these integrations; all content can be exported as JSON or XML easily.
2. Deliver a great mobile experience: Responsive theming makes for better user experience on mobile; the web services mean better integration with mobile apps; even the admin and editing interface is mobile ready or can be managed via an app.
3. Ease of Use & Content Authoring: The admin user interface and content editing forms have been completely redesigned; In-place editing allows content to be edited at a click in the frontend; Images can be added to pages with drag and drop; there is now a true Preview of content, and they are working on enabling this to preview at different screen sizes.

The first alpha of Drupal 8 will be released during this week, and the developers are working towards the code freeze deadline of 1st July 2013. The aim is to have a stable release by the end of 2013 or early 2014. Sites will be able to use Drupal 8 in 2014.

Dries summarises by saying it is important for Drupal to 'skate to where the puck is', and that it is no longer enough to simply manage content. Drupal has to be a key part in enabling site owners to manage the whole experience for users.

By moving in this direction, Dries says that developers get to work on what matters, Drupal companies get to deliver what clients want, Drupal users can improve their business by shifting more of it to digital, and for the world we'll be doing well, and doing good.

Keith and I are pleased to announce the immediate availability of TeleBT, a new Altus Metrum ground station product providing the equivalent of TeleDongle plus Bluetooth.

TeleBT working with AltosDroid on an Android device provides everything needed to monitor a rocket in flight, record telemetry, and know how to walk right to the airframe after it's back on the ground.

The Bluetooth capability of TeleBT is also supported by AltosUI on Linux, and with a micro USB cable TeleBT works just like TeleDongle on Windows, Mac, and Linux systems running AltOS version 1.2.1 or later.

Been quiet some time since my last blog about Krita, well, I had been a bit busy with college work. Nonetheless, with whatever time I had, and all the help from Boud, I have been able to import a particular feature from the Perspective Grid to the Perspective Assistant. A little background Perspective Grid The […]

Today was a tribute to the Druplicon, our community's iconic Drupal drop: Lashing rain and a river of 3300 Drupalistas (biggest Drupalcon ever) flowing into a sea of attractions: Sessions and BOFs, the exhibit hall, a hilarious game-show moderated by Rob and JAM, Holly Ross' and Dries' keynote, the group picture, you name them. It's just amazing to see how the Drupal community is growing from year to year; Kudos to the Drupal Association for doing a great job. Before I let you enjoy Michel's pictures further down, Dries' key messages from the keynote: The future of Drupal is not about managing content, but web experience management / Drupal 8 timeline: Code freeze scheduled for 1 July 2013. Expect Drupal 8 to go into production in 2014.

         

I'm getting ready to head in to DrupalCon, where over the next few days I'll be talking education and open learning with anyone who is interested.

And as I'm heading in, I have MOOCs on the brain - not because I'm particularly a fan of MOOCs, but because of the tendency to take a great thing (in this case, information and interpersonal exchanges distributed broadly over the web) and reduce it into something that feels more manageable, but is ultimately something lesser (in this case, MOOC platforms). More on this later.

The Web Is Your MOOC

Part of the reason that I'm thinking these thoughts prior to heading into DrupalCon is that I've long held the notion that open source communities have been engaging in effective peer-supported learning, even while many for-profit companies and academic communities have been struggling to distill the process of peer-supported learning into something resembling a replicable product. From having participated in and built many types of learning communities over the years, simpler is often better - many open source communities have done amazing work with listservs and issue queues, and many more feature-rich platforms have withered because, over time, a site owners "must-have" feature is the post launch usability nightmare. There's a moral in there about user-centered design and user testing, but that's a subject for another post.

But getting back to MOOCs, the early MOOCs - the ones run by Stephen Downes, Alec Couros, Dave Cormier, George Siemens, (and yes, I know I'm forgetting people - please fill in the gaps in the comments) etc - encouraged participation from anywhere. If you had a blog with an RSS feed, you were in. Participants remained in control of their work (depending, of course, on the publishing tool they were using. Open source platforms generally offer more options for data ownership and portability than their closed brethren). The MOOC was like a marauding mob of information, with the potential to sprout anywhere.

It's All About The Portfolio

In the post-lifestream, post-MOOC era, it's been rare to see much excitement about portfolios. This doesn't surprise me, because like all good ideas, portfolios have been around for a while, and thus lack the shiny newness that generates great marketing copy. However, the need for the concept hasn't diminished - any time you see a site that promises to collect the sources of your learning into a single location, so you can show your employers what you know! - you should think, "portfolio." All of the sites that promise to simplify collecting and curating your digital footprint? Portfolios. A lot of the conversations around documenting and receiving credit for informal learning have their roots (and possibly solutions) in portfolios.

In the conversations we have had about portfolios over the years, we have seen three main barriers, or areas of misunderstanding:

• Distinguishing between a working and a presentation portfolio: simply put, the working portfolio is a running collection of just about everything you do. The presentation portfolio is a selection of elements from the working portfolio selected for a specific purpose. Portfolios can serve different purposes for different reasons, and the relationship between the working portfolio and the presentation portfolio is key.
• Portfolios need care and feeding over time: as mentioned before, the working portfolio is messy. Periodically, the working portfolio needs to be pruned and cleaned up. But, messy is great, and if it's not messy, that could be a sign that things aren't working as they should.
• Ownership and control of the portfolio: because most portfolio implementations are paid for by an organization, the organization usually controls access to the portfolio and any information in it. Organizational control is also seen as an essential element to assessment. However, this flies in the face of learner control and ownership of the means by which they learn. Ultimately, this is a data portability issue with implications for the learning experience. More on this later.

Concluding Thoughts

One of the things that has been particularly underwhelming about the corporate MOOCs that have cropped up is their uncanny resemblance to an LMS with an open enrollment policy. While there are many differences between the platform-stylehttps://chronicle.com/article/Providers-of-Free-MOOCs-Now/136117/ MOOCs and the original versions, the lack of learner control is a key element. Like Vegas, work in a MOOC stays in a MOOC (unless, of course, a company pays money to study student data).

In the platform-style MOOCs, the open web is missing. From a learner perspective, the portfolio is MIA. For a learner, throwing the evidence of your learning into a space that someone else controls isn't a viable long term strategy.

So, if you're at DrupalCon and want to talk open learning, let's make some time and sit down together. Open source, and the methodologies that support sustainable open source development, have a lot in common with open learning. I'd love to hear what other people are doing in this space.

Hello planet!!!

After nearby two years of disrupted services, my blog is back online with about 95% of archives available, only two or three photos are missing. However, my blog doesn't come alone, I follow him. Indeed after 6-8 month of inactivity around KDE project, I can say, I miss KDE a lot.

In the near future I'm going to take part in three KDE related events.

Coding, Promoting and Sharing The Coding Way - Atelier KDE (Toulouse, France)

Firstly, this Saturday I'm going to the "Atelier KDE", the local KDE monthly event. During the whole day I want to share a bit with other attendees (i.e. I don't go anywhere to be unsocial :) ), and also to contribute to KDE software by coding (I hope my C++/Qt skill hasn't gone during my break). I'm not sure yet in which part of which software/library I want to do some hacking, but not sure in which piece of software. Perhaps in KDE Frameworks or KDevelop. I think I will choose Saturday depending on my mood.

I also want to say a big thanks to morice-net to take care of this kind of beautiful event.

The Promoting Way - Solution Linux (Paris, France)

After contributing by coding, I'm going to explore another way of contributing in Paris. Indeed the 27th and 28th of May I'm going to Paris in order to help Paris KDE Team in Solution Linux. Solution Linux is a yearly French exhibition around Open Source. It's my first time in Solution Linux, but I'm going there with some veterans like Sébastien Renard or Aurélien Gateau. In order to share this experience I will try to do a daily report of this event in my blog.

The Sharing and Hacking Ways aKademy (Bilbao, Spain)

In mid July like a lot of people I will attend aKademy in Bilbao. Like each year aKademy allow to share with all the great KDE community.

AKademy is also each year a great place to hack around KDE software. So I want to make a contribution in KDE Frameworks and also hack a bit in KDevelop.

Don't stop after that

Yes I'm really enthusiast about KDE, I want to contribute to KDE more and more. Now I say it, now I need to give more to KDE. Yes I know for New Year's resolution is a bit late.

So I will try to promote better KDE in my daily life, but also by going to events like Solution Linux. And, also be involved more in KDE related project development and all kinds of stuff related to KDE environment.

Yes I still have the aim to help my "master" to dominate the world with KDE.

So now I can say I want to come back in KDE. And, I hope to see you soon!

That's all

We're pleased to announce PyPy 2.0.2. This is a stable bugfix release over 2.0 and 2.0.1. You can download it here:

http://pypy.org/download.html

It fixes a crash in the JIT when calling external C functions (with ctypes/cffi) in a multithreaded context.

What is PyPy?

PyPy is a very compliant Python interpreter, almost a drop-in replacement for CPython 2.7. It's fast (pypy 2.0 and cpython 2.7.3 performance comparison) due to its integrated tracing JIT compiler.

This release supports x86 machines running Linux 32/64, Mac OS X 64 or Windows 32. Support for ARM is progressing but not bug-free yet.

Highlights

This release contains only the fix described above. A crash (or wrong results) used to occur if all these conditions were true:

• your program is multithreaded;
• it runs on a single-core machine or a heavily-loaded multi-core one;
• it uses ctypes or cffi to issue external calls to C functions.

This was fixed in the branch emit-call-x86 (see the example file bug1.py).

Cheers, arigo et. al. for the PyPy team

Dear KDE developers,

My name is Andreas Schilling. I am a research assistant at the chair of Information Systems and Services at Bamberg University. I would appreciate it very much if you could give me some quick feedback regarding your motivation and your experiences in KDE. Could you please fill in a short questionnaire of multiple-choice questions? This will take you no longer than 10 minutes.

From all survey participants who answered all questions, a randomly picked person will win the current Google Nexus 4 phone.

This survey follows my previous work surveying KDE participants in the Google Summer of Code. In my research I examine factors that keep developers motivated and committed in their open-source projects. Because the best way to find out about these factors is to directly ask the people who are concerned, I revert to you. As soon as I finish my research, I will publish a blog post with the anonymous aggregated results of this survey, and I will compare them with the results of other surveys with KDE developers such as with previous participants of Google Summer of Code.

In order to reach a broad developer audience, I teamed up with Danny Allen of the KDE Commit-Digest. Danny is currently asking all KDE developers to complete/update their personal information for the Commit-Digest. Danny and I thought this is a good opportunity for asking you for feedback regarding your motivation and your experiences at KDE so far. Participants of the Commit-Digest survey will be forwarded automatically to this optional questionnaire.

You can find the Commit-Digest Survey (for people with KDE developer accounts) and my questionnaire at:

https://commit-digest.org/data/

If you have any questions about this questionnaire or about my research, please do not hesitate to contact me directly (andreas.schilling[at]uni-bamberg.de).

Best regards,

Andreas Schilling

Here's a nice slide deck on general benchmarking principles that I came across recently: Tokutek benchmarking principles

For hard-core perf types, there's not much new here, but I thought it was worth passing along in case some found it interesting.

Among my favorite observationss from the slide deck:

• benchmark frequently, to catch performance regressions soon after they occur
• keep a benchmark history, for analysis of data over time
• use graphs and plots to help with data interpretation
• share benchmarking data widely ("developers love data")

At various companies, at various times, I've worked with benchmarking teams, and it's nice to see the slow-but-steady "systematization of knowledge" going on here, since some of these are hard lessons and it's worth your time not to re-learn them yourself.