FLOSS Project Planets

Hernan Grecco: The state of the PyVISA ecosystem (and more)

Planet Python - Mon, 2015-08-31 10:42
Yesterday we released several packages of the Python Instrumentation Ecosystem. You can upgrade to PyVISA 1.8, PyVISA-py 0.2 and PyVISA-sim 0.3  by:

pip install -U pyvisa pyvisa-py pyvisa-sim

For those of you who are new to instrumentation in Python, PyVISA is a Python frontend for the VISA specification that enables controlling all kinds of measurement equipment through GPIB, RS232, USB and Ethernet among others interfaces. If you are familiar with VISA instruments in LabVIEW, Matlab, C or .NET you already know how it works and you can make use of PyVISA as a nice Pythonic API to write your programs. If you have never done any instrumentation, Python and PyVISA is great combination to start.

Code: https://github.com/hgrecco/pyvisaDocs: http://pyvisa.readthedocs.org/Tracker: https://github.com/hgrecco/pyvisa/issues
As I mentioned before PyVISA is a frontend for the VISA specification, but what does this means in terms of software? PyVISA can connect to multiple backends, which are the ones doing part of the hard work. We currently have 3:

ni: is a wrapper to the NI-VISA library, which is the de facto standard implementation of VISA. It is feature complete but requires that you install the proprietary library provided by National Instruments. This is the default backend and is bundled with PyVISA. (Notice that we provide the wrapper, you need to install NI-VISA yourself as explained in the PyVISA docs)

py: is an implementation of the VISA specification using popular python packages to talk over the different interfaces: PySerial, PyUSB, linux-gpib and socket (which is inside the Python standard library). It is almost feature complete for Message Based Instruments (ASRL, USB, TCPIP, GPIB). It is available through the PyVISA-py.

Code: https://github.com/hgrecco/pyvisa-pyDocs: http://pyvisa-py.readthedocs.org/Tracker: https://github.com/hgrecco/pyvisa-py/issues
sim: allows you create simulated instruments using simple text files. It is great for testing and off-line developing of complex instrumentation applications. It is available through the PyVISA-sim.

Code: https://github.com/hgrecco/pyvisa-simDocs: http://pyvisa-sim.readthedocs.org/Tracker: https://github.com/hgrecco/pyvisa-sim/issues
As always, these releases would not have been possible without our awesome community that provides code, bug reports, testing and support.

Finally I would like to mention that PyVISA is great but is kind of middle level. For complex applications you want better abstractions that allow you to forget about how the voltage is asked to a particular Voltmeter. For that purpose, I created Lantz a few years ago. It provides a very nice way to write drivers that encapsulate instrument specific information to then use them in scripts and GUI apps. It is not a replacement for PyVISA. Lantz builds on top of it to do the low level communication.

It turned out that I was not the only one thinking and coding in these direction. And I have always felt that dividing the community among many projects is a waste of time and energy. Particular in this type of projects in which the community is not as large the numerical programming and there is already an established standard.

That is is why triggered by his contribution to PintMatthieuDartiailh and I have decided to work together. He did some awesome instrumentation related coding in eapii and HQCMeas. Other people joined these initial conversations and a python instrumentation initiative that we have called LabPy was born. We have created an organization in GitHub to host Lantz and other projects with a common goal: making instrumentation better in Python.

Matthieu has been championing the refactoring of Lantz, putting the best of the different toolkits together in a cohesive package. I have no doubt that Lantz 0.4 will be even better than 0.3

Join us at https://github.com/LabPy
Categories: FLOSS Project Planets

Kubuntu Site Revamped

Planet KDE - Mon, 2015-08-31 10:34

With the move to Plasma 5, updating the Kubuntu website seemed timely. Many people have contributed, including Ovidiu-Florin Bogdan, Aaron Honeycutt, Marcin Sągol and many others.

We want to show off the beauty of Plasma 5, as well as allow easy access for Kubuntu users to the latest news, downloads, documentation, and other resources.

We want your help! Whether you code/program or not.

Web development, packaging, bug triage, documentation, promotion and social media are all areas where we can use your talents and skill, as well as offering help to new or troubled users.

For instance, people regularly report problems on Facebook, Reddit, Google+, Twitter, Telegram now and of course, #kubuntu on Freenode IRC, rather than filing bugs.

Sometimes their problems are easily solved, sometimes they have encountered real bugs, which we can help them file.

Please use our new site to find what you need, and tell us if you find something which needs improvement.

Categories: FLOSS Project Planets

Bryan Pendleton: To Say Nothing of the Dog: a very short review.

Planet Apache - Mon, 2015-08-31 09:58

What started out as a book to take on my backpacking trip turned in to a two week obsession, but now I've finally finished Connie Willis's marvelous To Say Nothing of the Dog.

The book somehow combines historical fiction, romance, mystery, and adventure into a time travel novel about the London Blitz, with generous amounts of Coventry conspiracy and how decoded Enigma messages were always concealed by sending a recon plane to try to conceal the fact that the codes were broken.

Oh, somehow that all makes it sounds dry and not any fun at all.

It's more like: what would a time travel book be, if it was written by P.G. Wodehouse?

Or, of course, by Jerome K. Jerome?

Well, it would be Connie Willis's joyful To Say Nothing of the Dog, a light but lovely masterpiece that stands up very well 20 years after it was first released.

And now, back to my other reading...

Categories: FLOSS Project Planets

Annertech: How to Integrate your Drupal Website with CiviCRM

Planet Drupal - Mon, 2015-08-31 08:54
How to Integrate your Drupal Website with CiviCRM

Continuing in our series of integrating CRMs with Drupal, we're now going to take a look at CiviCRM, an open-source, web-based CRM aimed at charities and non-profits which integrates closely with Drupal, Joomla and WordPress.

Categories: FLOSS Project Planets

Mike Driscoll: PyDev of the Week: Kenneth Love

Planet Python - Mon, 2015-08-31 08:30

This week we welcome Kenneth Love as our PyDev of the Week. He is active in the Python community via the Portland chapter of Django Ladies. He works for Team Treehouse creating Python courses. Let’s spend some time getting to know him better.

Can you tell us a little about yourself (hobbies, education, etc):

Sure. I live near Portland, OR with my family. I work for Treehouse, which is an online school aimed at getting people ready for jobs in the tech world through videos and code challenges. I teach all things Python.

I have a multimedia design degree but I’ve never used it professionally. I’m self-taught when it comes to programming if you don’t count all of the people that wrote the documentation and blogs I used to learn things (thanks tech writers!).

I like to collect, and sometimes play, tabletop games. I currently have about 90-100 games in my collection.

Why did you start using Python?

It was actually kind of an accident. I was really interested in Ruby and really tired of writing PHP. But Ruby wasn’t proving easy to learn and a friend and I both thought Python looked neat. He got me to do the Django tutorial and I fell in love. I’ve pushed to use Python at every job I’ve had since then and, luckily, usually gotten my way if the place wasn’t already using it.

As for why, well, I came from PHP The fact that Python had PEP 8 just tweaked that little OCD part of my brain that wanted code to be uniform. I also really loved the terseness of Python so I could get my thoughts into the computer faster.

What other programming languages do you know and which is your favorite?

Having spent most of my career as a full-stack freelancer, I’m familiar with PHP and JavaScript. I’m also fairly proficient at CSS and HTML since I’ve had to bring them out from time-to-time, too. Like most programmers that have learned more than one language, I can usually read another language, even if I don’t get the entire thing.

Honestly, Python is still my favorite. But, with my love for functional programming, I keep trying to pick up Haskell or Lisp, so that might change if I ever actually sit down to learn one of these.

What projects are you working on now?

Coming from a background of freelancing, now that I have a full-time job, I try to keep my downtime, well, down. I don’t work on a lot of side projects. My friend Chris Jones and I still maintain django-braces, our package of mixins for Django’s class-based generic views but that’s about it.

I actually just wrapped up co-organizing the first Django Girls Portland with my good friend Lacey Williams Henscel, and we’re planning on doing another one of those soon (February?)

And, at work, I’m currently working on 2 or 3 new courses and workshops.

Which Python libraries are your favorite (core or 3rd party)?

3rd party: Flask and Django are huge for me. Requests is just beautiful.

Core: As weird as it might be, I actually really like the re library. Regular Expressions are horrible at first but they are such a powerful tool. I’m also a big fan of the turtle library for teaching Python to kids.

What is your take on the current market for Python programmers?

I think the current market is really solid. I haven’t heard of any Python programmers looking for work for long, so we seem to be in high demand. That said, the world is changing, especially for web-related jobs, so start learning asyncio!

Is there anything else you’d like to say?

Thanks so much for having me on here! Also, we need more teachers in the Python world! Write blog posts, write documentation, teach classes, help Django Girls and PyLadies, etc.

Categories: FLOSS Project Planets

OSTraining: Let's Welcome GoDaddy Into Our Open Source Communities

Planet Drupal - Mon, 2015-08-31 08:29

This week, we published the first post about Drupal on GoDaddy.com: "What you need to know about Drupal 8".

GoDaddy is a really fascinating company, but most of you probably didn't expect them to be interested in Drupal.

You probably know some of the GoDaddy story.

You've seen the commercials with half-naked women. You may know about Bob Parsons, their ex-CEO, who was a public relations genius ... until he wasn't. Parsons built an empire on using SuperBowl commercials, NASCAR sponsorship and risqué advertising to sell hosting and domain names. But, in later years, Parsons made several major PR gaffes. And, it didn't help that GoDaddy also had a reputation for poor-quality hosting. If you've been to a tech conference, you've probably heard speakers making jokes at GoDaddy's expense.
Categories: FLOSS Project Planets

Mike Crittenden: How Drupal 7 Works, Part 1: The Request

Planet Drupal - Mon, 2015-08-31 05:53

This is a chapter out of my in-progress book, Drupal Deconstructed. You can read it online for free, download it as a PDF/ePUB/MOBI, or contribute to it on GitHub.

Have you ever wondered how Drupal does what it does? Good, me too. In this series of posts, I'm going to explain what Drupal is doing behind the scenes to perform its magic.

In Part 1, we'll keep it fairly high level and walk through the path a request takes as it moves through Drupal. In later parts, we'll take deeper dives into individual pieces of this process.

Step 0: Some background information

For this example, let's pretend that George, a user of your site, wants to visit your About Us page, which lives at http://oursite/about-us.

Let's also pretend that this page is a node (specifically, the node with nid of 1) with a URL alias of about-us.

And to keep things simple, we'll say that we're using Apache as our webserver, and there's nothing fancy sitting in front of Drupal, such as a caching layer or the like.

So basically, we're talking about a plain old Drupal site on a plain old webserver.

Step 1: The request hits the server

There's some pretty hot action that happens before Drupal even sees the request. George's browser sends a request to http://oursite.com/about-us, and this thing we call the internet figures out that that should go to our server. If you're not well versed on how that part happens, you may benefit from reading this infographic on how the internet works.

Once our server has received the request, that's when the fun begins. This request will land on port 80, and Apache just so happens to be listening on that port, so Apache will immediately see the request and decide what to do with it.

Since the request is going to oursite.com then Apache can look into its configuration files to see what the root directory is for oursite.com (along with lots of other info about it which is out of scope for this post). We'll say that the root directory for our site is /var/www/oursite, which is where Drupal lives. So Apache is going to send the request there.

Step 2: The .htaccess file

But Drupal hasn't taken over just yet. Drupal ships with a .htaccess file which is a way of telling Apache some things. In fact, Drupal's .htaccess tells Apache a whole lot of things. The most important things it does are:

  1. Protects files that could contain source code from being viewable, such as .module files or .inc files, both of which contain PHP.
  2. Allows requests that point directly to files in the filesystem to pass through without touching Drupal.
  3. Redirects all other requests to Drupal's index.php file.

It also does some other fancy things such as disabling directory indexes and allowing Drupal to serve gzipped CSS and JS, but those are the biggies.

So, in our case, Apache will see that we're looking for /about-us, and will say:

  1. Is this request trying to access a file that it shouldn't? No.
  2. Is this request directly pointing to a file in the filesystem? No.
  3. Then this request should be sent to index.php!

And away we go...

Step 3: Drupal's index.php file

Finally, we have reached Drupal, and we're looking at PHP. Drupal's index.php is super clean, and in fact only has 4 lines of code, each of which are easy to understand.

Line 1: Define DRUPAL_ROOT

php define('DRUPAL_ROOT', getcwd());

This just sets a constant called DRUPAL_ROOT to be the value of the current directory that index.php is in. This constant is used all over the place in the Drupal core codebase.

In our case, this means that DRUPAL_ROOT gets set to /var/www/oursite.

Lines 2 and 3: Bootstrap Drupal

php require_once DRUPAL_ROOT . '/includes/bootstrap.inc'; drupal_bootstrap(DRUPAL_BOOTSTRAP_FULL);

These lines run a full Drupal bootstrap, which basically means that they tell Drupal "Hey, grab all of the stuff you're going to need before we can get to actually handling this request."

For more information about the bootstrap process, see Part 2 of this series.

Line 4: Do everything else

php menu_execute_active_handler();

This simple looking function is where the heart and soul of Drupal lives. For more information about what happens in this ball of wax, visit the Menu Router chapter.

This is a chapter out of my in-progress book, Drupal Deconstructed. You can read it online for free, download it as a PDF/ePUB/MOBI, or contribute to it on GitHub.

Categories: FLOSS Project Planets

TheodorosPloumis blog: My Watchlist for DrupalCon Barcelona 2015

Planet Drupal - Mon, 2015-08-31 05:37

Time has almost come! The yearly Europe DrupalCon will start on September 21nth 2015 in Barcelona. It will be my fist time on a DrupaCon even if I am working with Drupal for 7 years now. Don't ask me why, there are no excuses.

As a frontend developer and a freelancer it was too hard for me to select the sessions to attend. DrupalCon hosts the best of the best talks for Drupal out there and the benefits for Drupal *ers is enormous. Sessions will be recorded but live sessions are these that matter.

So, after a deep review on each session I decided to attend sessions that are dealing with the keywords below (notice that order matters):

  1. DevOps
  2. CI (Continuous Integration) and CD (Continuous Development)
  3. Drupal 8 plug system
  4. Headless Drupal
  5. Testing

I am sure that with some sessions (eg these related with Docker or AngularJS) there will be a "crowding". Remember that. It happens everywhere, from local meetups to large events. Docker, Headless and Devops are here to stay and will bother us for years. Thankfully Drupal with the new 8.x version is "up to date" with the new technologies and methods and the DrupalCon shows that movement clearly.

Here is My Schedule (I know that there are duplicates or even triplicates but I will decide last minute. Maybe after asking other attendees...)


09/22/2015 - 11:00-09/22/2015 - 12:00 Session Speaker(s) Experience level Room Symfony for Drupal developers Crell Intermediate 115


09/22/2015 - 13:00-09/22/2015 - 14:00 Session Speaker(s) Experience level Room Drupal development with Vagrant     128 Fundamentals of Front-End Ops prestonso Intermediate 111: Adyax Docker powered team and deployment daven Intermediate 118-119: Platform.sh


09/22/2015 - 14:15-09/22/2015 - 15:15 Session Speaker(s) Experience level Room Drupal 8 Plugin Deep Dive EclipseGc Advanced 116: Pantheon Headful Drupal nod_ Intermediate 122-123: Interoute


09/22/2015 - 15:45-09/22/2015 - 16:45 Session Speaker(s) Experience level Room Composer and Drupal 8 webflo, tstoeckler Intermediate 122-123: Interoute Dependency Injection, what, why, how and when mr_r_miller Intermediate 115


09/22/2015 - 17:00-09/22/2015 - 18:00 Session Speaker(s) Experience level Room Expose Drupal with RESTful e0ipso Intermediate 112: Exove Caching at the Edge: CDNs for everyone Wim Leers, Fabianx Advanced 117: Acquia


09/23/2015 - 10:45-09/23/2015 - 11:45 Session Speaker(s) Experience level Room Local vs. Remote Development: Do Both by Syncing Your Site From Kitchen to Cloud With Jenkins SolomonGifford Beginner 116: Pantheon Prototypes and Drupal: from designing in-browser to implementing custom templates ygerasimov, mortenc Intermediate 111: Adyax


09/23/2015 - 13:00-09/23/2015 - 14:00 Session Speaker(s) Experience level Room Docker and DevOps – Building platforms globally William Morrish - Interoute Intermediate 120-121 Linked Data 101: The Non-Terrifying Intro to Semantic Content nozurbina Intermediate 115


09/23/2015 - 14:15-09/23/2015 - 15:15 Session Speaker(s) Experience level Room Drupal and Security: what you need to know scor, klausi Intermediate 111: Adyax


09/23/2015 - 15:45-09/23/2015 - 16:45 Session Speaker(s) Experience level Room Hassle-free Hosting and Testing with DevShop & Behat. Jon Pugh Intermediate 112: Exove Decoupling Drupal modules into PHP libraries bojanz Advanced 111: Adyax


09/23/2015 - 17:00-09/23/2015 - 18:00 Session Speaker(s) Experience level Room What's your type? Andreu Balius Intermediate 113: Amazee Labs Perfecting Drupal IA: Harmonious Menus, Paths, and Breadcrumbs Jody Lynn Intermediate 118-119: Platform.sh


09/24/2015 - 10:45-09/24/2015 - 11:45 Session Speaker(s) Experience level Room Making Drupal a better out-of-the-box product: Report on usability testing results and how we can make 8.1.x+ shine webchick, Bojhan, LewisNyman Beginner 122-123: Interoute Building the Front End with Angular.js ceng Intermediate 112: Exove


09/24/2015 - 13:00-09/24/2015 - 14:00 Session Speaker(s) Experience level Room CIBox - full stack open source Continuous Integration flow for Drupal/Symfony teams podarok, ygerasimov Intermediate 113: Amazee Labs


09/24/2015 - 14:15-09/24/2015 - 15:15 Session Speaker(s) Experience level Room The wonderland of HTTP in PHP dawehner Intermediate 115 Introducing Probo.CI tizzo Intermediate 118-119: Platform.sh


Categories: FLOSS Project Planets

Interview with Brian Delano

Planet KDE - Mon, 2015-08-31 04:00
Could you tell us something about yourself?

My name is Brian Delano. I’m a musician, writer, futurist, entrepreneur and artist living in Austin, Texas. I don’t feel I’m necessarily phenomenal at any of these things, but I’m sort of taking an approach of throwing titles at my ego and seeing whichones stick and sprout.

Do you paint professionally, as a hobby artist, or both?

I’m more or less a hobby artist. I’ve made a few sales of watercolors here and there and have had my pieces in a few shows around town, but, so far, the vast majority of my art career exists as optimistic speculation between my ears.

What genre(s) do you work in?

I mostly create abstract art. I’ve been messing around with web comic ideas a bit, but that’s pretty young on my “stuff I wanna do” list. Recently, I’ve been working diligently on illustrating a children’s book series that I’ve been conceptualizing for a few years.

Whose work inspires you most — who are your role models as an artist?

Ann Druyan & Carl Sagan, Craig & Connie Minowa, Darren Waterston, Cy Twombly, Theodor Seuss Geisel, Pendelton Ward, Shel Silverstein and many others.

How and when did you get to try digital painting for the first time?

My first exposure to creating digital art was through the mid-nineties art program Kid Pix. It was in most every school’s computer lab and I thought it was mind-blowingly fun. I just recently got a printout from one of my first digital paintings from this era (I think I was around 8 or so when I made it) and I still like it. It was a UFO destroying a beach house by shooting lightning at it.

What makes you choose digital over traditional painting?

Don’t get me wrong, traditional (I call it analog :-P) art is first and foremost in my heart, but when investment in materials and time is compared between the two mediums, there’s no competition. If I’m trying to make something where I’m prototyping and moving elements around within an image while testing different color schemes and textures, digital is absolutely the way to go.

How did you find out about Krita?

I was looking for an open source alternative to some of the big name software that’s currently out for digital art. I had already been using GiMP and was fairly happy with what it offered in competition with Photoshop, but I needed something that was more friendly towards digital painting, with less emphasis on imaging. Every combination of words in searches and numerous scans through message boards all pointed me to Krita.

What was your first impression?

To be honest, I was a little overwhelmed with the vast set of options Krita has to offer in default brushes and customization. After a few experimental sessions, some video tutorials, and a healthy amount of reading through the manual, I felt much more confident in my approach to creating with Krita.

What do you love about Krita?

If I have a concept or a direction I want to take a piece, even if it seems wildly unorthodox, there’s a way to do it in Krita. I was recently trying to make some unique looking trees and thought to myself ” I wish I could make the leafy part look like rainbow tinfoil…” I messed around with the textures, found a default one that looked great for tinfoil, made a bunch of texture circles with primary colored brush outlines, selected all opaque on the layer, added a layer below it, filled in the selected space with a rainbow gradient, lowered the opacity a bit on the original tin foil circle layer, and bam! What I had imagined was suddenly a (digital) reality!

What do you think needs improvement in Krita? Is there anything that really annoys you?

Once in a while, if I’m really pushing the program and my computer, Krita will seem to get lost for a few seconds and become non responsive. Every new release seems to lessen this issue, though, and I’m pretty confident that it won’t even be an issue as development continues.

What sets Krita apart from the other tools that you use?

Krita feels like an artist’s program, created by artists who program. Too many other tools feel like they were created by programmers and misinterpreted focus group data to cater to artists’ needs that they don’t fully understand. I know that’s a little vague, but once you’ve tried enough different programs and then come to Krita, you’ll more than likely see what I mean.

If you had to pick one favourite of all your work done in Krita so far, what would it be, and why?

I’m currently illustrating a children’s book series that I’ve written which addresses the size and scope of everything and how it relates to the human experience. I’m calling the series “BIG, small & Me”, I’m hoping to independently publish the first book in the fall and see where it goes. I’m not going to cure cancer or invent faster than lightspeed propulsion, but if I can inspire the child that one day will do something this great, or even greater, then I will consider my life a success.

What techniques and brushes did you use in it?

I’ve been starting out sketching scenes with the pencil brushes, then creating separate layers for inking. Once I have the elements of the image divided up by ink, I turn off my sketch layers, select the shapes made by the ink layers and then fill blocks on third layers. When I have the basic colors of the different elements completed in this manner, I turn off my ink lines and create fourth and fifth layers for texturing and detailing each element. There are different tweaks and experimental patches in each page I’ve done, but this is my basic mode of operation in Krita.

Where can people see more of your work?

I have a few images and a blog up at artarys.com, and will hopefully be doing much more with that site  pretty soon. I’m still in the youngish phase of most of the projects I’m working on so self promotion will most likely be ramping up over the next few months. I’m hoping to set up a kickstarter towards the end of the year for a first pressing of BIG, small & Me, but until then most of my finished work will end up on either artarys.com or my facebook page.

Anything else you’d like to share?

It’s ventures like Krita that give me hope for the future of creativity. I am so thankful that there are craftspeople in the world so dedicated to creating such a superior tool for digital art.

Categories: FLOSS Project Planets

Martín Ferrari: Romania

Planet Debian - Mon, 2015-08-31 02:59

It's been over 2 years since I decided to start a new, nomadic life. I had the idea of blogging about this experience as it happened, but not only I am incredibly lazy when it comes to writing, most of the time I have been too busy just enjoying this lifestyle!

The TL;DR version of these last 2 years:

  • Lounged a bit in Ireland after leaving work, went on a great road trip along the West coast.
  • Lived in Nice 3 months, back in the same house I lived between 2009 and 2010.
    • During that time, my dad visited and took him for a trip around Nothern Italy, the Côte d'Azur and Paris; then travelled to DebConf in Switzerland, visited Gregor in Innsbruck, and travelled back to Nice by train, crossing the alps and a big chunk of Italy.
  • Then went to Buenos Aires for 3 months (mom was very happy).
  • Back in Europe, attended Fosdem, and moved to Barcelona for 3 months; so far, the best city I ever lived in!
  • Went back to Dublin for a while, ended up staying over 8 months, including getting a temporary job at a normal office (booo!).
    • Although one of these months I spent travelling in the States (meh).
    • And of course, many more short trips, including a couple of visits to Barcelona, Lille, Nice, and of course Brussels for Fosdem.
  • Again went to Buenos Aires, only 2 months this time.
  • Another month in Dublin, then holidays visiting my friends in Lisbon, wedding in Oviedo, and a road trip around Asturias and Galicia.
  • From Spain I flew to Prague and stayed for 2 months (definitely not enough).
  • Quick trip to Dublin, then CCC camp and DebConf in Germany.

And now, I am in Cluj-Napoca, Romania.

View from my window
Categories: FLOSS Project Planets

Martín Ferrari: IkiWiki

Planet Debian - Mon, 2015-08-31 00:55

I haven't posted in a very long time. Not only because I suck at this, but also because IkiWiki decided to stop working with OpenID, so I can't use the web interface any more to post.. Very annoying.

Already spent a good deal of time trying to find a solution, without any success.. I really don't want to migrate to another software again, but this is becoming a showstopper for me.

Categories: FLOSS Project Planets

Russ Allbery: Review: Through Struggle, The Stars

Planet Debian - Sun, 2015-08-30 23:54

Review: Through Struggle, The Stars, by John J. Lumpkin

Series: Human Reach #1 Publisher: John J. Lumpkin Copyright: July 2011 ISBN: 1-4611-9544-6 Format: Kindle Pages: 429

Never let it be said that I don't read military SF. However, it can be said that I read books and then get hellishly busy and don't review them for months. So we'll see if I can remember this well enough to review it properly.

In Lumpkin's future world, mankind has spread to the stars using gate technology, colonizing multiple worlds. However, unlike most science fiction futures of this type, it's not all about the United States, or even the United States and Russia. The great human powers are China and Japan, with the United States relegated to a distant third. The US mostly maintains its independence from either, and joins the other lesser nations and UN coalitions to try to pick up a few colonies of its own. That's the context in which Neil and Rand join the armed services: the former as a pilot in training, and the latter as an army grunt.

This is military SF, so of course a war breaks out. But it's a war between Japan and China: improved starship technology and the most sophisticated manufacturing in the world against a much larger economy with more resources and a larger starting military. For reasons that are not immediately clear, and become a plot point later on, the United States president immediately takes an aggressive tone towards China and pushes the country towards joining the war on the side of Japan.

Most of this book is told from Neil's perspective, following his military career during the outbreak of war. His plans to become a pilot get derailed as he gets entangled with US intelligence agents (and a bad supervisor). The US forces are not in a good place against China, struggling when they get into ship-to-ship combat, and Neil's ship goes on a covert mission to attempt to complicate the war with political factors. Meanwhile, Rand tries to help fight off a Chinese invasion of one of the rare US colony worlds.

Through Struggle, The Stars does not move quickly. It's over 400 pages, and it's a bit surprising how little happens. What it does instead is focus on how the military world and the war feels to Neil: the psychological experience of wanting to serve your country but questioning its decisions, the struggle of working with people who aren't always competent but who you can't just make go away, the complexities of choosing a career path when all the choices are fraught with politics that you didn't expect to be involved in, and the sheer amount of luck and random events involved in the progression of one's career. I found this surprisingly engrossing despite the slow pace, mostly because of how true to life it feels. War is not a never-ending set of battles. Life in a military ship has moments when everything is happening far too quickly, but other moments when not much happens for a long time. Lumpkin does a great job of reflecting that.

Unfortunately, I thought there were two significant flaws, one of which means I probably won't seek out further books in this series.

First, one middle portion of the book switches away from Neil to follow Rand instead. The first part of that involves the details of fighting orbiting ships with ground-based lasers, which was moderately interesting. (All the technological details of space combat are interesting and thoughtfully handled, although I'm not the sort of reader who will notice more subtle flaws. But David Weber this isn't, thankfully.) But then it turns into a fairly generic armed resistance story, which I found rather boring.

It also ties into the second and more serious flaw: the villain. The overall story is constructed such that it doesn't need a personal villain. It's about the intersection of the military and politics, and a war that may be ill-conceived but that is being fought anyway. That's plenty of conflict for the story, at least in my opinion. But Lumpkin chose to introduce a specific, named Chinese character in the villain role, and the characterization is... well.

After he's humiliated early in the story by the protagonists, Li Xiao develops an obsession with killing them, for his honor, and then pursues them throughout the book in ways that are sometimes destructive to the Chinese war efforts. It's badly unrealistic compared to the tone of realism taken by the rest of the story. Even if someone did become this deranged, it's bizarre that a professional military (and China's forces are otherwise portrayed as fairly professional) would allow this. Li reads like pure caricature, and despite being moderately competent apart from his inexplicable (but constantly repeated) obsession, is cast in a mustache-twirling role of personal villainy. This is weirdly out of place in the novel, and started irritating me enough that it took me out of the story.

Through Struggle, The Stars is the first book of a series, and does not resolve much by the end of the novel. That plus its length makes the story somewhat unsatisfying. I liked Neil's development, and liked him as a character, and those who like the details of combat mixed with front-lines speculation about politics may enjoy this. But a badly-simplified mustache-twirling victim and some extended, uninteresting bits mar the book enough that I probably won't seek out the sequels.

Followed by The Desert of Stars.

Rating: 6 out of 10

Categories: FLOSS Project Planets

Armin Ronacher: Samsung Pay's MTS Transactions and Merchant's Ability to Detect "Cloned" Magstripe Tracks

Planet Python - Sun, 2015-08-30 20:00

I have a weird obsession with payment systems. They fascinate me. I find it very satisfying to make a credit card transaction and to get a text message confirming the purchase on my phone a second afterwards. As someone obsessed with networks, scalability and user experience I find this a very interesting field even though it's embedded in probably the least agile and most regulated industry. But not just the technology is interesting, also the fraud aspect is. Fraud prevention is an equally interesting topic to ponder about.

What makes frauds in payments so interesting is that there are many different payment protocols that exist throughout the world and your credit card is valid with almost all of them. The fraud vectors are huge and very often the only thing that keeps fraud rates down is a random spot checks and common sense.

The reason my interest got piqued again recently was Samsung Pay, particularly the MST part. MST, if you are not familiar with it, stands for magnetic secure transmission. The idea is that the phone emits a magnetic field that carries the information of track 2 on a credit card (at least in principle). What this means is that you can go to a lot of magstrip readers, hold your phone to it, and the reader thinks the card was swiped. (Assuming there are no other checks that a card is in a slot)

From a fraud perspective this seems crazy. You scan someone's credit card, duplicate it onto your phone and off you go. Here are the results of my investigation about how this is supposed to be used securely.

But for this we need to cover some ground.

A Bit of History

If we don't go too far back, the earliest forms of standardized credit card processing were based on a credit card number. The credit card number in itself is split into two parts. The first six digits are the IIN or Issuer Identification Number. It identifies the network of the card (MasterCard, AMEX, Visa, etc.) and might identify the bank within that network. The rest (the remaining 10-13 digits) are the PAN or Primary Account Number. IIN + PAN + expiration date + name of cardholder are the basic requirements for making a credit card transaction.

However as you can guess, since all that information is on the card there is very little that actually protects a payment. That's why on most of those transactions done that way they will also ask for the signature of the cardholder. That signature really only plays a role if the transaction gets disputed.

The Magstripe

What makes credit cards convenient for in-store purchases is that you do not need to write down numbers, instead you can "swipe" the card. At least you do that in the US ;) When you swipe the card, the reader reads the two tracks on the magstripe. They are almost the same with a different data density. Both tracks contain: IIN + PAN, country code, expiration date and a field for discretionary data. It also contains the service code. The service code tells the terminal how the card wants to be confirmed (does it work internationally, does it need online verification, does it need a pin, AM only etc.)

Track 1 which has higher density also contains the card holder name and has a bit of extra space for the discretionary data. So if you swipe the card, you have pretty much all the info that's written on it. What's in the discretionary data we will cover later.

Transaction Types and Security Codes

An important tool for understanding fraud and to combat it is to split the one huge problem of credit card fraud into smaller sub-problems. In particular the most important split is "card present" or "card not present" (CNP) which should indicate if the physical card was present at the origin of the transaction or not. So how do you do that if the data is the same? The earliest form of trying to combat this was the addition of two security codes. They have various different names (CVC, CVV, CID) and on most cards it comes in two flavors: code 1 and code 2. One is stored in the magstripe in the discretionary data field, the other is printed on the back of the card. The idea is that you can differentiate between transactions carrying no security code, or CVC1 or CVC2. If someone skimmed your card through a magstripe reader, they can get to all data with the exception of CVC2. If someone takes your card number via phone they won't get your CVC1.

At this point you can already see that there are different types of transactions with different fraud parameters. If someone does not use a CVC code it does not mean that the transaction will be declined outright, but it indicates that something is fishy.


EMV is the answer for all problems and has been for a long time. The reason it plays little role here is because EMV in itself is secure (bad chip implementations notwithstanding). However EMV is still not rolled out in the US and as such, there is a huge market where magstripe is still something people need to deal with. Also EMV without NFC support cannot support MST which is the topic of discussion here. We will come back to that later however.

Modern Transaction Types

What should be clear now is that there are many different ways to make a credit card transaction. But what is that actual transaction? At one point you want your money. If you get your money or not as a merchant depends on if the transaction was fraudulent or not, and if it was, if you had a chance to detect the fraud yourself.

At one point you need to actually try to charge the issuer of the card as a merchant. Ideally you do it as quickly as possible. If you do it at the time you swipe the card, you might directly go online and check with the card issuer if everything is in order. This happens in most terminals now where the terminal directly talks to the bank to record the transaction.

A more evolved version of this method is to replace the magstripe with a EMV chip. That chip can a challenge/response game with the payment terminal which means that each purchase is unique and skimming the data off the chip will not be any good for future transactions. That again will only work for transactions that actually use the EMV chip. If you just steal the magstripe and go to the US where all readers are magstripe, this will do absolutely nothing to you.

Likewise for online payments many issuing banks will use 3D Secure for payment verification. The idea is that the online form for your credit card number also presents you an iframe with an extra input form by the bank. This allows a second factor to confirm the payment. For instance on my Austrian Erste Mastercard the second factor is a confirmation with a transaction code. The transaction will be declined unless I confirm the payment in the iframe with a unique token sent to my phone via SMS.

Tokenization: Apple Pay / Samsung Pay

In an ideal world the magstripe would no longer exist and all terminals would use the EMV chip and online transactions would require 3D secure. However that's clearly not happening because the US seem to take bloody ages to replace their infrastructure. And not just the US. The idea to force everybody to newer and in this case kinda incompatible technologies did not work for many years, so an alternative has to appear.

One alternative is what's often called "Tokenization" and oddly enough, it works by replacing the customer equipment rather then the merchant one. Instead of making all merchants upgrade their terminals to support EMV, you instead upgrade the customer's credit card to a phone.

To understand why that's necessary you need to understand that NFC is not always NFC and in case of Samsung it might not even involve an actual RFID chip at all. In Europe when you use NFC for a payment the card transmits a response to a challenge like an EMV chip is. The transaction gets confirmed safely either directly by the card or in combination with the user's PIN. In either case the transaction gets confirmed through the issuer. In the United States however EMV often does not exist, so NFC has an alternative method where it transmits the MSD (magnet stripe data) instead. Apple Pay can do that similar to how Samsung Pay can transmit the very same data via magnetic pulses or NFC.

So how does that make anything any more secure? Because of tokenization. Remember how the credit card number is split into IIN and PAN and how the magstripe contains this extra discretionary data. The idea is that assuming the terminal is connected to the internet and verifies transactions with the issuing bank the phone can play a little trick. The bank provides the phone with a method to "clone" the card securely onto the phone. At this point the phone acts as a hardware token generator. Whenever it confirms a transaction it replaces the PAN with a uniquely generated one and places some extra data in the discretionary data part. Both of that information gets transmitted to the issuing bank or TSP (token service provider, so MasterCard or Visa) where the token PAN (DPAN) gets replaced for the real PAN. The actual flow is a bit more complex than that, but in the end the transaction goes through like before.

The Merchant and Tokenization

The important part here however is the merchant and this is where things get tricky. With Apple Pay the transaction is always done through a form of NFC. Either NFC with MSD or proper EMV NFC. It means that the merchant explicitly agrees with this form of payment and will introduce the system to the employees that accept the transactions. To confirm such a payment as a merchant you just make sure that the transaction is made from an iphone and everything else "should be secure". The only case of fraud is if someone managed to get a card on their phone which they were not entitled too, but that's the bank's problem because they should make that flow secure.

The situation however is different with Samsung Pay and the reason for that is MST. As Samsung Pay works with non NFC POS terminals the question is how a merchant can differ a phone that uses Tokenization properly or a fraudulent phone that just relays the magstripe tracks from a stolen card. In fact, the merchant can't really do anything there because the transaction is as far as I know indistinguishable from what is shown on the terminal. The only party that could reliably block the transaction is the issuer or TSP. This interestingly enough can be solved by supporting EMV :)

A modern card (one that would be used with Samsung Pay) could come with magstripe and EMV and the magstripe could indicate that the card prefers the chip over swiping. In this case you could still clone the magstripe into your phone, but the transaction would be declined if it used neither tokenization nor the chip. For this to work however, all merchants need to support EMV which currently is not the case in the US.

The Non EMV Apocalypse of 2015

Something interesting is going to happen end of October 2015. The US will finally start to force merchants to upgrade to terminals that support EMV. From that point onwards any card that has an EMV chip, but the chip was not used for the transaction and that transaction was fraudulent will become the merchant's problem. Assuming Samsung Pay becomes widespread it could make this liability shift a bit more painful because as a merchant you can not tell a good Samsung phone from a bad Samsung phone, whereas you could probably tell an original credit card with embossed numbers from a fake card with mismatching numbers and making your own embossed cards with all the cards you skimmed is a lot more work than to clone a card into a phone.

So maybe EMV will become a bigger thing as a result of Samsung Pay even if the technology in itself has some potential for magstripe abuse.

Death of MSD

Interestingly enough the roll-out of EMV in the US might have some bad aspects for European travellers and others. Our cards have a very different fraud profile than American ones because domestic transactions are done via EMV for nearly thirty nears now, with the liability shift having happened more than 10 years ago. In Europe cards prefer chip and pin for terminals and NFC is only supported for EMV transactions.

The US terminals might use the MSD data for NFC however. So as a European customer you might see an NFC logo somewhere, but because it uses NFC MSD your European bank will decline the transaction because they only allow EMV based NFC. This is to be seen however, right now NFC terminals in the US are still not very widespread and the liability shift did not happen yet.

Safety of Samsung Pay

So is it safe? Implemented correctly with tokenization Samsung Pay seems pretty safe.

Will merchants like it? If they have EMV terminals, they will not have a problem with it. If they only have legacy terminals without chip support, they might become fraud magnets and they have little method to defend themselves against it.

Will the magstripe finally die? Seems like magstripe found a second coming in the US thanks to tokenization, MSD NFC and maybe even Samsung Pay but most likely only as a transitional technology for EMV.

I'm actually quite interested in if there are means of detecting a relayed magstripe track for a merchant. If someone knows, please let me know and I will amend the article to reflect that.

Categories: FLOSS Project Planets

Next week I’ll be in Randa

Planet KDE - Sun, 2015-08-30 17:24

Since Akademy 2015 KSecrets Service development continued. I did lots of code cleanup. The async API now uses QFuture and the secrets file backend, based on libgcrypt, was added. Tests are there to confirm it’s not yet working.

One week from now I’ll be in Randa. The hacking ambiance will surely help to hopefully get a first pre-alpha version of the service. I also look forward to peer reviews and feedback on this new codebase.

This kind of events is made possible by our generous donors. If you’d like to join them and donate, helping the KDE community and me, then just click the image below:

Categories: FLOSS Project Planets

Andrew Cater: Rescuing a Windows 10 failed install using GParted Live on CD

Planet Debian - Sun, 2015-08-30 16:09
Windows 10 is here, for better or worse. As the family sysadmin, I've been tasked to update the Windows machines: ultimately, failure modes are not well documented and I needed Free software and several hours to recover a vital machine.

The "free upgrade for users of prior Windows versions" is a limited time offer for a year from launch. Microsoft do not offer licence keys for the upgrade: once a machine has updated to Windows 10 and authenticated on the 'Net, then a machine can be re-installed and will be regarded by Microsoft as pre-authorised. Users don't get the key at any point.

Although Microsoft have pushed the fact that this can be done through Windows Update, there is the option to use Microsoft's Media Creation tool to do the upgrade directly on the Windows machine concerned. This would be necessary to get the machine to upgrade and register before a full clean install of Windows 10 from media.

This Media Creation Tool failed for me on three machines with "Unable to access System reserved partition'

All the machines have SSDs from various makers: a colleague suggested that resizing the partition might enable the upgrade to continue.  Of the machines that failed, all were running Windows 7 - two were running using BIOS, one using UEFI boot on a machine with no Legacy/CSM mode.

Using GParted live .iso  - itself based on Debian Live - allowed me to resize the System partition from 100MiB to 200MiB by moving the Windows partition but  Windows became unbootable.

In two cases, I was able to boot from DVD Windows installation media and make Windows bootable again at which point the Microsoft Media Creation Tool could be used to install Windows 10

The UEFI boot machine proved more problematic: I had to create a Windows 7 System Repair disk and repair Windows booting before Windows 10 could proceed.

My Windows-using colleaague had used only Windows-based recovery disks: using Linux tools allowed me to repair Windows installations I couldn't boot

Categories: FLOSS Project Planets

Antonio Terceiro: DebConf15, testing debian packages, and packaging the free software web

Planet Debian - Sun, 2015-08-30 15:12

This is my August update, and by the far the coolest thing in it is Debconf.


I don’t get tired of saying it is the best conference I ever attended. First it’s a mix of meeting both new people and old friends, having the chance to chat with people whose work you admire but never had a chance to meet before. Second, it’s always quality time: an informal environment, interesting and constructive presentations and discussions.

This year the venue was again very nice. Another thing that was very nice was having so many kids and families. This was no coincidence, since this was the first DebConf in which there was organized childcare. As the community gets older, this a very good way of keeping those who start having kids from being alienated from the community. Of course, not being a parent yet I have no idea how actually hard is to bring small kids to a conference like DebConf. ;-)

I presented two talks:

  • Tutorial: Functional Testing of Debian packages, where I introduced the basic concepts of DEP-8/autopkgtest, and went over several examples from my packages giving tips and tricks on how to write functional tests for Debian packages.
  • Packaging the Free Software Web for the end user, where I presented the motivation for, and the current state of shak, a project I am working on to make it trivial for end users to install server side applications in Debian. I spent quite some hacking time during DebConf finishing a prototype of the shak web interface, which was demonstrated live in the talk (of course, as usual with live demos, not everything worked :-)).

There was also the now traditional Ruby BoF, where discussed the state and future of the Ruby ecosystem in Debian; and an in promptu Ruby packaging workshop where we introduced the basics of packaging in general, and Ruby packaging specifically.

Besides shak, I was able to hack on a few cool things during DebConf:

  • debci has been updated with a first version of the code to produce britney hints files that block packages that fail their tests from migrating to testing. There are some issues to be sorted out together with the release team to make sure we don’t block packages unecessarily, e.g. we don’t want to block packages that never passed their test suite — most the test suite, and not the package, is broken.
  • while hacking I ended up updating jquery to the newest version in the 1.x series, and in fact adopting it I guess. This allowed emeto drop the embedded jquery copy I used to have in the shak repository, and since then I was able to improve the build to produce an output that is identical, except for a build timestamp inside a comment and a few empty lines, to the one produced by upstream, without using grunt (.
Miscellaneous updates
Categories: FLOSS Project Planets

DebConf team: DebConf15: Farewell, and thanks for all the Fisch (Posted by DebConf Team)

Planet Debian - Sun, 2015-08-30 14:24

A week ago, we concluded our biggest DebConf ever! It was a huge success.

We are overwhelmed by the positive feedback, for which we’re very grateful. We want to thank you all for participating in the talks; speakers and audience alike, in person or live over the global Internet — it wouldn’t be the fantastic DebConf experience without you!

Many of our events were recorded and streamed live, and are now available for viewing, as are the slides and photos.

To share a sense of the scale of what all of us accomplished together, we’ve compiled a few statistics:

  • 555 attendees from 52 countries (including 28 kids)
  • 216 scheduled events (183 talks and workshops), of which 119 were streamed and recorded
  • 62 sponsors and partners
  • 169 people sponsored for food & accommodation
  • 79 professional and 35 corporate registrations

Our very own designer Valessio Brito made a lovely video of impressions and images of the conference.

Your browser does not support the video tag.

We’re collecting impressions from attendees as well as links to press articles, including Linux Weekly News coverage of specific sessions of DebConf. If you find something not yet included, please help us by adding links to the wiki.

We tried a few new ideas this year, including a larger number of invited and featured speakers than ever before.

On the Open Weekend, some of our sponsors presented their career opportunities at our job fair, which was very well attended.

And a diverse selection of entertainment options provided the necessary breaks and ample opportunity for socialising.

On the last Friday, the Oscar-winning documentary “Citizenfour” was screened, with some introductory remarks by Jacob Appelbaum and a remote address by its director, Laura Poitras, and followed by a long Q&A session by Jacob.

DebConf15 was also the first DebConf with organised childcare (including a Teckids workshop for kids of age 8-16), which our DPL Neil McGovern standardised for the future: “it’s a thing now,” he said.

The participants used the week before the conference for intensive work, sprints and workshops, and throughout the main conference, significant progress was made on Debian and Free Software. Possibly the most visible was the endeavour to provide reproducible builds, but the planning of the next stable release “stretch” received no less attention. Groups like the Perl team, the diversity outreach programme and even DebConf organisation spent much time together discussing next steps and goals, and hundreds of commits were made to the archive, as well as bugs closed.

DebConf15 was an amazing conference, it brought together hundreds of people, some oldtimers as well as plenty of new contributors, and we all had a great time, learning and collaborating with each other, says Margarita Manterola of the organiser team, and continues: The whole team worked really hard, and we are all very satisfied with the outcome. Another organiser, Martin Krafft adds: We mainly provided the infrastructure and space. A lot of what happened during the two weeks was thanks to our attendees. And that’s what makes DebConf be DebConf.

Our organisation was greatly supported by the staff of the conference venue, the Jugendherberge Heidelberg International, who didn’t take very long to identify with our diverse group, and who left no wishes untried. The venue itself was wonderfully spacious and never seemed too full as people spread naturally across the various conference rooms, the many open areas, the beergarden, the outside hacklabs and the lawn.

The network installed specifically for our conference in collaboration with the nearby university, the neighbouring zoo, and the youth hostel provided us with a 1 Gbps upstream link, which we managed to almost saturate. The connection will stay in place, leaving the youth hostel as one with possibly the fastest Internet connection in the state.

And the kitchen catered high-quality food to all attendees and their special requirements. Regional beer and wine, as well as local specialities, were provided at the bistro.

DebConf exists to bring people together, which includes paying for travel, food and accomodation for people who could not otherwise attend. We would never have been able to achieve what we did without the support of our generous sponsors, especially our Platinum Sponsor Hewlett-Packard. Thank you very much.

See you next year in Cape Town, South Africa!

Categories: FLOSS Project Planets

Philipp Kern: Automating the 3270 part of a Debian System z install

Planet Debian - Sun, 2015-08-30 13:36
If you try to install Debian on System z within z/VM you might be annoyed at the various prompts it shows before it lets you access the network console via SSH. We can do better. From within CMS copy the default EXEC and default PARMFILE:


Now edit DEBAUTO EXEC A and replace the DEBIAN in 'PUNCH PARMFILE DEBIAN * (NOHEADER' with DEBAUTO. This will load the alternate kernel parameters file into the card reader, while still loading the original kernel and initrd files.

Replace PARMFILE DEBAUTO A's content with this (note the 80 character column limit):

ro locale=C                                                              
s390-netdevice/choose_networktype=qeth s390-netdevice/qeth/layer2=true   
netcfg/get_ipaddress=<IPADDR> netcfg/get_netmask=       
netcfg/get_gateway=<GW> netcfg/get_nameservers=<FIRST-DNS>    
netcfg/confirm_static=true netcfg/get_hostname=debian                    

Replace <IPADDR>, <GW>, and <FIRST-DNS> to suit your local network config. You might also need to change the netmask, which I left in for clarity about the format. Adjust the device address of your OSA network card. If it's in layer 3 mode (very likely) you should set layer2=false. Note that mixed case matters, hence you will want to SET CASE MIXED in xedit.

Then there are the two URLs that need to be changed. The authorized_keys_url file contains your SSH public key and is fetched unencrypted and unauthenticated, so be careful what networks you traverse with your request (HTTPS is not supported by debian-installer in Debian).

preseed/url is needed for installation parameters that do not fit the parameters file - there is an upper character limit that's about two lines longer than my example. This is why this example only contains the bare minimum for the network part, everything else goes into this preseeding file. It file can optionally be protected with a MD5 checksum in preseed/url/checksum.

Both URLs need to be very short. I thought that there was a way to specify a line continuation, but in my tests I was unable to produce one. Hence it needs to fit on one line, including the key. You might want to use an IPv4 as the hostname.

To skip the initial boilerplate prompts and to skip straight to the user and disk setup you can use this as preseed.cfg:

d-i debian-installer/locale string en_US
d-i debian-installer/country string US
d-i debian-installer/language string en
d-i time/zone US/Eastern
d-i mirror/country manual
d-i mirror/http/mirror string httpredir.debian.org
d-i mirror/http/directory string /debian
d-i mirror/http/proxy string

I'm relatively certain that the DASD disk setup part cannot be automated yet. But the other bits of the installation should be preseedable just like on non-mainframe hardware.
Categories: FLOSS Project Planets

Steve Purkiss: Drupalaton 2015 - the memories

Planet Drupal - Sun, 2015-08-30 13:26
Sunday, 30th August 2015Drupalaton 2015 - the memories

"The advantages of variables in Drupal 7 is they're all loaded into memory. The disadvantages of variables in Drupal 7 is they're all loaded into memory".

With this, Károly Négyesi ('chx'), opened his Storing Data with Drupal 8 workshop at Drupalaton to a standing room only crowd - half whom chuckled knowingly, with the rest following shortly after once they'd had a little time to think about this technical juxtaposition.

It is this pull no punches attitude which made chx's workshop, for me at least, the highlight of the event - I like to know what is wrong, why it is wrong, and what we can or are doing to fix it. It was different when I'd asked the Hungarian, now living in Vancouver, to keynote our local DrupalCamp Brighton held back in January. In a departure from his normal techie talks, chx delivered a highly enjoyable and enlightening session talking about the profession of programming and how computer game music was the best type of music to listen to when coding. Here at Drupalaton though, we were back to the nitty and gritty of Drupal's internals.

Chx proceeded to give an excellent overview of the different subsystems of Drupal and how they'd changed from version 7 to 8. Drupal 8 brings together a lot of pieces which were fragmented in Drupal 7 - for example Drupal's Entity API where most of the functionality existed in a contributed module. By bringing it all together into core Drupal and providing APIs it improves maintainability through common knowledge, translation, access, performance and testing. He went on to cover the improved data stores in Drupal 8, for example the state API to store information such as when maintenance updates were last run using cron, a private tempstore for functionality like autosave data and the new quick edit functionality, and a shared tempstore for things like views.

After an intense hour and a half and a short break chx for a moment thought he'd lost half the audience but most soon filtered back in once their brains had a little more time for the coffee to take effect! The workshop continued with much information about how to use these new APIs, and how all configuration was now in YAML files then loaded into the database at runtime. In previous versions of Drupal, many settings were stored in the database and work-arounds such as using the Features module were used to extract settings. These approaches had varying results - as someone coming from an Enteprise Java world where storing settings in files is the norm I for one am very excited about this as it provides far more easier development, deployment and versioning. Features still exist in Drupal 8, but for their original purpose - to create exportable features of functionality.

Contributing to Drupal can be tiring!

After the workshop I made my way to the only air-conditioned room for what felt like miles around and managed to catch this picture of chx resting - with lots of talk recently about 'Headless Drupal' I thought the angle was quite funny.

I'd gone there to do some more work on Drupal 8 Rules - the fifth most popular module for Drupal which enables people to create functionality through the user interface, for example to send an email when someone adds a comment, or to apply a certain tax rate for a certain product in a certain country.

Although I'd previously worked on a few core issues for Drupal 8 like splitting up the 'password strength' and 'password matches' code and cleaning up some unused variables, I'd not really found an area I could focus on, but then I attended Drupalaton last year where there was a whole day of Rules, with a morning workshop bringing people up to speed on Rules in Drupal 8 and an afternoon spent focusing on how you could get involved and contribute.

For me, the workshop format is what makes Drupalaton so special - most other DrupalCamps have half-to-an-hour long sessions where you get an overview of something but don't really get time to delve in deep. I wanted to learn Drupal 8, and helping Rules out where I could to me seemed like a pretty good way of learning it. Since then I've helped update a few Rules Actions, Conditions, and Events to Drupal 8 and although I haven't done half as much as I've wanted to, it feels great to be learning lots and hopefully helping progress things a little bit.

Cruise Party

After spending the afternoon learning more about exporting, importing, and deploying configuration management in Fabian Bircher's workshop (slides and a great blog post from Fabian here), it was time for the yearly Drupalaton Cruise Party.

The cruise party is an excellent chance to chill out, see a little more of Central Europe's largest lake, talk Drupal, and enjoy a most wonderful sunset!

Behold Behat!

The next morning, after I'd managed to complete 30 lengths of the swimming pool and a 20km cycle for the second day in a row (yes, I know, shocker, but the Sun makes me a different person than I am in the dreary rainy grey UK!) it was workshop time again - this morning's being 'From User Story to Behat Test' with Pieter Frenssen.

I'm almost as excited about using Behat as I am about Drupal 8 - after many years of discussions about how functionality X worked, or whether feature X and function Y was included in the original quote, this way of defining requirements provides an excellent interface all project stakeholders can be a part of, with a solid technical backing enabling tests to be performed against sets of Plain English user stories.

The workshop was to go through setting up Behat with Drupal 8 which was great as I'd spent some time recently getting it set up but lacked in-depth understanding of the setup which Pieter's workshop helped enormously with so now I'll be using it on all my projects. There's plenty of info online about Behat so I won't go into more details here.

Grill Party

The afternoon was spent discussing using Drupal as a prototyping tool with Kristof Van Tomme László Csécsy and finding out about Pretotyping. This is of particular interest to me as I continue my work on abilit.es - a topic for another blog I think!

In the evening the lovely Drupalaton organising team arranged a Grill Party - this didn't happen last year but was great as was right next to the hotel and means we weren't dispersed across many different places.

As well as great food a few people had been working on some lyrics and provided lots of entertainment with a fab Drupalaton song!

So long Drupalaton, till next year! Sunday was fairly quiet, no sessions were on but the sprint room was open so I interspersed some rules work with some cycling. After a few days of the hotel's all-inclusive menu I wasn't expecting to lose any weight, but certainly enjoyed getting the metabolism up and running again nicely!   In 36 degrees cycling was not easy but certainly fun and I'm glad I took my Brompton on its first trip abroad - I'd been wanting an opportunity to test out the Vincita Sightseer bag and it did not disappoint. I look forward to taking my bike to more places around the world - you definitely get to see a lot more of a place than just by foot and with the bag going into normal hold it doesn't cost any extra than normal baggage.   Monday finally came round and it was time to make the journy back to the UK via Budapest. The train to Budapest is always an experience - it stops *everywhere* and takes three hours to go just over a hundred miles, has no air conditioning, and wouldn't let me on without buying a 'boading pass' even though I had a ticket already. Luckily it was not much and I had a small amount of local currency to cover it otherwise I would've been stuck!   Felt great to cycle out of the hotel down to the train station the pack the bike up in its own bag (it fits on the back rack when riding) and I can't wait till I can do it all again, perhaps DrupalCon Barcelona, although I'm not a huge fan of cycling in cities & DrupalCon is quite intense so perhaps not, we'll see!   A big thank you to all those who had something to do with this lovely event, from organising to speaking, sponsoring, and of course attending, am already looking forward to next year's holiday, erm - I mean, Drupalaton! tags: DrupalatonDrupal 8Drupal PlanetPlanet Drupal
Categories: FLOSS Project Planets

Jeff Knupp: Python Dictionaries

Planet Python - Sun, 2015-08-30 13:20

Aside: one thing I dislike about the official Python documentation is that only a small percentage of entries have example code. We should change that...)

One of the keys to becoming a better Python programmer is to have a solid grasp of Python's built-in data structures. Using the structured format below, today you'll learn what a dict is, when to use it, and see example code of all of its member functions. I have some other data structures in the works, so this may turn into a little series.

Dictionary AKA

"Associate Array", "Map", "Hash Map", "Unordered Map"




Contains a series of key -> value mappings where the "key" is of any type that is hashable (meaning it has both a __eq__() and a __hash__() method). The "value" may be of any type and value types need not be homogeneous.

That means, for example, we can have a dictionary where some keys map to strings and others to ints. Probably not a great idea in practice, but there's nothing stopping you from doing it.

What Makes it Special

The conceptual implementation is that of a hash table, so checks for existence are quite fast. That means we can determine if a specific key is present in the dictionary without needing to examine every element (which gets slower as the dictionary gets bigger). The Python interpreter can just go to the location key "should be" at (if it's in the dictionary) and see if key is actually there.

Construction Literal
  • {}: pair of braces for empty dictionary
  • {1:2, 3:4}: comma-separated list of the form key: value enclosed by braces

  • dict(one=2, three=4): using dict() with keyword arguments mapping keys to values (where one and two are valid identifiers)

  • dict([(1, 2), (3, 4)]): using dict() with an iterable containing iterables with exactly two objects, the key and value
  • dict(zip([1, 3], [2, 4])): using dict() with two iterables of equal length; the first contains a list of keys and the second contains their associated values.
  • dict({1:2, 3:4}): using dict() with the literal form as an argument. This is silly. Why would you want this?




When to Use It

When describing what you want to do, if you use the word "map" (or "match"), chances are good you need a dictionary. Use whenever a mapping from a key to a value is required.

Example Usage state_capitals={ 'New York': 'Albany', 'New Jersey': 'Trenton', }

"New York" is a key and "Albany" is a value. This allows us to retrieve a state's capital if we have the state's name by doing capital = state_capitals[state]

How Not to Use It

Remember, the great thing about dictionaries is we can find a value instantly, without needing to search through the whole dictionary manually, using the form value = my_dict['key'] or value = my_dict.get('key', None).

If you're searching for a value in a dictionary and you use a for loop, you're doing it wrong. Stop, go back, and read the previous statement.

All too often in beginner code I see the equivalent of the following (continuing the previous example):

state_im_looking_for = 'New Jersey' my_capital = '' for state in state_capitals: if state == state_im_looking_for: my_capital = state_capitals[state]

Or like this:

state_im_looking_for = 'New Jersey' my_capital = '' for state, capital in state_capitals.items(): if state == state_im_looking_for: my_capital = capital Methods and Uses d.clear()

Remove all entries in d






Delete all items in a dictionary

d.clear() d.copy()

Make a shallow copy of d. The dictionary returned by d.copy() will have the same references as d, not copies of the items.


A new dict, representing a shallow copy of d




Create copy of a dictionary

d = {1: 'a', 2: 'b', 3: 'c'} copied_dict = d.copy() copied_dict # {1: 'a', 2: 'b', 3: 'c'} d[1] = 'z' copied_dict # {1: 'a', 2: 'b', 3: 'c'} del k[d]

Used to remove a value from a dictionary




KeyError if key is not in dictionary


Delete entry with key 'hello'

my_dictionary = {'hello': 1, 'goodbye': 2} del my_dictionary['hello'] print(my_dictionary) # {'goodbye': 2} dict.fromkeys(seq[, value])

Create a new dictionary with the same keys as seq. If value is provided, each item's value is set to value. If value is not set, all item values are set to None






Create a dictionary from a list with all values initialized to 0

my_list = [1, 2, 3] my_dictionary = dict.fromkeys(my_list, 0) my_dictionary # {1: 0, 2: 0, 3: 0}

Create a dictionary from a dictionary with all values automatically initialized to None

my_dictionary = {1: 1, 2: 2, 3: 3} new_dictionary = dict.fromkeys(my_dictionary) my_dictionary # {1: None, 2: None, 3: None} d.get(key[, default)

Used to retrieve the value associated with key key. The value of default is returned if key is not in d (rather than raising a KeyError). The default value of default is None.


Roughly equivalent to:

def get(key, default=None): if key in d: return d[k] else: return default Raises



Get a key's value or None if the key isn't present

{1: 'a', 2: 'b'}.get(3) k in d

Used to iterate over the keys, values, or both of the dictionary.






Iterate over keys

for key in my_dictionary:

Iterate over (key, value) tuples

for key, value in my_dictionary.items():

Iterate over values

for value in my_dictionary.values():

Check for existence

haystack = {} # ... if 'needle' in haystack: iter(d)

Used to iterate over the keys of d


An iterator which iterates over the keys of d


StopIteration when d has no more keys


Iterate over keys

for key in my_dictionary: d[key]

Used to access the value corresponding to the key key in d.


Value associated with the key (heterogeneous)


KeyError when key is not a member of d.

Examples capitals = {'New York': 'Albany'}` capital_of_ny = capitals['New York']` print capital_of_ny` 'Albany' len(d)

Used to determine the number of entries in a dictionary


Length of dictionary d



Examples print 'dictionary has {} entries'.format(len(d)) k not in d

Used for negative existence check. Equivalent to not key in value


True if key is not in value, False otherwise




Check for negative existence

haystack = {} # ... if 'needle' not in haystack: d.keys()

Iterate over the keys in a dictionary


An iterable over all of the keys in d (in an unspecified order)


StopIteration when d has no more keys


Iterate over keys:

for key in d.keys(): d.values()

Iterate over the values in a dictionary


An iterable over all of the values in d (in an unspecified order)


StopIteration when d has no more values


Iterate over values:

for value in d.values(): d.items()

Iterate over the elements ((key, value) pairs) in a dictionary


An iterable over all of the (key, value) pairs in d (in an unspecified order). Each (key, value) pairs is represented as a tuple.


StopIteration when d has no more elements


Iterate over items:

for key, value in d.items():

Note that, in the example, we can use multiple assignment to assign key to the key and value to the value of each item directly in the for loop.

d.pop(key[, default])

Used to remove an item from a dictionary and return its associated value


d[key] if key is in d. If key is not in d but default is specified, the default value is returned instead.


KeyError if key is not in dictionary and no default is specified


Delete entry with key 'hello' and print its value

my_dictionary = {'hello': 1, 'goodbye': 2} hello_value = my_dictionary.pop('hello') print(hello_value) # 1 print(my_dictionary) # {'goodbye': 2}

With default specified

my_dictionary = {'hello': 1, 'goodbye': 2} foo_value = my_dictionary.pop('foo', None) print(foo_value) # None print(my_dictionary) # {'goodbye': 2}

With no default specified

my_dictionary = {'hello': 1, 'goodbye': 2} foo_value = my_dictionary.pop('foo') # KeyError: 'foo' d.popitem()

Pop (i.e. delete and return) a random element from the dictionary


A (key, value) tuple if d is not empty.


KeyError if d is empty. I personally think that's a stupid exception to raise since no key was ever specified, but, hey, I didn't write the language.


Destructively iterate over values:

try: key, value = d.popitem(): print 'Got {}: {}'.format(key, value) except KeyError: print 'Done' d.setdefault(key[, default])

Get a key from the dictionary or, if it's not there, insert it with a default value and return that. default, erm, defaults to None


d[key] if key is in d.

If not, do d[key] = default and then return d[key] (which will always return default).




Count the number of times each word is seen in a file:

words = {} for word in file: occurrences = words.setdefault(word, 0) words[word] = occurrences + 1 d.update(other)

Update a dictionary with the keys and values in other, overwriting existing keys and values if there is any overlap.






Merge two dictionaries:

first = {'a': 1} second = {'b': 2} first.update(second) print first # {'a': 1, 'b': 2} print second # {'b': 2}

Using keyword arguments for other:

first = {'a': 1} first.update(b=2, c=3) print first # {'a': 1, 'c': 3, 'b': 2}
Categories: FLOSS Project Planets
Syndicate content