FLOSS Project Planets

Import Python: ImportPython Issue 70

Planet Python - Thu, 2016-04-28 09:40

Worthy Read
Have an idea for a Python App ?
Create a new Python app, free with Azure App Service. Curator's Note - Select Python from the language dropdown and select Python. You have templates for django, flask, bottle. Check it out.Sponsor
Python Sorted Collectionscore pythonPython is a little unusual regarding sorted collection types as compared with other programming languages. Three of the top five programming languages in the TIOBE Index include sorted list, sorted dict or sorted set data types. But neither Python nor C include these. For a language heralded as “batteries included” that’s a little strange.

Best Practices working with Django models in PythondjangoA very handy and to the point list of checklist. If you are a Django developer do read them.

Optimizing Slow Django REST Framework PerformanceDjango Rest FrameworkThe Django REST Framework (DRF for short) allows Django developers to build simple yet robust standards-based REST API for their applications. Even seemingly simple, straightforward usage of the Django REST Framework and its nested serializers can kill performance of your API endpoints. At it’s root, the problem is called the “N+1 selects problem”; the database is queried once for data in a table (say, Customers), and then, one or more times per customer inside a loop to get, say, customer.country.Name. Using the Django ORM, this mistake is easy to make. Using DRF, it is hard not to make.

select — Wait for I/O Efficiently — PyMOTW 3core pythonThe select module provides access to platform-specific I/O monitoring functions. The most portable interface is the POSIX function select(), which is available on UNIX and Windows. The module also includes poll(), a UNIX-only API, and several options that only work with specific variants of UNIX.

Episode 54 - Pip and the Python Package Authority with Donald StufftpodcastAs Python developers we have all used pip to install the different libraries and projects that we need for our work, but have you ever wondered about who works on pip and how the package archive we all know and love is maintained? In this episode we interviewed Donald Stufft who is the primary maintainer of pip and the Python Package Index about how he got involved with the projects, what kind of work is involved, and what is on the roadmap. Give it a listen and then give him a big thank you for all of his hard work!

django-river state machine and workflow libraryMain goal of developing this framework is to be able to edit any workflow item on the fly. This means, all elements in workflow like states, transitions, user authorizations(permission), group authorization are editable. To do this, all data about the workflow item is persisted into DB. Hence, they can be changed without touching the code and re-deploying your application.

Build real-time apps.
Syncano. Database. Backend. Middleware. Real-time. Support. Start for free!Sponsor
Computational Geometry in Python: From Theory to ApplicationWhen people think computational geometry, in my experience, they typically think one of two things: Wow, that sounds complicated. Oh yeah, convex hull. In this post, I’d like to shed some light on computational geometry, starting with a brief overview of the subject before moving into some practical advice based on my own experiences

A polyglot's guide to multiple dispatch - part 3core pythonThis is part 3 in the series of articles on multiple dispatch. Part 1 introduced the problem and discussed the issues surrounding it, along with a couple of possible solutions in C++. Part 2 revisited multiple dispatch in Python, implementing a couple of variations. In this part, I'm going back to the roots of multiple dispatch - Common Lisp - one of the first mainstream programming languages to introduce multi-methods and an OOP system based on multiple dispatch at its core.

Your Django Story: Meet Margaret MyrickinterviewMargaret Myrick is a program manager at Indeed, and a musician on the side. She has lived in Texas since she was a child.

YodaBot - An open source messenger bot to convert text to yoda speakMaster Yoda, I am. Left Degobah I have, and come to messenger as a bot. Message me. Reply in my own style, I would. Curator Note - Pretty funny check it out.

Python Prompt Toolkitcore pythonprompt_toolkit is a library for building powerful interactive command lines and terminal applications in Python.

Employer Refuses to Allow Python - Reddit Discussioncore pythonHey guys, Recently, I informed my manager that I was willing to go above-and-beyond and help improve some of our team operations by writing a handful of Python scripts. He is a stickler for following the rules, so he ordered me to ask permission to install Python on my development machine (I wasn't intending on asking permission). My request has been refused. Despite providing evidence from the Python Foundation on their open licensing (especially for my purposes, which is just local machine -- not production), they are still refusing on account that Python is type of GPL and it is interpretative. Have you guys run into something like this before? It seems ridiculous to me.

James Powell - `from __past__ import print_statement`: a Dadaist Rejection of Python 2 vs 3videoIf the title doesn't make any sense, then there's no hope that the description will be any better. This talk will be a strange dive into interpreter hacks, the pointlessness of the Python 2 vs 3 debate, and the twisted artistic drive that pushes the speaker to come up with these perversions of the Python language. Prepared to be simultaneously repulsed, intrigued, and completely bored.

Analyze Chrome’s Browsing History with PythonChrome stores its data locally in an SQLite database. So all we need to do here is write a consistent Python code that would make a connection to the database, query the necessary fields and extract the required data, which is the URLs visited and the corresponding total visit counts, and churn it out like a puppy.

Learn to use Python to Collect Intelligence Online
Learn how to write code to automatically extract and analyze data from the web and social media. Join students from around the world from law enforcement, journalism, information security and more.Sponsor

New Books

Mastering Python ForensicsLearn to perform forensic analysis and investigations with the help of Python, and gain an advanced understanding of the various Python libraries and frameworks. Analyze Python scripts to extract metadata and investigate forensic artifacts. The writers, Dr. Michael Spreitzenbarth and Dr. Johann Uhrmann, have used their experience to craft this hands-on guide to using Python for forensic analysis and investigations

Manager (Technical architect+Engineering leader) at CRISIL LIMITEDMumbai

Python Developer II at Kforce Professional StaffingSan Antonio, TX, United States

Upcoming Conference / User Group Meet
PyData London 2016
PyCon Israel 2016
Pycon Sweden

gym - 635 Stars, 71 ForkA toolkit for developing and comparing reinforcement learning algorithms.
rllab - 86 Stars, 12 Forkrllab is a framework for developing and evaluating reinforcement learning algorithms.
otek - 45 Stars, 2 ForkAn unopinionated project builder for everyone.
LowRankPropagation - 22 Stars, 3 ForkPropagation Technique for Solving Low Rank Matrix Completion
detux - 20 Stars, 3 ForkThe Multiplatform Linux Sandbox
doorman - 15 Stars, 1 Forkan osquery fleet manager
ballade - 14 Stars, 1 ForkBallade is a light weight http proxy based on tornado and an upstream proxy switcher using SwitchyOmega rules
LearnProgrammingBot - 14 Stars, 1 ForkBot for /r/learnprogramming using supervised learning
falcon-api - 11 Stars, 0 ForkWeb APIs for Falcon.
flatdoc - 4 Stars, 0 ForkFlat documentation generator
slactorbot - 4 Stars, 0 ForkA Slack bot with hot patch!
elastic-bill - 3 Stars, 1 ForkElastic bill is a multi cloud platform billing management tool.
Categories: FLOSS Project Planets

Premier livre sur Krita en français

Planet KDE - Thu, 2016-04-28 09:23

(Post in french, english version below)

Le mois dernier est sorti mon livre “Dessin et Peinture numérique avec Krita”. Il s’agit du premier livre en francais sur ce logiciel. J’espère qu’il contribuera à faire connaitre ce magnifique logiciel libre de dessin à tous les artistes francophones.

Ce livre est disponible en version imprimée couleur, en version numérique téléchargeable sans DRM ou encore en version consultable en ligne, sur le site de l’éditeur D-Booker. Je remercie d’ailleurs mon éditeur pour m’avoir permis d’écrire ce livre.

Last month, my book “Dessin et Peinture numérique avec Krita” has been released. It is the first book in french about this software. I hope it will contribute to introduce this wonderful Free Software to all french speaking artists.

This book is available in full-color printed version, as digital download without DRM or as online version, on the website of the publisher D-Booker. By the way, I’d like to thank my publisher who made it possible to write this book.

Categories: FLOSS Project Planets

Drop Guard: Probo.CI and Drop Guard work together for better QA process

Planet Drupal - Thu, 2016-04-28 06:23
Probo.CI and Drop Guard work together for better QA process Igor Kandyba Thu, 28.04.2016 - 12:23

Not long ago we were talking about the value of testing your updates in feature branch instances. It's the most efficient way of ensuring the quality of applied updates, but it's very time-consuming.

To use this process, you are required to maintain your own infrastructure to spin up QA servers quickly, run automated tests and share the testing instance between team members. And preferably, you do it every time an update is applied for any of the modules across your websites.

Probo.CI integration QA Drop Guard recipes Drupal Planet
Categories: FLOSS Project Planets

Guido van Rossum: King's Day Speech

Planet Python - Thu, 2016-04-28 03:39
Today the Netherlands celebrates King's Day. To honor this tradition, the Dutch embassy in San Francisco invited me to give a "TED talk" to an audience of Dutch and American entrepreneurs. Here's the text I read to them. Part of it is the tl;dr of my autobiography; part of it is about the significance of programming languages; part of it is about Python's big idea. Leve de koning! (Long live the king!)
Python: a programming language created by a communityExcuse my ramblings. I’ll get to a point eventually.
Let me introduce myself. I’m a nerd, a geek. I’m probably somewhere on the autism spectrum. I‘m also a late bloomer. I graduated from college when I was 26. I was 45 when I got married. I’m now 60 years old, with a 14 year old son. Maybe I just have a hard time with decisions: I’ve lived in the US for over 20 years and I am still a permanent resident.
I'm no Steve Jobs or Mark Zuckerberg. But at age 35 I created a programming language that got a bit of a following. What happened next was pretty amazing. But I'll get to that.
At age 10 my parents gave me an educational electronics kit. The kit was made by Philips, and it was amazing. At first I just followed the directions and everything worked; later I figured out how to design my own circuits. My prized possessions were the kit's three (!) transistors.
I took one of my first electronics models, a blinking light, to show and tell in 5th grade. It was a total dud — nobody cared or understood its importance. I think that's one of my earliest memories of finding myself a geek: until then I had just been a quiet quick learner.
In high school I developed my nerdiness further — I hung out with a few other kids interested in electronics, and during physics class we sat in the back of the class discussing NAND gates while the rest of the class was still figuring out Ohm's law.
Fortunately our physics teacher had figured us out: he employed us to build a digital timer that he used to demonstrate the law of gravity to the rest of the class. It was a great project and showed us that our skills were useful. The other kids still thought we were weird: it was the seventies and many were into smoking pot and rebelling; another group was already preparing for successful careers as doctors or lawyers or tech managers. But they left me alone, I left them alone, and I graduated as one of the best of my year.
After high school I went to the University of Amsterdam: It was close to home, and to a teen growing up in the Netherlands in the seventies, Amsterdam was the only cool city. (Yes, the student protests of 1968 did touch me a bit.) Much to my high school physics teacher's surprise and disappointment, I chose to major in math, not physics. But looking back I think it didn’t matter.
In the basement of the science building was a mainframe computer, and it was love at first sight. Card punches! Line printers! Batch jobs! More to the point, I quickly learned to program, in languages with names like Algol, Fortran and Pascal. Mostly forgotten names, but highly influential at the time. Soon I was, again, sitting in the back of class, ignoring the lecture, correcting my computer programs. And why was that?
In that basement, around the mainframe, something amazing was happening. There was a loosely-knit group of students and staff with similar interests, and we exchanged tricks of the trade. We shared subroutines and programs. We united in our alliances against the mainframe staff, especially in the endless cat-and-mouse games over disk space. (Disk space was precious in a way you cannot understand today.)
But the most important lesson I learned was about sharing: while most of the programming tricks I learned there died with the mainframe era, the idea that software needs to be shared is stronger than ever. Today we call it open source, and it’s a movement. Hold that thought!
At the time, my immediate knowledge of the tricks and the trade seemed to matter most though. The mainframe’s operating system group employed a few part-time students, and when they posted a vacancy, I applied, and got the job. It was a life-changing event! Suddenly I had unlimited access to the mainframe — no more fighting for space or terminals — plus access to the source code for its operating system, and dozens of colleagues who showed me how all that stuff worked.
I now had my dream job, programming all day, with real customers: other programmers, the users of the mainframe. I stalled my studies and essentially dropped out of college, and I would not have graduated if not for my enlightened manager and a professor who hadn't given up on me. They nudged me towards finishing some classes and pulled some strings, and eventually, with much delay, I did graduate. Yay!
I immediately landed a new dream job that would not have been open to me without that degree. I had never lost my interest in programming languages as an object of study, and I joined a team building a new programming language — not something you see every day. The designers hoped their language would take over the world, replacing Basic.
It was the eighties now, and Basic was the language of choice for a new generation of amateur programmers, coding on microcomputers like the Apple II and the Commodore 64. Our team considered the Basic language a pest that the world should be rid of. The language we were building, ABC, would "stamp out Basic", according to our motto.
Sadly, for a variety of reasons, our marketing (or perhaps our timing) sucked, and after four years, ABC was abandoned. Since then I've spent many hours trying to understand why the project failed, despite its heart being so clearly in the right place. Apart from being somewhat over-engineered, my best answer is that ABC died because there was no internet in those days, and as a result there could not be a healthy feedback loop between the makers of the language and its users. ABC’s design was essentially a one-way street.
Just half a decade later, when I was picking through ABC’s ashes looking for ideas for my own language, that missing feedback loop was one of the things I decided to improve upon. “Release early, release often” became my motto (freely after the old Chicago Democrats’ encouragement, “vote early, vote often”). And the internet, small and slow as it was in 1990, made it possible.
Looking back 25 years, the Internet and the Open Source movement (a.k.a. Free Software) really did change everything. Plus something called Moore's Law, which makes computers faster every year. Together, these have entirely changed the interaction between the makers and users of computer software. It is my belief that these developments (and how I managed to make good use of them) have contributed more to the success of “my” programming language than my programming skills and experience, no matter how awesome.
It also didn't hurt that I named my language Python. This was a bit of unwitting marketing genius on my part. I meant to honor the irreverent comedic genius of Monty Python's Flying Circus, and back in 1990 I didn't think I had much to lose. Nowadays, I'm sure "brand research" firms would be happy to to charge you a very large fee to tell you exactly what complex of associations this name tickles in the subconscious of the typical customer. But I was just being flippant.
I have promised the ambassador not to bore you with a technical discussion of the merits of different programming languages. But I would like to say a few things about what programming languages mean to the people who use them — programmers. Typically when you ask a programmer to explain to a lay person what a programming language is, they will say that it is how you tell a computer what to do. But if that was all, why would they be so passionate about programming languages when they talk among themselves?
In reality, programming languages are how programmers express and communicate ideas — and the audience for those ideas is other programmers, not computers. The reason: the computer can take care of itself, but programmers are always working with other programmers, and poorly communicated ideas can cause expensive flops. In fact, ideas expressed in a programming language also often reach the end users of the program — people who will never read or even know about the program, but who nevertheless are affected by it.
Think of the incredible success of companies like Google or Facebook. At the core of these are ideas — ideas about what computers can do for people. To be effective, an idea must be expressed as a computer program, using a programming language. The language that is best to express an idea will give the team using that language a key advantage, because it gives the team members — people! — clarity about that idea. The ideas underlying Google and Facebook couldn't be more different, and indeed these companies' favorite programming languages are at opposite ends of the spectrum of programming language design. And that’s exactly my point.
True story: The first version of Google was written in Python. The reason: Python was the right language to express the original ideas that Larry Page and Sergey Brin had about how to index the web and organize search results. And they could run their ideas on a computer, too!
So, in 1990, long before Google and Facebook, I made my own programming language, and named it Python. But what is the idea of Python? Why is it so successful? How does Python distinguish itself from other programming languages? (Why are you all staring at me like that? :-)
I have many answers, some quite technical, some from my specific skills and experience at the time, some just about being in the right place at the right time. But I believe the most important idea is that Python is developed on the Internet, entirely in the open, by a community of volunteers (but not amateurs!) who feel passion and ownership.
And that is what that group of geeks in the basement of the science building was all about.
Surprise: Like any good inspirational speech, the point of this talk is about happiness!
I am happiest when I feel that I'm part of such a community. I’m lucky that I can feel it in my day job too. (I'm a principal engineer at Dropbox.) If I can't feel it, I don't feel alive. And so it is for the other community members. The feeling is contagious, and there are members of our community all over the world.
The Python user community is formed of millions of people who consciously use Python, and love using it. There are active members organizing Python conferences — affectionately known as PyCons — in faraway places like Namibia, Iran, Iraq, even Ohio!
My favorite story: A year ago I spent 20 minutes on a video conference call with a classroom full of faculty and staff at Babylon University in southern Iraq, answering questions about Python. Thanks to the efforts of the audacious woman who organized this event in a war-ridden country, students at Babylon University are now being taught introductory programming classes using Python. I still tear up when I think about the power of that experience. In my wildest dreams I never expected I’d touch lives so far away and so different from my own.
And on that note I'd like to leave you: a programming language created by a community fosters happiness in its users around the world. Next year I may go to PyCon Cuba!
Categories: FLOSS Project Planets

Jason Meyers: Parallelizing Queries with SQLAlchemy, Gevent, and PostgreSQL

Planet Python - Thu, 2016-04-28 01:00

Recently, we had a need to execute multiple SQLAlchemy queries in a parallel fashion against a PostgreSQL database with Python 2 and psycopg2. We didn’t really need a full scale multithreading approach, so we turned to gevent. Gevent is a implementation of green threading that uses libevent. This post really isn’t about what threading model is better, as that is great subjective and changes based on your use case and needs.

Making Psycopg2 Coroutine Friendly

So the first thing we need to do to prepare to leverage gevent is make sure psycopg2 is configured properly. Because the main psycopg2 is a C extension, we can not just take advantage of gevent’s monkey patching. Nonetheless, psycopg2 exposes a hook that can be use with coroutine libraries to integrate with the event scheduler. There are a few libraries to help with this such as sqlalchemy_gevent or psycogreen that you can use to automatically set this part up for you. However, the code is fairly short and straight forward. Let’s look at the code from psycogreen, that I use in my applications.

import psycopg2 from psycopg2 import extensions from gevent.socket import wait_read, wait_write def make_psycopg_green(): """Configure Psycopg to be used with gevent in non-blocking way.""" if not hasattr(extensions, 'set_wait_callback'): raise ImportError( "support for coroutines not available in this Psycopg version (%s)" % psycopg2.__version__) extensions.set_wait_callback(gevent_wait_callback) def gevent_wait_callback(conn, timeout=None): """A wait callback useful to allow gevent to work with Psycopg.""" while True: state = conn.poll() if state == extensions.POLL_OK: break elif state == extensions.POLL_READ: wait_read(conn.fileno(), timeout=timeout) elif state == extensions.POLL_WRITE: wait_write(conn.fileno(), timeout=timeout) else: raise psycopg2.OperationalError( "Bad result from poll: %r" % state)

The code above will block the calling thread with a callback if we are busy reading or writing, which will return control back to the event loop that could start working on another thread if one needs to be worked on. The wait_read function blocks the current green let until the connection is ready to read from, and the wait_write does the same thing for writing to the connection.

With psycopg2 ready to work with gevent, we can start writing the code to execute our queries. I’m going to use a few different elements from gevent to control how the queries are executed, how many queries can be run at once, and how the query results are handled. There are many ways to use gevent and workers to accomplish this tasks, and some are quite simpler. My structure is set for future options and growth. I call this structure a QueryPool.

Building our Gevent based QueryPool

The QueryPool consists of an input queue to hold the queries we want to run, an output queue to hold the results, an overseer to load up the input queue, workers to actually get the queries and run them, and finally a function to drive the process.

The central element of our QueryPool, is the input queue. It is a special kind of FIFO queue called a JoinableQueue, which has a .join() method that blocks until all items in the queue have been acknowledged as processed with a task_done(). The input queue is loaded by an overseer as the first step of running the QueryPool. Then workers grab tasks one at a time from the input queue and when they finish their work they use the task_done() method to inform the queue that the item has been processed.

Let’s look at the __init__() for our Query pool:

import gevent from gevent.queue import JoinableQueue, Queue class QueryPool(object): def __init__(self, queries, pool_size=5): self.queries = queries self.POOL_MAX = pool_size self.tasks = JoinableQueue() self.output_queue = Queue()

So in our __init__() method, we start by storing the queries. Next we set a POOL_MAX which is maximum number of workers we are going to use. This controls how many queries can be executed in parallel in our QueryPool. Then we setup the tasks queue, which is where are workers will pick up work from. Finally we setup the output queue that workers will publish the query results.

Defining worker methods

Now we can look at the workers, I use two methods in the QueryPool calls. The first one contains the logic to prepare a database connection and execute the SQLAlchemy query.

def __query(self, query): conn = engine.connect() results = conn.execute(query).fetchall() return results

For the workers method within the QueryPool class, we start a loop that is going to run until their are not tasks left in the tasks (input) queue. It will get one task, call the __query() method with the details from the task, handle any exceptions, add the results to the output queue, and then mark the task as done. When adding the results to the output queue, we use the put method of gevent queues to add the results without blocking. This allows the event loop to look for the next thing to work on.

def executor(self, number): while not self.tasks.empty(): query = self.tasks.get() try: results = self.__query(query) self.output_queue.put(results) except Exception as exc_info: print exc_info print 'Query failed :(' self.tasks.task_done() Building an overseer method

Now that we have a something to handle the work, we need to load the task (input) queue. I use an overseer method that iterates over the supplied recipes and puts them on the task (input) queue.

def overseer(self): for query in self.queries: self.tasks.put(query) Building the run method

Finally, we are ready to build the run method that ties everything together and makes it work. Let’s look at the code first and then cover it step by step.

def run(self): self.running = [] gevent.spawn(self.overseer).join() for i in range(self.POOL_MAX): runner = gevent.spawn(self.executor, i) runner.start() self.running.append(runner) self.tasks.join() for runner in self.running: runner.kill()

The runner first spawn the overseer and immediately calls join on it, which blocks until the overseer is done loading items into the tasks (input) queue. This is overkill, but provides more options and would allow up to load asynchronously in the future. Next, we start a number of workers up to the POOL_MAX we defined earlier, and add them to a list so we can control them later. When we start them, they immediately begin working. Then we call .join() on the tasks (input) queue which will block execution until all the queries in the queue has been execute and acknowledge as completed. Finally, we clean up by using the .kill() method on all of the workers.

Using the QueryPool

Now we are ready to use our QueryPool class! I’ve created an Jupyter notebook available on GitHub with some example queries against a benchmarking database. I created my test database using the pgbench command to create a 5Gb database.

Execution Results

I setup 6 queries of varying complexity and execution time. I then executed those queries in a traditional serial execution style and then with our QueryPool using different worker counts.

  • Serial: 24 seconds
  • QueryPool - 5 workers: 8 seconds
  • QueryPool - 3 workers: 9 seconds
  • QueryPool - 2 workers: 15 seconds
  • 2016-04-28 Added missing return thanks to Mike B.
  • 2016-04-29 Updated information about .put() method based on feedback from David S.
  • 2016-04-30 Updated while 1 to while True based on feedback from /u/Asdayasman

Parallelizing Queries with SQLAlchemy, Gevent, and PostgreSQL was originally published by Jason Myers at from n00b import python on April 28, 2016.

Categories: FLOSS Project Planets

Dan Crosta: Delete Your Dead Code!

Planet Python - Thu, 2016-04-28 00:00

A few days ago, Ned Batchelder's post on deleting code made the rounds on HN, even though it was originally written in 2002. Here I want to echo a few of Ned's points, and take a stronger stance than he did: delete code as soon as you know you don't need it any more, no questions asked. I'll also offer some tips from the trenches for how to identify candidate dead code.

This is the first in a series on eating your vegetables in software engineering, on good, healthy practices for a happy and successful codebase. I don't (yet) know how long the series will be, so please stay tuned!

What Is Dead May Never Die

This heading isn't just an oh-so-clever and timely pop culture reference. Dead code, that is, code that can't possibly be executed by your program, is a real hindrance to the maintainability of your codebase. How many times have you gone to add what seemed like a simple feature or improvement, only to be stymied by the complexity of the code you have to work around and within? How much nicer would your life be if the practice of adding a feature or fixing a bug was as easy as you actually thought it would be during sprint planning?

Each time you want to make a change, you must consider how it interacts with each of the existing features, quirks, known bugs, and limitations of all the code that surrounds it. By having less code surrounding the feature you want to add, there's less to consider and less that can go wrong. Dead code is especially pernicious, because it looks like you need to consider interactions with it, but, since it's dead, it's merely a distraction. It can't possibly benefit you since it's never called.

The fact that dead code might never actually die is an existential threat to your ability to work with a given codebase. In the limit, if code that isn't called is never culled, the size of your application will grow forever. Before you know it, what might only be a few thousand lines of actual functionality is surrounded by orders of magnitude more code that, by definition, does nothing of value.

It's Got to Go

Ned (Batchelder, not Stark) was a little more nuanced and diplomatic than I'm going to be here:

Let's say you have a great class, and it has many methods. One day you discover that you no longer are calling a particular method. Do you leave it in or take it out?

There's no single answer to the question, because it depends on the class and the method. The answer depends on whether you think the method might be called again in the future.

From http://nedbatchelder.com/text/deleting-code.html.

I say: scorch the earth and leave no code alive. The best code is code that you don't even have.

For those less audacious than I, remember that version control has your back in case you ever need that code again.

That said, I've never experienced a need to add something back that I have previously deleted, at least not in the literal sense of adding back in, line for line, verbatim, a section of code I'd previously deleted.

Of course I'm not talking about things like reverting wrong-headed commits here -- we're all human, and I make as many mistakes as the next person. What I mean is, I've never deleted a feature, shipped to production, then come back weeks, or months later and thought to myself, "boy howdy, that code I wrote a year or more ago was probably pretty good, so let's put it back now." Codebases live and evolve with time, so the old code probably doesn't fit with the new ideas, techniques, frameworks, and styles in use today. I might refer back to the old version for a refresher, especially if it's particularly subtle, but I've never brought code back in, wholesale.

So, do yourself -- and your team -- a favor, and delete dead code as soon as you notice it.

How Did We Get Here?

Ned's post goes into great detail on how and why dead code happens -- perhaps the person making a change didn't think the code would be gone forever, and so commented it out or conditionally compiled it. Perhaps the person making a change didn't know enough to know that the code was actually dead (about which more later).

I'll add another hypothesis to the list: we might all just be lazy. It's definitely easier not to do something (i.e. to leave the code as-is) than to do something (delete it).

Laziness is, after all, one of the three great virtues of a programmer. But the Laziness that Larry Wall was talking about isn't this kind, but another kind: "The quality that makes you go to great effort to reduce overall energy expenditure." Viewed this way, deleting dead code is an act of capital-L Laziness -- doing something that's easy now to prevent yourself from having to do something hard later. We could all stand to develop more of this kind of Laziness, what I like to think of as "disciplined laziness," in our day-to-day habits.

How Do We Get Out Of Here?

I spend most of my time programming in Python, where, unfortunately, IDEs can't usually correctly analyze a complete codebase and identify never-called code automatically. But, with a combination of discipline and some run-time tooling, we can attack this problem from two sides.

For simple cases, a better sense of situational awareness can help identify and remove dead code while you're making changes. Imagine you're working on a particular function, and you notice that a branch of an if/else phrase can't be executed based on the valid values of the surrounding code. I call this "dead code in the small," and this is quite easy to reason about and remove, but it does require a bit more effort than one might ordinarily expend.

Until you develop the habit of noticing this during the course of your ordinary programming routine, you can add a step to your pre-commit checklist: review the code around your changes for any now-dead code. This could happen just before you submit the code to your co-workers for review (you do do code review, right?) so that they don't have to repeat that process while reading through your changes.

Another kind of dead code happens when you remove the last usage of a class or function from within the code you're changing, without realizing that it's the last place that uses it. This is "dead code in the large," and is harder to discover in the course of ordinary programming unless you're lucky enough to have eidetic memory or know the codebase like the back of your hand.

This is where run-time tooling can help us out. At Magnetic, we're using Ned's (yes, the same Ned) coverage.py package to help inform our decisions about dead code. Ordinarily coverage is used during testing to ensure that your test cases appropriately exercise the code under test, but we also use it within our code "running as normal" to get insight into what is or isn't used:

import coverage cov = coverage.Coverage( data_file="/path/to/my_program.coverage", auto_data=True, cover_pylib=False, branch=True, source=["/path/to/my/program"], ) cov.start() # ... do some stuff ... cov.stop() cov.save()

This sets up a Coverage object with a few options that make the report more usable. First, we tell coverage where to save its data (we'll use that later to produce a nice HTML report of what is and isn't used), and ask it to automatically load and append to that file with auto_data=True. Next we ask it not to bother calculating coverage over the standard library or in installed packages -- that's not our code, so we'd expect that a lot of what's in there might not be used by us. It's not dead code that we need to maintain, so we can safely ignore it. We ask it to compute branch coverage (whether the true and false conditions of each if statement are hit). And finally, we point it out our sources, so that it can link its knowledge of what is or isn't called back to the source code for report computation.

After our program runs, we can compute the HTML coverage report like:

$ COVERAGE_FILE=/path/to/my_program.coverage coverage html -d /path/to/output

Which generates a report like:

(A complete example coverage HTML coverage report is available as part of the coverage.py docs.)

The lines highlighted in red are lines that were never hit during the recorded execution of the program -- these are candidate lines (and, by extension, methods) for dead code elimination.

I'll leave you with three warnings about using this approach to finding and removing dead code:

  1. Be careful when reviewing the results of a coverage run -- the fact that a line or function wasn't executed during a single run of the program doesn't mean they're necessarily dead or unreachable in general. You must still check the codebase to determine whether they're completely dead in your application.
  2. Computing coverage means your program needs to do more work, so it will become slower when run in this mode. I wouldn't recommend running this all the time in production, but in a staging environment or in targeted scenarios you'll probably be OK. As always, if performance is an important concern, you should definitely measure what impact coverage has before you run it.
  3. Finally, dont trust code coverage reports from testing runs to find dead code. Some code might be dead, save for the tests that excercise it; and some code might be alive, but untested!
Parting Thoughts

To you, dear reader, I must apologize. I left out an important part of Ned's blog post when I quoted him earlier. He says:

There's no single answer to the question, because it depends on the class and the method. [...] A coarse answer could be: if the class is part of the framework, then leave it, if it is part of the application, then remove it.

If you're an author of a library or framework, rather than an application, then the question of dead code becomes in some ways harder and in other ways easier. In essence, you can't ever remove a part of your public API (except during major version bumps). Essentially, all of your public API is live code, even if you, yourself, don't use it. But behind the public interface, dead code can still happen, and it should still be removed.

Delete your dead code!

Categories: FLOSS Project Planets

Jeff Geerling's Blog: Migrate a custom JSON feed in Drupal 8 with Migrate Source JSON

Planet Drupal - Wed, 2016-04-27 22:21

Recently I needed to migrate a small set of content into a Drupal 8 site from a JSON feed, and since documentation for this particular scenario is slightly thin, I decided I'd post the entire process here.

I was given a JSON feed available over the public URL http://www.example.com/api/products.json which looked something like:

Categories: FLOSS Project Planets

Dirk Eddelbuettel: RcppRedis 0.1.7

Planet Debian - Wed, 2016-04-27 22:16

A new release of RcppRedis arrived on CRAN today. And just like for the previous release, Russell Pierce contributed a lot of changes via several pull requests which make for more robust operations. In addition, we have started to add support for MessagePack by relying on our recently-announced RcppMsgPack package.

Changes in version 0.1.7 (2016-04-27)
  • Added support for timeout constructor argument (PR #14 by Russell Pierce)

  • Added new commands exists, ltrim, expire and pexpire along with unit tests (PR #16 by Russell Pierce)

  • Return NULL for empty keys in serialized get for consistency with lpop and rpop (also PR #16 by Russell Pierce)

  • Minor corrections to get code and hget and hset documentation (also PR #16 by Russell Pierce)

  • Error conditions are now properly forwarded as R errors (PR #22 by Russell Pierce)

  • Results from Redis commands are now checked for NULL (PR #23 by Russell Pierce)

  • MessagePack encoding can now be used which requires MessagePackage headers of version 1.0 or later; the (optional) RcppMsgPack package can be used.

Courtesy of CRANberries, there is also a diffstat report for the most recent release. More information is on the RcppRedis page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Lullabot: A Framework for Project Kickoffs

Planet Drupal - Wed, 2016-04-27 19:05

Project kickoffs can be the shortest individual component of a project, but they can also be the most important. Done poorly, a kickoff can feel like a reading of a contract by inhuman actors doing inhuman work. Done well, a kickoff can bring a team together and push them towards success. Kickoffs are one of the project skills we don’t get many opportunities to iterate and learn. Developers at Lullabot commonly end up attached to a client or project for many months (or years!) at a time, so it’s entirely possible to go that period of time without having a formal kickoff. Here are some thoughts I’ve had after doing several kickoffs this year.

About the Client

In a distributed team, a kickoff usually happens with a phone call. While pre-sales communication will have already happened, the kickoff call is usually the first time when everyone working on a team will be together at once. As a team member from the vendor, this is your chance to ask questions of the business stakeholders who might not be available day to day. I like to find out:

  • Why are we all here? Are the business, technology, or creative concerns the primary driver?
  • What is the business looking for their team to learn and accomplish?
  • What are the external constraints on the project? Are there timelines and due dates, or other projects dependent on our work? What are the upcoming decisions and turning points in the business that could have a big affect on the project?
About Me

We all have ideas about how we want to work and be utilized on a project. Making sure they align with the client is very important to work out during a kickoff. Sometimes, a client has specific priorities of work to get done. Other times, they might not have realized you have skills in a specific subject area that they really need. It’s really important to understand your role on a project, especially if you have multiple skill sets. Perhaps you’re a great Drupal site builder, but what the client really needs is to use your skills to organize and clean up their content model. Figuring all of that out is a great kickoff topic.

About Us

Once we understand each other, then we can start to figure out how we work together. It’s kind of like moving in with someone. You might know each other very well, but how are you going to handle talking with your landlord? How are each person’s work schedules going to integrate?

For a distributed team, communication tools are at the core of this discussion. We all have email, chat rooms, instant messaging, video, and more. What tools are best used when? Are there specific tools the client prefers, or tools that they can’t use because of their company’s network setup? Finding the middle ground between “all mediums, all the time” and “it’s all in person until you ask” is key.

Recurring meetings are another good topic to cover. Some companies will take new team members, add them to every recurring meeting, and use up a 10 hour-per-week consulting engagement with nothing but agile ceremony. Perhaps that’s what you’re needed for—or perhaps they’ve just operated out of habit. Finding a good balance will go a long way towards building a sustainable relationship.

Sharing each person’s timezones and availability also helps to keep expectations reasonable. Some companies have recurring meetings (like Lullabot’s Monday / Friday Team Calls) which will always be booked. Sometimes individuals have days their hours are different due to personal or family commitments. Identify the stakeholders who have the “worst” availability and give them extra flexibility in scheduling. Knowing all of this ahead of time will help prevent lots of back-and-forth over meeting times.

Finally, find out who you should go to if work is blocked. That might be a stakeholder or project manager on the client’s side, but it could also be one of your coworkers. Having someone identified to the team as the “unblocker of work” helps keep the project running smoothly and personal tensions low.

About Tech

For development projects, the first question I ask is “will we need any sort of VPN access?”. VPN access is almost always a pain to get set up—many companies aren’t able to smoothly setup contractors who are entirely remote. It’s not unheard of for VPN access to take days or weeks to set up. If critical resources are behind a VPN, it’s a good idea to start setting that up before an official kickoff.

Barring the VPN-monster, figuring out where code repositories are, where tickets are managed, and how development and QA servers work are all good kickoff topics. Get your accounts created and make sure they all work. If a client is missing anything (like a good QA environment or ticket system), this is when you can make some recommendations.

About Onsites

Some projects will have a kickoff colocated somewhere, either at a client’s office or at a location central to everyone. In distributed teams, an in-person meeting can be incredibly useful in understanding each person. The subtle, dry humour of your video expert becomes apparent in-person, but could have been misunderstood online. Most of the above can be handled in the first hour of an onsite visit, leaving much more time to fill given the travel time!

We like to focus onsites on the topics that are significant unknowns, require a significant number of people across many teams, and are likely to require whiteboards, diagrams, and group brainstorming. Project discoveries are a classic fit; it’s common to meet with many different people from different departments, and doing first meetings in person can be a significant time saver. The goal of an onsite shouldn’t be to “kick off” the project—it should be to build the shared understanding a team needs so they can be effective.

But what about sales engineering?

I’m sure some readers are now thinking “Wait a minute! Aren’t these all things you should know before a contract is signed?”. It’s true! Going into a kickoff without any of this information would be a serious risk.

It’s important to remember that the team on a kickoff isn’t going to be identical to the team who did the sales engineering work. Both the client and the vendor will have new people just getting started. As well, it’s useful to hear the project parameters one more time. Discrepancies in the discussions can alert the team to any misunderstandings, or more likely changes in the business environment running up to the signing of the contract. Especially on projects where a team is already working, hearing about progress or changes made in the week between signing an SOW and kickoff can be invaluable.

What did you learn the last time you helped to kick off a project? Let us know in the comments!

Categories: FLOSS Project Planets

Mike Hommey: Announcing git-cinnabar 0.3.2

Planet Debian - Wed, 2016-04-27 18:42

Git-cinnabar is a git remote helper to interact with mercurial repositories. It allows to clone, pull and push from/to mercurial remote repositories, using git.

Get it on github.

These release notes are also available on the git-cinnabar wiki.

This is mostly a bug and regression-fixing release.

What’s new since 0.3.1?
  • Fixed a performance regression when cloning big repositories on OSX.
  • git configuration items with line breaks are now supported.
  • Fixed a number of issues with corner cases in mercurial data (such as, but not limited to nodes with no first parent, malformed .hgtags, etc.)
  • Fixed a stack overflow, a buffer overflow and a use-after free in cinnabar-helper.
  • Better work with git worktrees, or when called from subdirectories.
  • Updated git to 2.7.4 for cinnabar-helper.
  • Properly remove all refs meant to be removed when using git version lower than 2.1.
Categories: FLOSS Project Planets

Stig Sandbeck Mathisen: Using LLDP on Linux. What's on the other side?

Planet Debian - Wed, 2016-04-27 18:00

On any given server, or workstation, knowing what is at the other end of the network cable is often very useful.

There’s a protocol for that: LLDP. This is a link layer protocol, so it is not routed. Each end transmits information about itself periodically.

You can typically see the type of equipment, the server or switch name, and the network port name of the other end, although there are lots of other bits of information available, too.

This is often used between switches and routers in a server centre, but it is useful to enable on server hardware as well.

There are a few different packages available. I’ve looked at a few of them available for the RedHat OS family (Red Hat Enterprise Linux, CentOS, …) as well as the Debian OS family (Debian, Ubuntu, …)

(Updated 2016-04-29, added more recent information about lldpd, and gathered the switch output at the end.)


A simple daemon, with no configuration needed. This runs as a privilege-separated daemon, and has a command line control utility. You invoke it with a list of interfaces as command line arguments to restrict the interfaces it should use.

“ladvd” is not available on RedHat, but is available on Debian.

Install the “ladvd” package, and run “ladvdc” to query the daemon for information.

root@turbotape:~# ladvdc Capability Codes: r - Repeater, B - Bridge, H - Host, R - Router, S - Switch, W - WLAN Access Point, C - DOCSIS Device, T - Telephone, O - Other Device ID Local Intf Proto Hold-time Capability Port ID office1-switch23 eno1 LLDP 98 B 42

Even better, it has output that can be parsed for scripting:

root@turbotape:~# ladvdc -b eno1 INTERFACE_0='eno1' HOSTNAME_0='office1-switch23' PORTNAME_0='42' PORTDESCR_0='42' PROTOCOL_0='LLDP' ADDR_INET4_0='' ADDR_INET6_0='' ADDR_802_0='00:11:22:33:44:55 VLAN_ID_0='' CAPABILITIES_0='B' TTL_0='120' HOLDTIME_0='103'

…my new favourite :)


Another package is “lldpd”, which is also simple to configure and use.

“lldpd” is not available on RedHat, but it is present on Debian.

It features a command line interface, “lldpcli”, which can show output with different level of detail, and on different formats, as well as configure the running daemon.

root@turbotape:~# lldpcli show neighbors ------------------------------------------------------------------------------- LLDP neighbors: ------------------------------------------------------------------------------- Interface: eno1, via: LLDP, RID: 1, Time: 0 day, 00:00:59 Chassis: ChassisID: mac 00:11:22:33:44:55 SysName: office1-switch23 SysDescr: ProCurve J9280A Switch 2510G-48, revision Y.11.12, ROM N.10.02 (/sw/code/build/cod(cod11)) Capability: Bridge, on Port: PortID: local 42 PortDescr: 42 -------------------------------------------------------------------------------

Among the output formats are “json”, which is easy to re-use elsewhere.

root@turbotape:~# lldpcli -f json show neighbors { "lldp": { "interface": { "eno1": { "chassis": { "office1-switch23": { "descr": "ProCurve J9280A Switch 2510G-48, revision Y.11.12, ROM N.10.02 (/sw/code/build/cod(cod11))", "id": { "type": "mac", "value": "00:11:22:33:44:55" }, "capability": { "type": "Bridge", "enabled": true } } }, "via": "LLDP", "rid": "1", "age": "0 day, 00:53:23", "port": { "descr": "42", "id": { "type": "local", "value": "42" } } } } } } lldpad

A much more featureful LLDP daemon, available for both the Debian and RedHat OS families. This has lots of features, but is less trivial to set up.

Configure lldp for each interface #!/bin/sh find /sys/class/net/ -maxdepth 1 -name 'en*' | while read device; do basename "$device" done | while read interface; do { lldptool set-lldp -i "$interface" adminStatus=rxtx for item in sysName portDesc sysDesc sysCap mngAddr; do lldptool set-tlv -i "$interface" -V "$item" enableTx=yes | sed -e "s/^/$item /" done } | sed -e "s/^/$interface /" done Show LLDP neighbor information #!/bin/sh find /sys/class/net/ -maxdepth 1 -name 'en*' | while read device; do basename "$device" done | while read interface; do printf "%s\n" "$interface" ethtool $interface | grep -q 'Link detected: yes' || { echo " down" echo continue } lldptool get-tlv -n -i "$interface" | sed -e "s/^/ /" echo done [...] enp3s0f0 Chassis ID TLV MAC: 01:23:45:67:89:ab Port ID TLV Local: 588 Time to Live TLV 120 System Name TLV site3-row2-rack1 System Description TLV Juniper Networks, Inc. ex2200-48t-4g , version 12.3R12.4 Build date: 2016-01-20 05:03:06 UTC System Capabilities TLV System capabilities: Bridge, Router Enabled capabilities: Bridge, Router Management Address TLV IPv4: Ifindex: 36 OID: $ Port Description TLV some important server, port 4 MAC/PHY Configuration Status TLV Auto-negotiation supported and enabled PMD auto-negotiation capabilities: 0x0001 MAU type: Unknown [0x0000] Link Aggregation TLV Aggregation capable Currently aggregated Aggregated Port ID: 600 Maximum Frame Size TLV 9216 Port VLAN ID TLV PVID: 2000 VLAN Name TLV VID 2000: Name bumblebee VLAN Name TLV VID 2001: Name stumblebee VLAN Name TLV VID 2002: Name fumblebee LLDP-MED Capabilities TLV Device Type: netcon Capabilities: LLDP-MED, Network Policy, Location Identification, Extended Power via MDI-PSE End of LLDPDU TLV enp3s0f1 [...] on the switch side

On the switch, it is a bit easier to see what’s connected to each interface:

office switch

On the switch side, this system looks like:

office1-switch23# show lldp info remote-device LLDP Remote Devices Information LocalPort | ChassisId PortId PortDescr SysName --------- + ------------------------- ------ --------- ---------------------- [...] 42 | 22 33 44 55 66 77 eno1 Intel ... turbotape.example.com [...] office1-switch23# show lldp info remote-device 42 LLDP Remote Device Information Detail Local Port : 42 ChassisType : mac-address ChassisId : 00 11 22 33 33 55 PortType : interface-name PortId : eno1 SysName : turbotape.example.com System Descr : Debian GNU/Linux testing (stretch) Linux 4.5.0-1-amd64 #1... PortDescr : Intel Corporation Ethernet Connection I217-LM System Capabilities Supported : bridge, router System Capabilities Enabled : bridge, router Remote Management Address Type : ipv4 Address : Type : ipv6 Address : 20 01 0d b8 00 00 00 00 00 00 00 00 00 00 00 01 Type : all802 Address : 22 33 44 55 66 77 datacenter switch ssm@site3-row2-rack1> show lldp neighbors Local Interface Parent Interface Chassis Id Port info System Name [...] ge-0/0/38.0 ae1.0 01:23:45:67:89:58 Interface 2 as enp3s0f0 server.example.com ge-1/0/38.0 ae1.0 01:23:45:67:89:58 Interface 3 as enp3s0f1 server.example.com [...] ssm@site3-row2-rack1> show lldp neighbors interface ge-0/0/38 LLDP Neighbor Information: Local Information: Index: 157 Time to live: 120 Time mark: Fri Apr 29 13:00:19 2016 Age: 24 secs Local Interface : ge-0/0/38.0 Parent Interface : ae1.0 Local Port ID : 588 Ageout Count : 0 Neighbour Information: Chassis type : Mac address Chassis ID : 01:23:45:67:89:58 Port type : Mac address Port ID : 01:23:45:67:89:58 Port description : Interface 2 as enp3s0f0 System name : server.example.com System Description : Linux server.example.com 3.10.0-327.13.1.el7.x86_64 #1 SMP Thu Mar 4 System capabilities Supported : Station Only Enabled : Station Only Management Info Type : IPv6 Address : 2001:0db8:0000:0000:0000:dead:beef:cafe Port ID : 2 Subtype : 2 Interface Subtype : ifIndex(2) OID :
Categories: FLOSS Project Planets

Joey Hess: my Shuttleworth Foundation flash grant

Planet Debian - Wed, 2016-04-27 17:32

Six months ago I received a small grant from the Shuttleworth Foundation with no strings attached other than I should write this blog post about it. That was a nice surprise.

The main thing that ended up being supported by the grant was work on Propellor, my configuration management system that is configured by writing Haskell code. I made 11 releases of Propellor in the grant period, with some improvements from me, and lots more from other contributors. The biggest feature that I added to Propellor was LetsEncrypt support.

More important than features is making Propellor prevent more classes of mistakes, by creative use of the type system. The biggest improvement in this area was type checking the OSes of Propellor properties, so Propellor can reject host configurations that combine eg, Linux-only and FreeBSD-only properties.

Turns out that the same groundwork needed for that is also what's needed to get Propellor to do type-level port conflict detection. I have a branch underway that does that, although it's not quite done yet.

The grant also funded some of my work on git-annex. My main funding for git-annex doesn't cover development of the git-annex assistant, so the grant filled in that gap, particularly in updating the assistant to support the git-annex v6 repo format.

I've very happy to have received this grant, and with the things it enabled me to work on.

Categories: FLOSS Project Planets

DrupalEasy: Just in case - Drupal 8's /core/rebuild.php

Planet Drupal - Wed, 2016-04-27 14:59

Drupal 8 has lots of things that Drupal 7 doesn't have - a modern object-oriented foundation, the Twig templating system, and WYSIWYG out-of-the-box - just to name a few. There's also a good number of less flashy additions that are designed to improve the developer experience. One of these additions is the /core/rebuild.php file. 

While it is common knowledge that clearing rebuilding Drupal's caches is good practice during development, Drupal 8 brings a new tool to the table to get it done. Previous to Drupal 8, most developers utilized Drush to clear caches, some less-efficient folks cleared caches from the user interface (usually from the Admin Menu, but sometimes - gasp! - from the admin/config/development/performance page). 

Drupal 8 comes with a new /core/rebuild.php file that doesn't require the Drupal 8 site to be functioning (fatal errors, anyone?) nor does it require Drush or Drupal Console. Instead, as long as your site's configuration allows it, all you have to do is navigate to /core/rebuild.php in your browser. As Drupal's documentation states, this "Rebuilds all Drupal caches even when Drupal itself does not work."

How do you know if your site's configuration supports this functionality? Well, if you're working locally (and if you're developing, you should be working locally), then just make sure that $settings['rebuild_access'] = TRUE; in your settings.php (or, better yet, settings.local.php). The other method involves running the /core/scripts/rebuild_token_calculator.sh script from the commandline, and using the results as query parameters for /core/rebuild.php (see "Method II" on https://www.drupal.org/node/2153725).

Granted, most developers have Drush and/or Drupal Console installed everywhere they're working, but it's always good to have a backup method for rebuilding Drupal's caches - just in case.

Categories: FLOSS Project Planets

Niels Thykier: auto-decrufter in top 5 after 10 months

Planet Debian - Wed, 2016-04-27 14:44

About 10 months ago, we enabled an auto-decrufter in dak.  Then after 3 months it had become the top 11th “remover”.  Today, there are only 3 humans left that have removed more packages than the auto-decrufter… impressively enough, one of them is not even an active FTP-master (anymore).  The current score board:

5371 Luca Falavigna 5121 Alexander Reichle-Schmehl 4401 Ansgar Burchardt 3928 DAK's auto-decrufter 3257 Scott Kitterman 2225 Joerg Jaspert 1983 James Troup 1793 Torsten Werner 1025 Jeroen van Wolffelaar 763 Ryan Murray

For comparison, here is the number removals by year for the past 6 years:

5103 2011 2765 2012 3342 2013 3394 2014 3766 2015 (1842 removed by auto-decrufter) 2845 2016 (2086 removed by auto-decrufter)

Which tells us that in 2015, the FTP masters and the decrufter performed on average over 10 removals a day.  And by the looks of it, 2016 will surpass that.  Of course, the auto-decrufter has a tendency to increase the number of removed items since it is an advocate of “remove early, remove often!”.


Data is from https://ftp-master.debian.org/removals-full.txt.  Scoreboard computed as:

  grep ftpmaster: removals-full.txt | \ perl -pe 's/.*ftpmaster:\s+//; s/\]$//;' | \ sort | uniq -c | sort --numeric --reverse | head -n10

Removals by year computed as:

grep ftpmaster: removals-full.txt | \ perl -pe 's/.* (\d{4}) \d{2}:\d{2}:\d{2}.*/$1/' | uniq -c | tail -n6

(yes, both could be done with fewer commands)

Filed under: Debian
Categories: FLOSS Project Planets

Chromatic: Chromatic at DrupalCon New Orleans

Planet Drupal - Wed, 2016-04-27 12:36

DrupalCon New Orleans is nearly here and Chromatic will be attending in full force! Here's the rundown of what you need to know:

Learn About Render Arrays from Gus Childs

Gus will be presenting his session on Drupal 8 render arrays on Tuesday at 1pm in the Blackmesh Room (267-268): Aha! Understanding and Using Render Arrays in Drupal 8. If you're ever been confused by render arrays or just want to learn the best practices for how you're supposed to use them, be sure not to miss this session. Gus happens to be an awesome presenter to boot!

Schedule or Attend a BoF in the Chromatic Room

We're once again sponsoring a Birds of a Feather room. BoFs are a great way for folks to get together and discuss interesting topics in a more informal setting. There's already some great BoFs scheduled for the Chromatic room, including one on Drupal VM and Local Development from the well-known geerlingguy. We have a couple BoFs of our own too:

If you have a great idea for a BoF, schedule one for the Chromatic room!

Connect with Us at one of the Summits

In addition to attending the conference proper, we're once again sending our leadership to the Business Summit, as well as sending a couple of folks to the Media & Publishing Summit.

Grab Some Swag

Every year, DrupalCon attendees rave about how awesome and comfortable our t-shirts are. That's because we don't believe in making swag that we ourselves wouldn't love to wear. This year is no different. For NOLA, we've made a limited run of some special vintage baseball tees, printed on 3/4 sleeve American Apparel 50/50 cotton. These shirts are our best yet and we want to give you one for FREE!

See you in New Orleans!

Categories: FLOSS Project Planets

John Cook: How to create Green noise in Python

Planet Python - Wed, 2016-04-27 11:52

This is a follow-on to my previous post on green noise. Here we create green noise with Python by passing white noise through a Butterworth filter.

Green noise is in the middle of the audible spectrum (on the Bark scale), just where our hearing is most sensitive, analogous to the green light, the frequency where our eyes are most sensitive. See previous post for details, including an explanation of where the left and right cutoffs below come from.

Here’s the code:

from scipy.io.wavfile import write from scipy.signal import buttord, butter, filtfilt from scipy.stats import norm from numpy import int16 def turn_green(signal, samp_rate): # start and stop of green noise range left = 1612 # Hz right = 2919 # Hz nyquist = (samp_rate/2) left_pass = 1.1*left/nyquist left_stop = 0.9*left/nyquist right_pass = 0.9*right/nyquist right_stop = 1.1*right/nyquist (N, Wn) = buttord(wp=[left_pass, right_pass], ws=[left_stop, right_stop], gpass=2, gstop=30, analog=0) (b, a) = butter(N, Wn, btype='band', analog=0, output='ba') return filtfilt(b, a, signal) def to_integer(signal): # Take samples in [-1, 1] and scale to 16-bit integers, # values between -2^15 and 2^15 - 1. signal /= max(signal) return int16(signal*(2**15 - 1)) N = 48000 # samples per second white_noise= norm.rvs(0, 1, 3*N) # three seconds of audio green = turn_green(white_noise, N) write("green_noise.wav", N, to_integer(green))

And here’s what it sounds like:


(download .wav file)

Let’s look at the spectrum to see whether it looks right. We’ll use one second of the signal so the x-axis coincides with frequency when we plot the FFT.

from scipy.fftpack import fft one_sec = green[0:N] plt.plot(abs(fft(one_sec))) plt.xlim((1500, 3000)) plt.show()

Here’s the output, concentrated between 1600 and 3000 Hz as expected:

Categories: FLOSS Project Planets

ImageX Media: Inbound Marketing in 2016 - Why it's Essential to Your Web Success

Planet Drupal - Wed, 2016-04-27 10:44

Devising an effective inbound marketing strategy is vital for countless reasons, and is critical for the growth of business in terms of gaining exposure, building your brand, and truly 'connecting' with prospective leads and customers, members of your industry's target audience.  

There are countless reasons this type of marketing is so essential to your success on the Internet, including:

Categories: FLOSS Project Planets

Philip Semanchuk: Embedding Python: How To Confuse Python and Yourself

Planet Python - Wed, 2016-04-27 09:08

This is a cautionary tale about how embedded Python finds its runtime files under Windows. Don’t worry, though — everyone lives happily ever after.

The story begins with a client’s request to build an executable under Windows that embeds Python. This is not so hard; there is documentation for embedding Python.

I had two versions of Python installed on my Windows virtual machine because I was experimenting with different Pythons at the time. (Doesn’t everyone go through an experimental phase in their youth?) In C:\Python27 I had installed 64-bit Python 2.7.9 from Python.org, and in C:\Users\philip\Miniconda2 I had installed 64-bit Python 2.7.11 from Continuum. The 2.7.9 version was an older install. I was only interested in using the 2.7.11 version.

My executable’s C code told Python where to find its runtime files —


After compiling and linking, I ran my Python-embedding executable which imported my hello_world.py file. It printed “Hello world!” as expected.

Here Come the Dragons

I thought everything was fine until I added these print statements to my Python code —

import sys print sys.exec_prefix print sys.version

The output was not what I expected —

C:/Users/philip/Miniconda2 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)]

This is contradictory! The Miniconda Python was 2.7.11, yet sys.version identified itself as Python 2.7.9. How can the same module report two different Pythons simultaneously?

What Went Wrong

The sys module reported information about two Pythons simultaneously because I was mixing parts from two different Python runtimes simultaneously.

Python’s runtime consists of a dynamically loaded library (python27.dll), the standard library which is largely written in Python itself, and an executable. The executable is usually python.exe, but in this case it was my C program that embedded Python. When my C program asked Windows to find python27.dll, Windows searched for it in these directories as documented in the Windows DLL search strategy  —

  1. The directory where the executable module for the current process is located.
  2. The current directory.
  3. The Windows system directory (usually C:\Windows\system32).
  4. The Windows directory (usually C:\Windows).
  5. The directories listed in the PATH environment variable.

My problem was that Windows found C:\Windows\system32\python27.dll first, and that was from my Python 2.7.9 installation. Meanwhile, my call to Py_SetPythonHome() had told Python to use the standard library from Miniconda Python. The value of sys.version comes from a string hardcoded in the runtime DLL, while sys.exec_prefix is derived from the value I passed in Py_SetPythonHome(). I was using the standard library from one Python installation, and the runtime DLL from another.


Although I didn’t experiment with this for long, I might not have noticed that there was a problem if I hadn’t been lucky enough to double check my Python setup with the sys module. The standard library probably doesn’t care about which interpreter it runs under. I can imagine a few cases may exist where changes/bug fixes were made to the Python part of the standard library for versions 2.7.10 and 2.7.11 that rely on corresponding changes to the binary runtime, and that code might behave badly.

Both of the Pythons I was using were built with the same compiler, so theoretically binary extensions like numpy should run just fine under either Python. But I could certainly forgive numpy if it crashed as a result.

In short, this is neither a typical nor a supported use of Python which puts it in the “Here there be dragons” category.

The Solution

The solution was very simple. I copied C:\Users\philip\Miniconda2\python27.dll into the same directory as my custom executable. Since that’s the first location Windows searches when loading a DLL, it isolates my code from other Python DLLs that might appear in (or disappear from) other locations in the file system. Problem solved!

Categories: FLOSS Project Planets

TeX: Justification with no hyphenation

Planet KDE - Wed, 2016-04-27 08:56

I’ve been reading a copy of 1984 by George Orwell, published by Fingerprint publishing — a beautifully typeset one. Already into half of Part II, but that’s when I noticed that the book is typeset with full justification sans any hyphenation. Incidentally I was typesetting something else, which probably is the reason I noticed it now. And I wanted to typeset my article the same way, with full justification but no hyphenation.

Biggest strength of TeX is its line, paragraph and page breaking algorithms where hyphenation plays a big part. Thus removing hyphenation means taking away a lot of those advantages. In any case, there are seemingly multiple ways to do it. From the TeX FAQ:

Use hyphenat package with


or set \hyphenpenalty and \exhyphenpenalty to 10000. These avoided hyphenation, but justification was bad – there were long words extending beyond the right margin. So moved on to next solution:

\hyphenchar\font=-1 \sloppy

This one kept hyphenation to the minimum with justified text, but didn’t fully avoid it. And it would work only for text with current font. \sloppy sets \tolerance and \emergencystretch to large values.

And the last one, which provided full justification with no hyphenation is:

%\tolerance=1 \emergencystretch=\maxdimen \hyphenpenalty=10000

The \tolerance parameter sets how much badness is allowed, which influence paragraph/line breaking. \emergencystretch is the magical parameter which stretches text over multiple passes to balance the spacing. Setting \hyphenpenalty to infinite value actually has better effect than setting \tolerance.

Tagged: TeX
Categories: FLOSS Project Planets

Google Summer Of Code 2016

Planet KDE - Wed, 2016-04-27 08:52

The good news is i got selected for GSoC 2016, well its pretty exciting. As a part of my GSoC project with KDE i will be working on Propagator. So in this blog i will be spamming you with all my GSoC experiences and related technicalities.

Categories: FLOSS Project Planets
Syndicate content