FLOSS Project Planets

Learning Python and Django – PT I

Planet KDE - Fri, 2016-09-16 14:54

A month ago I subscribe to a course from a Brazilian friend, Henrique Bastos, an awesome guy that knows a lot about Python and Entrepreneurship.
My goal with this course is to learn others programming languages, and in this case, Django is the Python framework for the Web, and I think that would be a good skill to have.
I don’t know anything about Python, the only skill that I have is with C++ and Qt, and isn’t an advanced skill. And here in Brazil, a lot of my friends knows Python, and always happen a discussion about wich programming is better and why. But for me build arguments like that I need to learn the language. To see its weakness and strengths.

I’m finishing the second module of the course. And in the last lessons, Henrique focus on the test. Everything that we need to implement to handle requests and a subscription form of our app, need to have a test first. Is the TAFT method: Test all the fucking time!

In a first moment I got pissed, why not go directly to implementation? But with the development of the app and the tests that I was writing, I started to see the importance of the tests. Funny is, that in college I’m doing a class called Software Engineer II, the current topic is about tests, and how the concept of the software tests is defined and wich kind of tests we need to do in a software.

This is an example of a class inside my test.py, to check if a Post on the subscription form is valid or not:

class SubscribeInvalidPost(TestCase): def setUp(self): self.resp = self.client.post('/inscricao/', {}) def test_post(self): """Invalid POST should not redirect""" self.assertEqual(200, self.resp.status_code) def test_template(self): self.assertTemplateUsed(self.resp, 'subscriptions/subscription_form.html') def test_has_form(self): form = self.resp.context['form'] self.assertIsInstance(form, SubscriptionForm) def test_form_has_errors(self): form = self.resp.context['form'] self.assertTrue(form.errors)

I’m having some difficulties on the Python syntax. Since in all my short life like a programmer was using C/C++ or even Pascal(in the very beginning), I’m used to using the syntax of them. I got kind of upset after an if() that I forgot to add the ‘:’ in the end. >.<

However, so far, I like it a lot of this course, Henrique usually says that “Python comes with batteries”, that mean, Python has all the support on the API that you need, without the need to worry about what is happening below or to implement a new class to handle some specifications. And I’m learning that.

Django has an API called Forms, it’s a library only to handle formulary data. You can clean the data and have a dictionary returned with the cleaned data, you can validate all the data from the forms: If the data is an email, the Django already checks if is a valid email. Click here for the doc.

We are using Heroku, like the platform of deploy. And damn, it’s a hell of a tool. I hope in the future create more projects and update them to Heroku. You can work on everything on your localhost, and they configure Heroku, after that you can push to a branch inside Heroku platform, and it will install everything that you need for your project to run, build and put on the air. Easy like plug and play =D

A good thing to make clear is that Django is an API for backend in Python. To front-end, you can use everything that you like it. =D

Until the end of the course, I will finish a website with all database behind to handle subscriptions and manage of a hypothetical event.

I will continue to post about it, see you in the next post!

That’s all folks! =D



Categories: FLOSS Project Planets

CU Boulder - Webcentral: Drupal as a Service at CU Boulder

Planet Drupal - Fri, 2016-09-16 13:22
The Web Express service at CU Boulder runs over 1,000 Drupal-powered web sites and is designed, developed and maintained by a team of nine . For our team to support the service, we created a design, development and user experience strategy that allows us to support and scale the service to the campus needs. When we started with Drupal, we were eager to do any custom design and development work that was required for a project. In a short amount of time, we had a lot of custom sites on custom codebases. We learned that this was not something that...
Categories: FLOSS Project Planets

pythonwise: Simple Object Pools

Planet Python - Fri, 2016-09-16 12:49
Sometimes we need object pools to limit the number of resource consumed. The most common example is database connnections.

In Go we sometime use a buffered channel as a simple object pool.

In Python, we can dome something similar with a Queue. Python's context manager makes the resource handing automatic so clients don't need to remember to return the object.

Here's the output of both programs:

$ go run pool.go
worker 7 got resource 0
worker 0 got resource 2
worker 3 got resource 1
worker 8 got resource 2
worker 1 got resource 0
worker 9 got resource 1
worker 5 got resource 1
worker 4 got resource 0
worker 2 got resource 2
worker 6 got resource 1

$ python pool.py
worker 5 got resource 1
worker 8 got resource 2
worker 1 got resource 3
worker 4 got resource 1
worker 0 got resource 2
worker 7 got resource 3
worker 6 got resource 1
worker 3 got resource 2
worker 9 got resource 3
worker 2 got resource 1
Categories: FLOSS Project Planets

Enthought: Canopy Data Import Tool: New Updates

Planet Python - Fri, 2016-09-16 12:47

In May of 2016 we released the Canopy Data Import Tool, a significant new feature of our Canopy graphical analysis environment software. With the Data Import Tool, users can now quickly and easily import CSVs and other structured text files into Pandas DataFrames through a graphical interface, manipulate the data, and create reusable Python scripts to speed future data wrangling.

Watch a 2-minute demo video to see how the Canopy Data Import Tool works:

With the latest version of the Data Import Tool released this month (v. 1.0.4), we’ve added new capabilities and enhancements, including:

  1. The ability to select and import a specific table from among multiple tables on a webpage,
  2. Intelligent alerts regarding the saved state of exported Python code, and
  3. Unlimited file sizes supported for import.

Download Canopy and start a free 7 day trial of the data import tool

New: Choosing from multiple tables on a webpage

The latest release of the Canopy Data Import Tool supports the selection of a specific table from a webpage for import, such as this Wikipedia page

In addition to CSVs and structured text files, the Canopy Data Import Tool (the Tool) provides the ability to load tables from a webpage. If the webpage contains multiple tables, by default the Tool loads the first table.

With this release, we provide the user with the ability to choose from multiple tables to import using a scrollable index parameter to select the table of interest for import.

Example: loading and working with tables from a Wikipedia page

For example, let’s try to load a table from the Demography of the UK wiki page using the Tool. In total, there are 10 tables on that wiki page.

  • As you can see in the screenshot below, the Tool initially loads the first table on the wiki page.
  • However, we are interested in loading the table ‘Vital statistics since 1960’, which is the fifth table on the page. (Note that indexing starts at 0). For a quick history lesson on why Python uses zero based indexing, see Guido van Rossum’s explanation here).
  • After the initial read-in, we can click on the ‘Table index on page’ scroll bar, choose ‘4’ and click on ‘Refresh Data’ to load the table of interest in the Data Import Tool.

See how the Canopy Data Import Tool loads a table from a webpage and prepares the data for manipulation and interaction:

The Data Import Tool allows you to select a specific table from a webpage where multiple are present, with a simple drop down menu. Once you’ve selected your table, you can readily toggle between 3 views: the Pandas DataFrame generated by the Tool, the raw data and the corresponding auto-generated Python code. Consecutively, you can export the DataFrame to the IPython console for further plotting and further analysis.

  • Further, as you can see, the first row contains column names and the first column looks like an index for the Data Frame. Therefore, you can select the ‘First row is column names’ checkbox and again click on ‘Refresh Data’ to prompt the Tool to re-read the table but, this time, use the data in the first row as column names. Then, we can right-click on the first column and select the ‘Set as Index’ option to make column 0 the index of the DataFrame.
  • You can toggle between the DataFrame, Raw Data and Python Code tabs in the Tool, to peek at the raw data being loaded by the Tool and the corresponding Python code auto-generated by the Tool.
  • Finally, you can click on the ‘Use DataFrame’ button, in the bottom right, to send the DataFrame to the IPython kernel in the Canopy User Environment, for plotting and further analysis.

New: Keeping track of exported Python scripts

The Tool generates Python commands for all operations performed by the user and provides the user with the ability to save the generated Python script. With this new update, the Tool keeps track of the saved and current states of the generated Python script and intelligently alerts the user if he/she clicks on theUse DataFrame’ button without saving changes in the Python script.

New: Unlimited file sizes supported for import

In the initial release, we chose to limit the file sizes that can be imported using the Tool to 70 MB, to ensure optimal performance. With this release, we removed that restriction and allow files of any size to be uploaded with the tool. For files over 70 MB we now provide the user with a warning that interaction, manipulation and operations on the imported Data Frame might be slower than normal, and allow them to select whether to continue or begin with a smaller subset of data to develop a script to be applied to the larger data set.

Additions and Fixes

Along with the feature additions discussed above, based on continued user feedback, we implemented a number of UI/UX improvements and bug fixes in this release. For a complete list of changes introduced in version 1.0.4 of the Data Import Tool, please refer to the Release Notes page in the Tool’s documentation. If you have any feedback regarding the Data Import Tool, we’d love to hear from you at canopy.support@enthought.com.

Additional resources:

Download Canopy and start a free 7 day trial of the data import tool

See the Webinar “Fast Forward Through Data Analysis Dirty Work” for examples of how the Canopy Data Import Tool accelerates data munging:


Categories: FLOSS Project Planets

More Hiring: Qt Hacker!

Planet KDE - Fri, 2016-09-16 11:24

ownCloud is even more hiring. In my last post I wrote that we need PHP developers, a security engineer and a system administrator.
For all positions we got interesting inquiries already. That’s great, but should not hinder you from sending your CV in case you are still interested. We have multiple positions!

But there is even more opportunity: Additionally we are looking for an ownCloud Desktop Client developer. That would be somebody fluid in C++ and Qt who likes to pick up responsibility for our desktop client together with the other guys on the team. Shifted responsibilities have created this space, and it is your chance to
jump into the desktop sync topic which makes ownCloud really unique.

The role includes working with the team to plan and roll out releases, coordinate with the server- and mobile client colleagues, nail out future developments, engage with community hackers and help with difficult support cases. And last but not least there is hacking fun on remarkable nice Qt based C++ code of our desktop client, together with high profile C++ hackers to learn from.

It is an ideal opportunity for a carer type of personality, to whom it is not enough to sit in the basement and only hack, but also to talk to people, organize, and become visible. Having a Qt- and/or KDE background is a great benefit. You would work from where you feel comfortable with as ownCloud is a distributed company.

The ownCloud Client is a very successful part of the ownCloud platform, it has millions of installations out there, and is released under GPL.

If you want to do something that matters, here you are! Send your CV today and do not forget to mention your github account

Categories: FLOSS Project Planets

Mediacurrent: Meet Matt Davis

Planet Drupal - Fri, 2016-09-16 11:02

Hi Matt - thanks for letting us get to know a bit more about you! Let's get started:

1. What’s your role at Mediacurrent, both internally and client-related?

I am a Lead Drupal Architect, which means I do a lot of requirements gathering, discovery, and planning out large scale builds, and then I help lead development teams in executing those builds.

Categories: FLOSS Project Planets

OSTraining: How to Add Audio Files to Drupal 8 Sites

Planet Drupal - Fri, 2016-09-16 09:58

This week, an OSTraining member asked us about adding audio files to their Drupal 8 site.

Using the Drupal AudioField module, I will explain how you can display a simple HTML5 player for your site's audio files.

In order to get started, you must download, install and enable the AudioField module.

Categories: FLOSS Project Planets

Telegram Connection Manager (Morse) released.

Planet KDE - Fri, 2016-09-16 07:57

I work on Qt-based Telegram Connection Manager for Telepathy and today I finally released the first version: 0.1.0.

If you don’t know about the project, then you can read the previous post with introduction.

Please note that though I also work on KDE Telepathy, there is no any dependency on KDE, so the Connection Manager can benefit Empathy, Jolla (Messages), Ubuntu IM client and any other software, powered by Telepathy.

Since the last post, the project got basic group chat implementation, migrated to native (telegram) identifiers, got two-step verification support and so on. If you just want to try it right now, then skip the follow couple of paragraphs and scroll down to the Links section at the end of the post.

Let’s dive into details.

Features overview What is expected to work:
  • Contact list with first/last names
  • Contact avatars
  • Personal chat (one to one)
  • Simple group chat (no creation and no channels, see details below)
  • User typing events
  • Full message delivery status support (details below)
  • Own presence (online, offline, hidden)
  • Two-step verification (details below)
  • Loading unread messages on connect
  • Sessions (Means that you don’t have to get confirmation code again and again)
  • Restoring connection on network problem
What does not work:
  • Contact management. You can remove a contact for contact list, but you can not add they. Telepathy requires a contact identifier to add it to your contact list; Telegram gives you the contact identifier only as a result of adding it to your contact list by a phone number. Phone number can not be used as an identifier in Telepathy-Morse, because the contacts phone number can be unknown, can be changed or become unavailable and so on. Looks like a circle or a deadlock. Thanks to Robert McQueen for the workaround idea: we can introduce a “tel:+12345..” identifiers, which will be suitable only for subscription requests. I will try to implement this in the next version.
  • Multimedia messages does not work, except Geo. See details below.
Details Two-step verification

Two months ago I added two-step verification support to TelegramQt and Morse and found that this part of Telepathy can be significantly improved. Currently Telepathy spec offers two authentication interfaces: SASLAuthentication and CaptchaAuthentication. Captcha is obviously not what we need for Telegram, so we have to use SASLAuthentication and there is a limitation, which badly affects UX: it is not possible to distinguish authentication channels (requests), which means that there is no visual difference between phone code and password requests.

I would like to bring all Telegram features to Telepathy, which means that in long-term we need to support all methods to obtain the auth code. As far as I know, Telegram user can request an in-application code, a SMS message with code or a voice call to the account phone number (“a robot will repeat the confirmation code from a previously sent SMS message.”). For Telepathy it means that the set of available authentication methods can be changed dynamically (“SendCall” usually becomes available only after “SendSMS”). As some of you can recall, there was been a plan to use an awesome project KDE Connect, to (semi-)automatically pass the received auth code to the ConnectionManager. In order to support all this features in a generic manner, I have a plan to develop OTP Authentication Channel Interface.

Group chat

Group chat creation is not supported, because Telepathy needs to get the new “room” identifier and handle right on creation, but we need to wait for Telegram server reply to get the result of operation (the result will contain telegram identifier if the request is succeed). This does not work, because TelepathyQt does not support delayed DBus replies and I have no solid solution here yet.

As far as I know, Telepathy does not support “group chat list” (as it support contact list), so I had to workaround it: Morse exposes group chat list in room search result.

I’m planning to add Telegram Channel support sometime soon; I need to update the TelegramQt to newer MProto layer, but that should not be a problem.

Incredible (full) support of message (delivery) status

For a year Morse support follow standard things:

  • delivery status “Accepted” on message delivered to the server
  • delivery status “Read” on message read by the recipient
  • send “MessageRead” on message acknowledged by Telepathy client

But also you can:

  • Send a message from a device and see it “sent” in Telepathy client
  • Receive a message on all clients at the same time
  • Read a message from a device and get it removed from the Telepathy pending messages list by Morse. Please, let me know, if such feature is already implemented in any other telepathy connection manager. Also, please take a note, that I had to change TpQt API for this feature and the change will be in TelepathyQt-0.9.8, which is not released yet.

Geopoint messages

I added geo message support in form of text/plain RFC 5870 and application/geo+json mime type to Morse and implemented a “filter” for the text/plain version for the KDE Telepathy Text UI, so received geo messages will appear in OpenStreetMap iframe with the “geo:” text replaced by a link. Zoom argument also implemented in KTp, but there is no such thing in Telegram, so the default zoom value will be used instead.

As far as I see, it is not included in KDE Applications 16.08, but I’m sure it will be available in the upcoming 16.12 release. I disabled the filter by default for the first release, so you have to enabled it via Settings->Configure Chat Application->Plugins->Geopoint Preview.

Telegram identifiers

I changed Morse contact and group chat identifiers to expose native Telegram userId/chatId. UserId is the most suitable way to identify contacts, because it never changes, which is important for proper logging.

  • If you left a chat and then join back, the chat identifier will be the same.
  • If you removed a contact from the contact list, the identifier will be the same (phone number will be unavailable).
  • If you add a contact with a long comminication history to your contact list (effectively say Telegram that you know the contact phone number), the identifier will be the same.
  • If you or your contact changed a phone number, the identifier… you know.
ContactInfo interface

After Telepathy identifiers changed to be Telegram native identifiers, there was been no way to see phone numbers. I implemented Connection.Interface.ContactInfo to expose three (basic) contacts vCard fields:

  • nickname (username in terms of telegram)
  • tel (the phone number)
  • name (first and last name individually)

It turns out that KDE Telepathy does not support the ContactInfo interface after introduction of KPeople. It seems that TelepathyQt Factories do not support optional interfaces, so Telepathy backend for KPeople does not require ContactInfo feature to be operatable with ConnectionManager without the ContactInfo interface. It is possible to make Groups interface optional, because it is exposed as Connection feature and Connection object is available at the time for Factory, but ContactInfo is exposed as Contact feature and API don’t let us to know in advance if the ContactInfo is supported. I hope that required changes would make it into TelepathyQt-0.9.8 release.

The feature already works in Empathy as intended.

Debug interface

I implemented Debug interface in TelepathyQt and Morse, so now it is much easier to get debug messages from the connection manager. I changed KTp Debugger to make it possible to debug initialization and authorization steps of connection. Previously debugger waited for success connection before attach and that was been weird. I will share more details about made and upcoming KTp Debugger improvements in one of the next posts.

Short-term goals ContactSearch interface

I will try to implement ContactSearch interface; it should be easy enough, but to make it really useful and convenient we need to have a list of recent dialogs (Telegram Dialogs).

Connection / setAvatar()

Morse does not support the setAvatar() method yet, because at time of the avatar interface implementation there was no support for media files upload in TelegramQt. I implemented the file upload support, so this method finally can be added.

Long-term goals Multimedia messages

Multimedia messages support is mostly implemented in TelegramQt, but it will be a very complex task to bring this support to Telepathy. This task can take a year or more with current development speed. We need to change everything (except may be MissionControl) to make this works right.

TelegramQt supports multimedia messages (photos, stickers and more), but it can not be exposed via Telepathy yet

In theory, it can  be possible to implement image preview (thumbnail) with the current Telepathy specification, I will check if it is actually possible sometime soon. It is not supported in clients, so even this small step would take a lot of time.

I have some ideas and I will experiment with different approaches. May be I’ll be managed to make a Proof of Concept this winter.

Telegram Dialogs

The feature does not fit into Telepathy specification. I see two ways:

  • I can expose contacts from Dialogs in a contact list group, such as “Recent contacts”. The feature would be toggled via account parameter, such as “EnableRecentContacts”. It is a open question how to manage group chat.
  • Implement a new Telepathy interface for the Dialogs feature. Proper, but a very long way, which requires support from telepathy clients. The interface would be hard to standardize in Telepathy, because it is Telegram specific.
OTP Authentication Channel Interface

The key feature of the new interface would be the PasswordSent signal, but it would have a number of other, essential changes from the SASL interface too.

This is my plan on how the new interface will work:

  • User setups an account. The only required parameter is a phone number.
  • User sets presence to online.
  • Morse creates an OTP authentication channel (VerificationStatus: Not_Started, AvailableMethods: “sms”, PhoneNumber: the user phone number)
  • Telepathy auth application handles the channel. I have no strong decision on the UI, so the app will either:
    • calls UseMethod(“sms”) and informs the user, that verification code will be sent via sms (because it is the only available method)
    • or show a button box with sms and call actions with the last one disabled.
  • The auth handler connects via DBus to KDE Connect service signal about incoming sms. It will be even possible to find the right device (thanks to PhoneNumber interface property).
  • Morse:
    • Makes request for sms.
    • Changes VerificationStatus to In_Progress
    • On server respond, Morse emits PasswordSent signal with details:
      • PasswordRegexp is a five digits after a space
      • SenderRegexp is “Telegram” or “TeleVerify” (I would like Telegram server to tell us the exact sender phone/name.)
      • CallTimeout (received from the server. Something like 30 sec)
  • Telepathy auth application:
    • shows countdown if password details have “Expiration” value.
    • waits for KDE Connect incoming sms signal
    • on KDE Connect signal, ktp-auth-handler matches sender against SenderRegexp and then match the message against PasswordRegexp. If both checks succeed, then it either automatically call Respond with the password, or, at least, insert the code to the input line and waits for the user action.
  • Morse (optionally):
    • adds “call” method to the available method list.
  • Telepathy auth application:
    • shows buttons for all available methods (“Request an SMS”, “Request a call”)
  • Morse (if two-factor authentication is enabled (which a popular and highly recommended case)):
    • creates a usual SASLAuthentication channel for the password authentication…

There is also one important trick needed to make it works: Morse needs to know if the client is capable to handle OTP channel, because it should wait for the user decision, if the OTP interface is available (whether to send the code in-app or via sms) and it should automatically start authorization via sms otherwise, to let user to auth via plain SASL interface. This thing would require a lot of work in many Telepathy components, so it is especially good to know that ClientCapabilities interface is already available and we don’t have to invent it.

Known Issues
  • Initial low-level encryption sometimes generates bad random values, which leads to “connection doesn’t work” issue.
  • Can not send long messages (Missed TelegramQt gzip packing implementation; limit is about 400 characters; telegram protocol limitation is 4095 characters)

Both TelegramQt and Telepathy-Morse are Qt4 and Qt5 compatible.
Information about CMake build: by default CMake looks for the Qt5 build. Use USE_QT4 option (-DUSE_QT4=true) to get Qt4 build.

I think I will drop Qt4 support since version 0.2.0, if there would be no strong reasons to support it.

Account setup

It is said that there are issues with account setup. I think that problems related to configuration tools and there is nothing to fix in Morse. I would suggest to use a command-line tool as a fallback:
[kaffeine@kaffeine-pc ~]$ mc-tool add morse/telegram KaffeineTelegram string:account=1234567890
[kaffeine@kaffeine-pc ~]$ mc-tool enable morse/telegram/_312345678901
[kaffeine@kaffeine-pc ~]$
Replace KaffeineTelegram by your account display name and 1234567890 by your phone number (without leading plus).

Authentication on Sailfish

As far as I know, authentication handler just does not appear and I have no Sailfish device to debug it. I tried to use an emulator, but it did not work for me and I have tons of other things to do, so I didn’t spend hours on it.

It is possible to copy authorization key from your pc to the device, the directory is ~/.cache/telepathy-morse/secrets.

In order to fix the issue I need to get the debug information. I implemented Debug interface in Morse and work on making its possible to use KTp Debugger on Sailfish. Though this is not a high-priority task, the fix is on its way, stay tuned for updates.

  • Thanks to George Kiagiadakis for work on TelepathyQt and for the better name for the OTP Authentication Interface. I started to work on this interface named Channel.Interface.PhoneVerification, but suggested OTP seems to fit better.
  • Special thanks to David Edmundson!
  • Thanks to Telepathy developers and everyone in #telepathy irc channel on freenode.
  • Special thanks to Robert McQueen!
  • Thanks to the KDE community for awesome Telepathy Sprint in 2014 and Google Summer of Code after that.
  • That said, thanks to Google.
  • Thanks to Giovanni Santini and others for testing, adding issues and pinging me with “any progress?” and similar questions. You made me to work more on this project.

Thank you for reading!

Stay tuned

Stay tuned for the next posts:

  • New life for Telepathy Debugger. I would like to share my works on KDE Telepathy Debugger, which is getting a lot of improvements, which open possibility for QML interface and a port to mobile platforms.
  • Announce of a new, long-awaited interface for Telepathy. Comes with implementation for Morse and some NotAnnouncedYet Connection Manager (for a widely used protocol, which already has ConnectionManager with a lot of other features, but not this one). I also added the interface support into KDE Telepathy. May be someone will implement it on the telepathy-glib / Empathy side.



Categories: FLOSS Project Planets

CubicWeb: Monitor all the things! ... and early too!

Planet Python - Fri, 2016-09-16 07:34

Following the "release often, release early" mantra, I thought it might be a good idea to apply it to monitoring on one of our client projects. So right from the demo stage where we deliver a new version every few weeks (and sometimes every few days), we setup some monitoring.

Monitoring performance

The project is an application built with the CubicWeb platform, with some ElasticSearch for indexing and searching. As with any complex stack, there are a great number of places where one could monitor performance metrics.

Here are a few things we have decided to monitor, and with what tools.

Monitoring CubicWeb

To monitor our running Python code, we have decided to use statsd, since it is already built into CubicWeb's core. Out of the box, you can configure a statsd server address in your all-in-one.conf configuration. That will send out some timing statistics about some core functions.

The statsd server (there a numerous implementations, we use a simple one : python-pystatsd) gets the raw metrics and outputs them to carbon which stores the time series data in whisper files (which can be swapped out for a different technology if need be).

If we are curious about a particular function or view that might be taking too long to generate or slow down the user experience, we can just add the @statsd_timeit decorator there. Done. It's monitored.

statsd monitoring is a fire-and-forget UDP type of monitoring, it should not have any impact on the performance of what you are monitoring.

Monitoring Apache

Simply enough we re-use the statsd approach by plugging in an apache module to time the HTTP responses sent back by apache. With nginx and varnish, this is also really easy.

One of the nice things about this part is that we can then get graphs of errors since we will differentiate OK 200 type codes from 500 type codes (HTTP codes).

Monitoring ElasticSearch

ElasticSearch comes with some metrics in GET /_stats endpoint, the same goes for individual nodes, individual indices and even at cluster level. Some popular tools can be installed through the ElasticSearch plugin system or with Kibana (plugin system there too).

We decided on a different approach that fitted well with our other tools (and demonstrates their flexibility!) : pull stats out of ElasticSearch with SaltStack, push them to Carbon, pull them out with Graphite and display them in Grafana (next to our other metrics).

On the SaltStack side, we wrote a two line execution module (elasticsearch.py)

import requests def stats: return request.get('http://localhost:9200/_stats').json()

This gets shipped using the custom execution modules mechanism (_modules and saltutils.sync_modules), and is executed every minute (or less) in the salt scheduler. The resulting dictionary is fed to the carbon returner that is configured to talk to a carbon server somewhere nearby.

# salt demohost elasticsearch.stats [snip] { "indextime_inmillis" : 30, [snip] Monitoring web metrics

To evaluate parts of the performance of a web page we can look at some metrics such as the number of assets the browser will need to download, the size of the assets (js, css, images, etc) and even things such as the number of subdomains used to deliver assets. You can take a look at such metrics in most developer tools available in the browser, but we want to graph this over time. A nice tool for this is sitespeed.io (written in javascript with phantomjs). Out of the box, it has a graphite outputter so we just have to add --graphiteHost FQDN. sitespeed.io even recommends using grafana to visualize the results and publishes some example dashboards that can be adapted to your needs.

The sitespeed.io command is configured and run by salt using pillars and its scheduler.

We will have to take a look at using their jenkins plugin with our jenkins continuous integration instance.

Monitoring crashes / errors / bugs

Applications will have bugs (in particular when released often to get a client to validate some design choices early). Level 0 is having your client calling you up saying the application has crashed. The next level is watching some log somewhere to see those errors pop up. The next level is centralised logs on which you can monitor the numerous pieces of your application (rsyslog over UDP helps here, graylog might be a good solution for visualisation).

When it starts getting useful and usable is when your bugs get reported with some rich context. That's when using sentry gets in. It's free software developed on github (although the website does not really show that) and it is written in python, so it was a good match for our culture. And it is pretty awesome too.

We plug sentry into our WSGI pipeline (thanks to cubicweb-pyramid) by installing and configuring the sentry cube : cubicweb-sentry. This will catch rich context bugs and provide us with vital information about what the user was doing when the crash occured.

This also helps sharing bug information within a team.

The sentry cube reports on errors being raised when using the web application, but can also catch some errors when running some maintenance or import commands (ccplugins in CubicWeb). In this particular case, a lot of importing is being done and Sentry can detect and help us triage the import errors with context on which files are failing.

Monitoring usage / client side

This part is a bit neglected for the moment. Client side we can use Javascript to monitor usage. Some basic metrics can come from piwik which is usually used for audience statistics. To get more precise statistics we've been told Boomerang has an interesting approach, enabling a closer look at how fast a page was displayed client side, how much time was spend on DNS, etc.

On the client side, we're also looking at two features of the Sentry project : the raven-js client which reports Javascript errors directly from the browser to the Sentry server, and the user feedback form which captures some context when something goes wrong or a user/client wants to report that something should be changed on a given page.

Load testing - coverage

To wrap up, we also often generate traffic to catch some bugs and performance metrics automatically :

  • wget --mirror $URL
  • linkchecker $URL
  • for $search_term in cat corpus; do wget URL/$search_term ; done
  • wapiti $URL --scope page
  • nikto $URL

Then watch the graphs and the errors in Sentry... Fix them. Restart.

Graphing it in Grafana

We've spend little time on the dashboard yet since we're concentrating on collecting the metrics for now. But here is a glimpse of the "work in progress" dashboard which combines various data sources and various metrics on the same screen and the same time scale.

Further plans
  • internal health checks, we're taking a look at python-hospital and healthz: Stop reverse engineering applications and start monitoring from the inside (Monitorama) (the idea is to distinguish between the app is running and the app is serving it's purpose), and pyramid_health
  • graph the number of Sentry errors and the number of types of errors: the sentry API should be able to give us this information. Feed it to Salt and Carbon.
  • setup some alerting : next versions of Grafana will be doing that, or with elastalert
  • setup "release version X" events in Graphite that are displayed in Grafana, maybe with some manual command or a postcreate command when using docker-compose up ?
  • make it easier for devs to have this kind of setup. Using this suite of tools in developement might sometimes be overkill, but can be useful.
Categories: FLOSS Project Planets

Marius Gedminas: Ansible is great

Planet Python - Fri, 2016-09-16 06:59

I haven’t used configuration management tools in a very long time because I didn’t know which tool to pick (Chef? Puppet? Ansible? Salt? cfengine?). Now I use Ansible and am very happy with it.

Here’s how you start – with an empty git repository:

$ mkdir my-infra $ cd my-infra $ git init .

Let’s start by making sure a couple of Ubuntu servers will have unattended upgrades enabled. First we tell Ansible what the servers are

$ vim hosts $ cat hosts debesis ranka

Let’s make sure Ansible can ssh to those hosts:

$ ansible -i hosts all -m ping ranka | SUCCESS => { "changed": false, "ping": "pong" } debesis | SUCCESS => { "changed": false, "ping": "pong" }

I’ve an ~/.ssh/config that expands these nicknames to the full hostnames, and I’ve a private SSH key that allows me access without providing my password every time. The key is passphrase-protected and I use ssh-agent to avoid typing my passphrase every time. Notice a trend? I like convenience.

Speaking of convenience, it’s annoying to have to specify -i hosts every time (since the default inventory file is /etc/ansible/hosts and I never ever want that), so

$ vim ansible.cfg $ cat ansible.cfg [defaults] inventory = hosts

Now ansible all -m ping should work. All this needs on the servers is SSH and Python 2.x (on Ubuntu 16.04 you’ll have to apt install python).

We can do more useful ad-hoc stuff too, like apply any pending security updates:

$ ansible all -m apt -a 'update_cache=yes upgrade=dist' -u root debesis | SUCCESS => { "changed": false, "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", "stderr": "", "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", "stdout_lines": [ "Reading package lists...", "Building dependency tree...", "Reading state information...", "0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded." ] } ranka | SUCCESS => { "changed": false, "msg": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", "stderr": "", "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\n0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.\n", "stdout_lines": [ "Reading package lists...", "Building dependency tree...", "Reading state information...", "0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded." ] }

This runs apt-get update && apt-get dist-upgrade, so it needs root access. You can ssh directly as root (-u root), or you can use sudo (-b -K). I prefer ssh’ing as root because I don’t want to be typing my sudo password every time I run ansible. Let’s make this the default then:

$ vim ansible.cfg $ cat ansible.cfg [defaults] inventory = hosts remote_user = root

Now, back to unattended-upgrades. Let’s create a playbook:

$ vim setup.yml $ cat setup.yml --- - hosts: all tasks: - name: install unattended-upgrades apt: name=unattended-upgrades state=present - name: enable unattended-upgrades copy: dest: /etc/apt/apt.conf.d/50unattended-upgrades-local content: | APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Unattended-Upgrade "1"; Unattended-Upgrade::Mail "root"; Unattended-Upgrade::Remove-Unused-Dependencies "true"; Unattended-Upgrade::Automatic-Reboot "true"; Unattended-Upgrade::Allowed-Origins { "${distro_id}:${distro_codename}-security"; "${distro_id}:${distro_codename}-updates"; };

It does two things: install a package and set up a config file. Let’s see what it would do:

$ ansible-playbook setup.yml -CD PLAY [all] ********************************************************************* TASK [setup] ******************************************************************* ok: [ranka] ok: [debesis] TASK [install unattended-upgrades] ********************************************* ok: [ranka] ok: [debesis] TASK [enable unattended-upgrades] ********************************************** changed: [ranka] --- before +++ after: /tmp/tmpcIWt6O @@ -0,0 +1,9 @@ +APT::Periodic::Update-Package-Lists "1"; +APT::Periodic::Unattended-Upgrade "1"; +Unattended-Upgrade::Mail "root"; +Unattended-Upgrade::Remove-Unused-Dependencies "true"; +Unattended-Upgrade::Automatic-Reboot "true"; +Unattended-Upgrade::Allowed-Origins { + "${distro_id}:${distro_codename}-security"; + "${distro_id}:${distro_codename}-updates"; +}; changed: [debesis] --- before +++ after: /tmp/tmpePUnG4 @@ -0,0 +1,9 @@ +APT::Periodic::Update-Package-Lists "1"; +APT::Periodic::Unattended-Upgrade "1"; +Unattended-Upgrade::Mail "root"; +Unattended-Upgrade::Remove-Unused-Dependencies "true"; +Unattended-Upgrade::Automatic-Reboot "true"; +Unattended-Upgrade::Allowed-Origins { + "${distro_id}:${distro_codename}-security"; + "${distro_id}:${distro_codename}-updates"; +}; PLAY RECAP ********************************************************************* debesis : ok=3 changed=1 unreachable=0 failed=0 ranka : ok=3 changed=1 unreachable=0 failed=0

-CD stands for --check --diff, which tells Ansible not to make any changes, but show what it would do, with full unified diffs. Ansible’s killer feature IMHO. Probably the main reason I why switched from Fabric to Ansible.

Everything looks fine, so we can apply the changes – but let’s be cautious and only do it for only one machine for now

$ ansible-playbook setup.yml --limit debesis PLAY [all] ********************************************************************* TASK [setup] ******************************************************************* ok: [debesis] TASK [install unattended-upgrades] ********************************************* ok: [debesis] TASK [enable unattended-upgrades] ********************************************** changed: [debesis] PLAY RECAP ********************************************************************* debesis : ok=3 changed=1 unreachable=0 failed=0

And we’re done!

$ git add . $ git commit -m "Set up unattended-upgrades"

Ansible has great documentation, if you’d like to learn more.

Categories: FLOSS Project Planets

Import Python: ImportPython Issue 90 - Real-time streaming data pipeline, generators, channels, and more

Planet Python - Fri, 2016-09-16 06:40

Worthy Read
Real-time streaming data pipeline in pythonstreamingMotorway is a real-time data pipeline, much like Apache Storm - but made in Python :-) We use it over at Plecto and we're really happy with it - but we're continously developing it. The reason why we started this project was that we wanted something similar to Storm, but without Zookeeper and the need to take the pipeline down to update the topology.

Working with streaming data: Using the Twitter API to capture tweetstwitterThis tutorial tries to teach event driven programming by making use of streaming API offered by twitter.

Python has come a long way. So has job hunting.sponsorTry Hired and get in front of 4,000+ companies with one application. No more pushy recruiters, no more dead end applications and mismatched companies, Hired puts the power in your hands.

Introduction to Python generatorsgeneratorsIn this guide we 'll cover generators in depth . We 'll talk about how and why to use them , the difference between generator functions and regular functions , the yield keyword , and provide plenty of examples.This guide assumes you have a basic knowledge of Python ( especially regular functions).Throughout this guide we are going to work towards solving a problem .

Python 3.6.0 beta 1 is now available!python3Python 3.6.0b1 is the first of four planned beta releases of Python 3.6, the next major release of Python, and marks the end of the feature development phase for 3.6. There are quite many new features have a look.

Channels adopted as an official Django projectdjangoThe Django team is pleased to announce that the Channels project is now officially part of the Django project, under our new Official Projects program. Channels is the effort to bring WebSockets, long-poll HTTP, and other non-request-response protocol and business logic handling to Django, as part of our ongoing effort to establish what makes a useful web framework in 2016.

Testing dates in DjangotestingDjango makes unit & functional testing easy (especially with WebTest). Tests on routing, permissions, database updates and emails are all straightforward to implement but how do you test dates & time? You might for example want to test regular email notifications.

A Brief Introduction to Django ChannelsdjangoThe idea behind Channels is quite simple. To understand the concept, let’s first walk through an example scenario, let’s see how Channels would process a request.

Episode 74 - Python at Zalandopodcast, communityOpen source has proven its value in many ways over the years. In many companies that value is purely in terms of consuming available projects and platforms. In this episode Zalando describes their recent move to creating and releasing a number of their internal projects as open source and how that has benefited their business. We also discussed how they are leveraging Python and a couple of the libraries that they have published.

Book Contest: Win a Copy of Python 201booksTo win your copy of this book, all you need to do is come up with a comment below highlighting the reason “why you would like to win this book”. Try your luck guys :)

Machine Learning in a yearmachine learningOnly people with masters degrees or Ph.D’s work with machine learning professionally isn't true. The truth is you don’t need much maths to get started with machine learning, and you don’t need a degree to use it professionally. Here is Per Harald Borgen journey. Yes he is using Python.

12 versions of the same algorithm in JavaScript, Python, and GolanguagesI recently had to write nearly the same code in Go and Python on the same day, and I realized I had written it many times before in different languages. But it does point up some interesting language differences. This article explores many different ways to write the same code.


Python Fullstack Developer (M/W) at Paessler AGNürnberg, DeutschlandUnser Technical-Debt-Monster bereitet sich auf die nächste Episode mit uns vor und dieses Mal werden wir es nicht ohne einen Fullstack Magician besiegen. Zusammen mit einem gut balancierten Team wirst Du Dich in tiefe Abgründe begeben und mit neuen Erfahrungen und Fertigkeiten herauskommen.

Upcoming Conference / User Group Meet
PyCon JP 2016
PyCon PL 2016
OSCON Europe 2016
PyCon DE 2016
PyCon CZ 2016

NakedTensor - 53 Stars, 3 ForkBare bottom simplest example of machine learning in TensorFlow.
tensorflow_image_classifier - 15 Stars, 4 ForkTensorFlow Image Classifier Demo by @Sirajology on Youtube
packyou - 10 Stars, 0 ForkImport any python project from github easily
lambdazen - 7 Stars, 2 ForkA better python lambda syntax based on runtime in-memory source rewriting
python-twitter-toolbox - 6 Stars, 1 ForkTwitter Toolbox for Python.
pymail - 3 Stars, 1 Fork:mailbox_with_mail: Command-line email client
export-kobo - 3 Stars, 0 ForkA Python tool to export annotations and highlights from a Kobo SQLite file.
Categories: FLOSS Project Planets

autoconf-archive @ Savannah: Noteworthy changes in release 2016.09.16 [stable]

GNU Planet! - Fri, 2016-09-16 06:12


While updating AX_CODE_COVERAGE, the variable CODE_COVERAGE_LDFLAGS was renamed to CODE_COVERAGE_LIBS.

AX_VALGRIND_CHECK has been extended considerably. In this process. the undocumented internal rule check-valgrind-tool was removed.

It checks if the host cpu supports various instruction set: mmx, sse, popcnt, avx, avx2, etc. If the instruction were supported, it would export HAVE_XXX_INSTRUCTIONS C preprocessor macros and add the instruction set flag to X86_FEATURES_CFLAGS variable. By default it would also set CFLAGS, but this can be disabled by the ACTION-IF-FOUND hook.

AX_WITH_CURSES and AX_WITH_CURSES_* macros have been modernized, in line with PKG_CHECK_MODULES's style. This change breaks the interface, but makes AX_WITH_CURSES compatible and even completely replaceable with the interface of PKG_CHECK_MODULES. Ultimately, these macros should be replaced in the future by calls to PKG_CHECK_MODULES. Variables in AX_WITH_CURSES have been renamed


and for the AX_WITH_CURSES_* macros


such that once the variables have been renamed in configure.ac and Makefile.am, the calls AX_WITH_CURSES and PKG_CHECK_MODULES([CURSES], [ncurses]) are completely interchangeable.

Categories: FLOSS Project Planets

OpenLucius: Module release: 'Pop-up question redirect'

Planet Drupal - Fri, 2016-09-16 05:54

The 'Popup question redirect' module allows Drupal site builders to show website visitors a pop-up in which a question can be asked. If the vistor clicks "Yes" they will be redirected to a given webpage.

Categories: FLOSS Project Planets

Breeze mimetype WIP

Planet KDE - Fri, 2016-09-16 04:54

Here are some results of the breeze-icon mimetype redesign. As you can see we make use of your feedback.

What is still missing? The archive, vector graphic images and virtualbox icons will get some love. After the shapes are finished Alex will work on the colors and gnastyle on the MS Office icons.

Be part of it

git clone:https://github.com/kde/breeze-icons
Categories: FLOSS Project Planets

DrupalCon News: Drupal in Ireland - Project 5

Planet Drupal - Fri, 2016-09-16 04:00

As part of our series of showcasing what Drupal is doing in Ireland, we are happy to bring you another awesome online experience that Drupal has created.  

Project 5 - Dún Laoghaire-Rathdown County Council

Monsoon Consulting worked with Dún Laoghaire-Rathdown County Council to build the next generation of DLR websites and internal intranet platform, using Drupal 7 as the content management system. 

Categories: FLOSS Project Planets

Dirk Eddelbuettel: anytime 0.0.2: Added functionality

Planet Debian - Thu, 2016-09-15 22:28

anytime arrived on CRAN via release 0.0.1 a good two days ago. anytime aims to convert anything in integer, numeric, character, factor, ordered, ... format to POSIXct (or Date) objects.

This new release 0.0.2 adds two new functions to gather conversion formats -- and set new ones. It also fixed a minor build bug, and robustifies a conversion which was seen to be not quite right under some time zones.

The NEWS file summarises the release:

Changes in Rcpp version 0.0.2 (2016-09-15)
  • Refactored to use a simple class wrapped around two vector with (string) formats and locales; this allow for adding formats; also adds accessor for formats (#4, closes #1 and #3).

  • New function addFormats() and getFormats().

  • Relaxed one tests which showed problems on some platforms.

  • Added as.POSIXlt() step to anydate() ensuring all POSIXlt components are set (#6 fixing #5).

Courtesy of CRANberries, there is a comparison to the previous release. More information is on the anytime page.

For questions or comments use the issue tracker off the GitHub repo.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Danny Englander: Drupal 8 Theming: How to Define Variables in Twig Templates to Use as Custom HTML Classes

Planet Drupal - Thu, 2016-09-15 21:06

For the past seven months, I've been designing, building, and theming with Drupal 8 for my own side project. Along the way, I've seen huge advancements in these realms versus what was possible with Drupal 7. One particular module I've been working with is Entity Construction Kit AKA "ECK".

ECK provides a way to create freestanding entities, bundles, and fields independent of traditional means via nodes, terms, users, etc... Within my project, I've implemented ECK fields that offer content editors options for layouts, colors, element padding, and so on.

Discover the array path with kint

I'll use the case of an accordion group entity that I've designed. One such field within this entity bundle is called "Slice margin." This field allows a content editor to set the top and bottom margins within a specific element by choosing values from a select list. Examining this field with kint, the value of the selected item's array path is:

$var['entity']['field_slice_margin']['0']['#markup'] Check for the field and define the variable

Now with the array path value determined, a variable in the accordion entity can be set in a Twig template:

{% if entity.field_slice_margin|render %} {% set margin_settings = 'margin__' ~ entity.field_slice_margin['0']['#markup'] %} {% endif %}

In the above code, the code checks to be sure the field is being rendered and then a variable is set for the value. In addition, margin__ is concatenated (~) as a class prefix.

Set a class using the variable

The block of code above does not do much on its own, so lets put the new variable to use in the template. For that, Drupal 8's Twig set_classes comes in handy.

{% set classes = [ 'eck', 'eck__section', eck__bundle-accordion, 'margin__default', margin_settings|clean_class, ] %}

Note in the above, the new variable is output using the Drupal Twig clean_class method, margin_settings|clean_class. clean_class cleans up any spaces and underscores in a string and converts those to standard dashes for use as a standard html class name.

Output the HTML with the classes

Now that this is done, the class can be rendered in an HTML element in the same twig template using the attributes.addClass(classes) method:

<section {{ attributes.addClass(classes) }}> <ul class="eck__collapsible collapsible popout" data-collapsible="accordion"> {{ entity.field_accordion_reference|field_value }} </ul> </section> Rendered HTML

Finally, this will render in HTML as:

<section class="eck eck__section margin__default margin__flush-above-below eck__bundle-accordion">

The key class rendered here from entity.field_slice_margin is margin__flush-above-below, that's the specific value the content editor chose from the select list on the node edit page. Now this and other values from this field can be used in theming like so using Sass:

.margin { &__default { margin-bottom: 70px; } &__flush-above { margin-top: 0; } &__flush-below { margin-bottom: 0; } &__flush-above-below { margin-top: 0; margin-bottom: 0; } } Summary

As you can see, Twig is really powerful in Drupal 8. No preprocess functions were used here which really helps streamline theming. We stayed within one Twig template for all of the functions and code above. A lot of possibilities here to give content creators more control over layout, colors, custom classes, and more.

  • Drupal
  • Twig
  • Theming
  • Drupal Planet
  • Drupal 8
Categories: FLOSS Project Planets

Python Sweetness: Deploying modern Python apps to ancient infrastructure with pkgsrc

Planet Python - Thu, 2016-09-15 19:27

In recent adventures for work, I once again found myself on a team with an interesting and relatively new-to-me problem that I hadn’t been forced to consider quite so much before.

This team is responsible for supplying a variety of web apps built on a modern stack (mostly Celery, Django, nginx and Redis), but have almost no control over the infrastructure on which it runs, and boy, is some of that infrastructure old and stinky.

We have no root access to these servers, most software configuration requires a ticket with a lead time of 48 hours plus, and the watchful eyes of a crusty old administrator and obtuse change management process. The machines are so old that many are still running on real hardware, and those that are VMs still run some ancient variety of Red Hat Linux, with, if we’re lucky, Python 2.4 installed.

Naturally the Python world moves quickly, and, as anyone who ever worked on real code behind closed doors (as opposed to the joyous freshness of hobby projects that can run the latest and greatest) can attest, the Python community’s attitude towards compatibility is more or less atrocious. The thought of deploying a modern Django app using Python 2.4 is simply out of the question, and if it weren’t a problem with Python, then outdated core libraries (such as OpenSSL) would soon become a problem.

This won’t be about compatibility (oh my, could I spend words on that), but instead how to sanely solve the unfortunate situation of deploying modern apps to ancient infrastructure, when you have no control over the base Linux system, and when that Linux system sucks.

A Bad Situation

Naturally if the base system is unusable you are going to have to replace it, or say, rewrite your shiny new application in Perl 4. But how? On arrival, I was greeted with selection of huge shell scripts running under Jenkins that manually fetched individual source tarballs for Python, Nginx, OpenSSL, and every other dependent library from the Internet, ./configure --prefix=/home/ourapp, make install, and so on.

The result was effectively a miniature Linux distribution hand-cobbled together as and when needs arose. Should a new feature of an application require, say, the Python ImageMagick package, one could easily expect to spend half a day editing shell scripts in a massively wasteful loop retrying the build repeatedly, iteratively updating it to include base library dependencies until everything built, and the desired Python binding could be installed.

That wasn’t the worst part. The cruddy old base system was building under Docker, using a slightly more modern version of Red Hat Linux than installed in production. Some of the builds were succeeding only because a newer-than-available library dependency was satisfied by the Docker image, and so errors like below were commonplace:

libpcre.so.1: cannot open shared object file: No such file or directory

The solution to these library problems in the usual case was to simply copy the versions installed in the Docker container, resulting in a final hand-cobbled base system that was a mixture of outdated self-built programs that relied on a random smattering of libraries sourced only from some untrusted third party Docker image.

In other yet more horrific cases, the problem was solved by symlinking older versions of a library to a different filename, explicitly violating the library author’s signal that binary compatibility had changed, and provoking some delightfully impossible to debug unpredictable crash sometime deep into the future.

In a setup like this, asking a simple question like “do our apps have any security vulnerabilities?” becomes an impossible half-day’s work to answer. “Let me just check the package manage.. oh, we don’t have one of those. Actually we do, but there are two package managers to consult depending on which libary is involved, one of which we have no control over, and a huge ugly shell script to manually audit.”

Since this home-grown combobulation of binaries depended on libraries that weren’t installed in system paths, it wasn’t even possible to run them without first sourcing a shell script to set LD_LIBRARY_PATH so the runtime linker could find the libraries in our nonstandard location.

In short, the situation was a total mess.

Enter pkgsrc

The pkgsrc project is a huge collection of package descriptions for modern software releases, and forms the basis for NetBSD’s equivalent to Linux’s apt-get or yum commands. For Mac users it is most similar to MacPorts or Homebrew, but unlike those systems it not only targets NetBSD but also Linux, OS X, and a bunch of lesser loved operating systems. To get an idea of the scope of pkgsrc you can browse the collection.

Unlike our mess that came before it, pkgsrc is actively maintained by a large community, the program and library combinations in its tree can be relied on to be tested and intercompatible, with quarterly stable releases, and it even comes with tooling to scan for security vulnerabilities.

Best of all, it does not require root to install. After a relatively short bootstrap process completes, a comprehensive modern UNIX system can be installed into a non-root filesystem location on any supported OS, such as our application’s home directory on its ancient production machine.

Since pkgsrc is a real grown up solution, we can ask it to build from source, say, ImageMagick, and it will automatically figure out all the dependencies required to build ImageMagick, and recursively build them in a supported configuration that is in use by many people.

The level of foresight on the maintainers’ part is quite amazing. In order to address accidentally picking up random dependencies from our Docker image, pkgsrc has a feature to allow on a per-dependency basis preferring any system-installed version if available (the default), preferring a fresh pkgsrc-built and managed version, or simply to always prefer building fresh up-to-date tools and libaries from the tree, which is exactly what we want, as it produces a base system that depends only on the installed Linux distribution for its kernel and C library, which are very stable dependencies.

For icing on the cake, packages build by default with useful compile-time features such as consistent use of the ELF rpath header, allowing programs to run in the target environment without requiring LD_LIBRARY_PATH to be set at all, since the built binaries contain internal annotations telling the runtime linker where its library dependencies are to be found.

Linker flags enabling this feature neatly percolate down through the Python build process, leaving a configuration of distutils that knows how to build Python extension modules with the same suite of flags, without any further deviation from the usual “/home/ourapp/root/bin/pip install numpy”.

pkgsrc Crash Course

All that is required to get a build up and running is a compiler suite (e.g. the build-essential metapackage on Debian), and a copy of the pkgsrc tree tarball.

wget https://ftp.netbsd.org/pub/pkgsrc/pkgsrc-2016Q1/pkgsrc-2016Q1.tar.bz2 tar jxvf pkgsrc-2016Q1.tar.bz2

Once the tarball is unpacked, a short bootstrap shell script must be run in order to build a copy of bmake, the make implementation used by the tree along with pkgutils, a small binary package administration tool akin to rpm or dpkg.

SH=/bin/bash pkgsrc/bootstrap/bootstrap \ --unprivileged \ --prefix /home/ourapp/root \ --make-jobs 4

Important note about –prefix: the supplied directory must be one that can be written to by your application’s user account in all its target environments. For example, if your staging and production environment have different usernames, to avoid having to rebuild pkgsrc with a different configuration for both targets, it may be wise to simply update every environment to have the same username. Where this isn’t possible, the $ORIGIN trick described later may be interesting.

Configuration of the build is done by way of <prefix>/etc/mk.conf, where global bmake variables are set. The defaults require no editing in the usual case, but for our purposes, we want to enable that tasty feature of force-building all dependencies from scratch:

echo PREFER_PKGSRC = yes >> /home/ourapp/root/etc/mk.conf

For demonstration purposes, and to show just how easy it is to get a perfect result, let’s also tell the nginx package to enable its uwsgi protocol module when it gets built:

echo PKG_OPTIONS.nginx = uwsgi >> /home/ourapp/root/etc/mk.conf

Now all that’s left is to build some useful software! That is done by running bmake install from inside the software’s directory inside the tree. Of course since bmake is not installed globally, we must use an absolute path:

cd pkgsrc/www/nginx /home/ourapp/root/bin/bmake install

On a fast machine, by the time lunch is finished you should return to an up-to-date build of Nginx installed in /home/ourapp/root/bin, with all dependencies (such as PCRE and OpenSSL) correctly built and linked so that invoking /home/ourapp/root/bin/nginx is all required to start a vulnerability-free correctly compiled Nginx.

Best of all, the binaries in /home/ourapp/root can reasonably be expected to run on any version of Linux available in the past 15 years: simply extract a tarball of the build to the target machine.

Putting It All Together

Below is the final Jenkins script that builds the base system for one of our applications. The final /home/ourapp/root is quite huge, coming in close to 500mb, but that includes every tool required during build, along with every library we never knew we depended on built from a recent supported version.

The script expects to run under Docker, with a --volume mounted from /workspace that points back to the Jenkins job’s workspace directory.

#!/bin/bash -ex INSTALL_PREFIX=/home/ourapp/root mkdir -p /workspace/output trap 'chown -R $(stat -c "%u:%g" /workspace) /workspace/output' EXIT curl -sS https://ftp.netbsd.org/pub/pkgsrc/pkgsrc-2016Q1/pkgsrc-2016Q1.tar.bz2 \ | tar -jx export SH=/bin/bash pkgsrc/bootstrap/bootstrap \ --unprivileged \ --prefix $INSTALL_PREFIX \ --make-jobs 4 cat >> $INSTALL_PREFIX/etc/mk.conf <<-EOF PREFER_PKGSRC = yes PKG_OPTIONS.nginx = uwsgi EOF $INSTALL_PREFIX/bin/bmake -C pkgsrc/lang/python27 install $INSTALL_PREFIX/bin/bmake -C pkgsrc/devel/py-pip install $INSTALL_PREFIX/bin/bmake -C pkgsrc/devel/py-readline install $INSTALL_PREFIX/bin/bmake -C pkgsrc/databases/py-ldap install $INSTALL_PREFIX/bin/bmake -C pkgsrc/databases/redis install $INSTALL_PREFIX/bin/bmake -C pkgsrc/databases/py-sqlite3 install $INSTALL_PREFIX/bin/bmake -C pkgsrc/databases/py-mysqldb install $INSTALL_PREFIX/bin/bmake -C pkgsrc/www/py-uwsgi install $INSTALL_PREFIX/bin/bmake -C pkgsrc/www/nginx install $INSTALL_PREFIX/bin/bmake -C pkgsrc/textproc/libxml2 install ln -s $INSTALL_PREFIX/bin/python{2.7,} ln -s $INSTALL_PREFIX/bin/pip{2.7,} ln -s $INSTALL_PREFIX/bin/uwsgi{-2.7,} tar czf /workspace/output/ourapp-system.tar.gz $INSTALL_PREFIX

The Jenkins job output is an ourapp-system.tar.gz to be extracted on the target machine.


Of course nothing comes without downsides. The major risk with pkgsrc is the sheer complexity of the makefiles that implement it. If your use case isn’t covered by the provided tree, then significant engineering and forward maintenance may be required to modify it and keep your modifications up to date.

An example of such a modification would be if your project needs an exact library version that differs from that available in the pkgsrc release tree. Thankfully such needs are rare, but it was overwhelmingly my primary concern while evaluating pkgsrc for this solution. Another complexity scare story is that when pkgsrc goes wrong, boy, does it generate some inscrutable errors.

I stopped worrying when I realized these problems are equally as hard to solve with apt-get or yum, C compilers are quite capable of producing equally as weird errors, and after a coworker helped me realize pkgsrc need not be a 100% solution: a broken package could simply be manually compiled just as it was in the past, with the remainder handled automatically.

Despite saying that, one ongoing minor source of worry is the size of the pkgsrc user community relative to a typical Linux package manager’s community. Stack Overflow is much less likely to contain an answer for the exact error you’re encountering.

I’m reaffirmed in the knowledge that the small community that does exist around NetBSD is both extremely helpful and in general vastly more clueful on average than the typical user support communities available in mainstream Linux.

Cutting Down On Build Size

One area that did not concern me so much is the size of the built tree. 500mb is nothing for us, but it is easy to imagine cases where this could be troublesome. Thankfully there are plenty of options to solve it, such as:

  • More granular PREFER_PKGSRC settings to avoid rebuilding tools like perl that really don’t need to be rebuilt. In our case, it was simpler to just take a slightly bloated tarball full of supported but useless-at-runtime tools than it was to try finding the minimum set required.

  • Use of the pkg_delete command to remove tools that aren’t required after build. I just couldn’t be bothered with this, it’s ongoing maintenance for maintenance’ sake.

  • Brute force rm -rf of useless directories like locale files, /home/ourapp/root/share/doc, share/man, etc. Again the cost of the fat tarball must be weighed against the unpredictable man hours involved in rebuilding everything in an emergency since an important file was accidentally deleted, and a new application release triggers latent breakage.

The $ORIGIN Trick

There is one final avenue I itched to explore, but could not find the personal motivation nor justification for business hours to chase. That is the use of the runtime linker’s $ORIGIN feature, which allows the ELF rpath header to be set relative to the location of the binary containing the header. In other words, it allows for (almost) fully relocateable builds, where the filesystem path of the build tree is no longer a fixed variable.

A working build that used $ORIGIN would have allowed our group to have a single pkgsrc build for all our applications (which run with a variety of uncontrollable home directory paths), with the minor downside that a variety of installed applications might randomly fail since they hard-coded paths to their configuration files (and suchlike) using the build-time supplied --prefix.

Again, this solution was not fully explored, but it seem in principle quite possible to implement with careful testing of the produced tree, and explicit configuration of paths on the command-line and in configuration files for every program (such as Nginx).

After building a tree, one need simply use the chrpath utility to update the ELF headers for every installed program and library in the build, so that it includes a rpath relative to the program or library’s location.

For example, /home/ourapp/root/bin/nginx normally builds with an rpath of /home/ourapp/root/lib. Using $ORIGIN, that would change to $ORIGIN/../lib. Now the runtime linker knows how to find PCRE and OpenSSL independent of the absolute filesystem prefix.

I never finished testing this approach, but in case the reader finds it desirable, when run from inside --prefix, the following short script takes care to manage setting $ORIGIN for every binary correctly regardless of its subpath depth under --prefix:

import os import subprocess def chrpath(path): depth = path.count('/') - 1 libdir = '$ORIGIN/%slib' % ('../' * depth,) subprocess.Popen(['chrpath', path, '-cr', libdir]).wait() for dirpath, dirnames, filenames in os.walk('.'): for filename in filenames: path = os.path.join(dirpath, filename) if path.endswith('.so') or 'bin/' in path: chrpath(path)

I knew this would take forever to write up, I started thinking about it a week ago, and finally spent 3 hours on it tonight. It is in response to, and hopefully neatly complements Mahmoud Hashemi’s post Python Packaging at PayPal.

Categories: FLOSS Project Planets

Mike Driscoll: wxPython Cookbook – Hard Cover Editions Available

Planet Python - Thu, 2016-09-15 18:15

I’ve been toying with the idea of doing a hard cover version of my books for a long time so I thought I might try it out with the Cookbook. I went ahead and ran the numbers using my printer on demand resources and discovered that a regular hard cover isn’t crazy expensive. So I have added a new perk priced at $35 for a hard cover version of the book.

But wait! There’s more! Since the cookbook will have so many screenshots and code examples, it occurred to me that it might be fun to offer a color version of the book. That way you can see the screenshots in full color and you’ll also be able to view the syntax highlighting in the code examples too! Unfortunately, printing in color is crazy expensive. But if you are interested in that option, it is available at $100.

All versions of the book will be signed. If there’s a lot of interest in the black-and-white hard cover, I’ll increase the number of items available for that one.

To get either of these editions or the regular paperback, you will need to pledge on Kickstarter.

Categories: FLOSS Project Planets

Cheeky Monkey Media: More than Websites

Planet Drupal - Thu, 2016-09-15 15:17
More than Websites chris Thu, 09/15/2016 - 19:17

We have established ourselves as experts in the Drupal and Web development sphere. However, few people are aware that we bring much more to the table. We are a diverse team of creative thinkers, designers, and marketers with the single-minded desire to make your project a success. It takes more than just building a Website these days, so it's important to us that our clients know that engaging with Cheeky Monkey Media can be a full spectrum experience. 

To help illustrate our hidden talents, we put together this quick little reel highlighting just some of the “Other things we do”, things like graphic design, marketing, and SEO.

Categories: FLOSS Project Planets
Syndicate content