FLOSS Project Planets

Friday Free Software Directory IRC meetup: July 28th starting at 12:00 p.m. EDT/16:00 UTC

FSF Blogs - Tue, 2017-07-25 12:15

Participate in supporting the Directory by adding new entries and updating existing ones. We will be on IRC in the #fsf channel on irc.freenode.org.

Tens of thousands of people visit directory.fsf.org each month to discover free software. Each entry in the Directory contains a wealth of useful information, from basic category and descriptions, to providing detailed info about version control, IRC channels, documentation, and licensing info that has been carefully checked by FSF staff and trained volunteers.

While the Directory has been and continues to be a great resource to the world for over a decade now, it has the potential to be a resource of even greater value. But it needs your help!

It was on this day in 1983 that NASA launched the Telstar 3A into geosynchronous orbit at 96 degrees W. With this in mind the Directory is going to look at the tangentially related category of astronomy software. Stellar examples of such software allow for the simulation of astral bodies, and there is also the well-known Gnuastro, an official GNU package that enables the manipulation and analysis of astronomical data. We hope to see you there on Friday when we blast off for the stars!

If you are eager to help, and you can't wait or are simply unable to make it onto IRC on Friday, our participation guide will provide you with all the information you need to get started on helping the Directory today! There are also weekly Directory Meeting pages that everyone is welcome to contribute to before, during, and after each meeting.

Categories: FLOSS Project Planets

Xeno Media: WordPress coding standards for the Drupal developer

Planet Drupal - Tue, 2017-07-25 11:17

If you've been doing Drupal development for any amount of time, chances are that you have installed the Drupal Code to help you write clean, compliant code. Coder allows you to check your Drupal code against the Drupal coding standards and other best practices using PHP_CodeSniffer.  It can be configured to work in your IDE, and also works on the command line.

Writing code according to standards helps avoid common errors, and helps teams understand the code faster.

I installed Coder using Composer per the well written instructions.  Using this method installs it globally, so I can use it on all of my projects, and installs all the dependencies, including PHP_CodeSniffer.

I recently was tasked with working on a Wordpress site, and I started looking into the WordPress Coding Standards.  My setup didn't jive with the standard installation method since I already had PHP_CodeSniffer installed globally using composer.  I had to do a little digging to add these additional standards to my already installed setup.

Here is a quick recap on how to install Coder using composer.

Install Coder composer global require drupal/coder

To make the commands available globally, add this line to your .~/bash_profile, and that it is sourced (or restarted your terminal).

# Composer recommended PATH export PATH="$PATH:$HOME/.composer/vendor/bin"

Tell phpcs where the Drupal and DrupalPractice standards are located:

phpcs --config-set installed_paths ~/.composer/vendor/drupal/coder/coder_sniffer

Verify it worked with:

phpcs -i

You should see:

The installed coding standards are MySource, PEAR, PHPCS, PSR1, PSR2, Squiz, Zend, Drupal, and DrupalPractice

You can now navigate to your Drupal project and run the following command to use:

phpcs --standard=Drupal file.nameInstall Wordpress Coding Standards

Thanks to some help I found in the issue queue, here are the steps to install the Wordpress Coding Standards globally using composer.

composer global require wp-coding-standards/wpcs:dev-master

Again, to make these commands available globally, make sure you have this line in your ~/.bash_profile, and that it is sourced (or restarted your terminal).

# Composer recommended PATH export PATH="$PATH:$HOME/.composer/vendor/bin"

Like we did with Drupal, we need to tell phpcs where the Wordpress standards are located. We use the same installed_paths configuration set, and use a comma to list both the Drupal and Wordpress paths.

phpcs --config-set installed_paths $HOME/.composer/vendor/drupal/coder/coder_sniffer,$HOME/.composer/vendor/wp-coding-standards/wpcs

Verify it worked with:

phpcs -i

You should now see:

The installed coding standards are MySource, PEAR, PHPCS, PSR1, PSR2, Squiz, Zend, Drupal, DrupalPractice, WordPress, WordPress-Core, WordPress-Docs, WordPress-Extra and WordPress-VIP

You can now navigate to your Wordpress project and run the following command to use:

phpcs --standard=Wordpress file.nameAdd aliases

If you've worked with me, or read my posts before, you know I love aliases. They streamline your process and help make you more productive. Add these aliases into your .bash_profile, .bashrc, or wherever you keep your aliases, and source it, or restart your terminal.

alias drupalcs="phpcs --standard=Drupal --extensions='php,module,inc,install,test,profile,theme,css,info,txt,md'" alias wpcs="phpcs --standard=Wordpress"

After this you can simply type drupalcs folder_name or wpcs file.name and start writing better code!

Acknowledgements

Thanks to Micheal Porter, Albert Jankowski, and Mike Acklin for the technical review of this article, and to all the maintainers!

Photo by Ilya Pavlov on Unsplash

Categories: FLOSS Project Planets

Russell Coker: Forking Mon and DKIM with Mailing Lists

Planet Debian - Tue, 2017-07-25 10:50

I have forked the “Mon” network/server monitoring system. Here is a link to the new project page [1]. There hasn’t been an upstream release since 2010 and I think we need more frequent releases than that. I plan to merge as many useful monitoring scripts as possible and support them well. All Perl scripts will use strict and use other best practices.

The first release of etbe-mon is essentially the same as the last release of the mon package in Debian. This is because I started work on the Debian package (almost all the systems I want to monitor run Debian) and as I had been accepted as a co-maintainer of the Debian package I put all my patches into Debian.

It’s probably not a common practice for someone to fork upstream of a package soon after becoming a comaintainer of the Debian package. But I believe that this is in the best interests of the users. I presume that there are other collections of patches out there and I hope to merge them so that everyone can get the benefits of features and bug fixes that have been separate due to a lack of upstream releases.

Last time I checked mon wasn’t in Fedora. I believe that mon has some unique features for simple monitoring that would be of benefit to Fedora users and would like to work with anyone who wants to maintain the package for Fedora. I am also interested in working with any other distributions of Linux and with non-Linux systems.

While setting up the mailing list for etbemon I wrote an article about DKIM and mailing lists (primarily Mailman) [2]. This explains how to setup Mailman for correct operation with DKIM and also why that seems to be the only viable option.

Related posts:

  1. DKIM and Mailing Lists Currently we have a problem with the Debian list server...
  2. An Update on DKIM Signing and SE Linux Policy In my previous post about DKIM [1] I forgot to...
  3. Installing DKIM and Postfix in Debian I have just installed Domain Key Identified Mail (DKIM) [1]...
Categories: FLOSS Project Planets

Stefan Behnel: What's new in Cython 0.26?

Planet Python - Tue, 2017-07-25 10:33

Cython 0.26 has finally been released and it comes with some big and several smaller new features, contributed by quite a number of non-core developers.

Probably the biggest addition, definitely codewise, is support for Pythran as a backend for NumPy array expressions, contributed by Adrien Guinet. Pythran understands many usage patterns for NumPy, including array expressions and some methods, and can directly be used from within Cython to compile NumPy using code by setting the directive np_pythran=True. Thus, if Pythran is available at compile time, users can avoid writing manual loops and instead often just use NumPy in the same way as they would from Python. Note that this does not currently work generically with Cython's memoryviews, so you need to declare the specific numpy.ndarray[] types in order to benefit from the translation. Also, this requires the C++ mode in Cython as Pythran generates C++ code.

The other major new feature in this release, and likely with an even wider impact, is pickling support for cdef classes (a.k.a. extension types). This is enabled by default for all classes with Python compatible attribute types, which explicitly excludes pointers and unions. Classes with struct type attributes are also excluded for practical reasons (such as a high code overhead), but can be enabled to support pickling with the class decorator @cython.auto_pickle(True). This was a long-standing feature for which users previously had to implement the pickle protocol themselves. Since Cython has all information about the extension type and its attributes, however, there was no technical reason why it can't also generate the support for pickling them, and it now does.

As always, there are several new optimisations including speed-ups for abs(complex), comparing strings and dispatching to specialised function implementations based on fused types arguments. Particularly interesting might be the faster GIL re-entry with the directive fast_gil=True. It tries to remember the current GIL lock state in the fast thread local storage and avoids costly calls into the thread and GIL handling APIs if possible, even when calling across multiple Cython modules.

A slightly controversial change now hides the C code lines from tracebacks by default. Cython exceptions used to show the failing line number of the generated C code in addition to the Cython module code line, now only the latter is shown, like in Python code. On the one hand, this removes clutter that is irrelevant for most users. On the other hand, it hides information that could help developers debug failures from bug reports. For debugging purposes, the re-inclusion of C code lines can now be enabled with a runtime setting as follows:

import cython_runtime cython_runtime.cline_in_traceback=True
Categories: FLOSS Project Planets

Stefan Behnel: What's new in Cython 0.25?

Planet Python - Tue, 2017-07-25 10:33

Cython 0.25 has been released in October 2016, so here's a quick writeup of the most relevant new features in this release.

My personal favourites are the call optimisations. Victor Stinner has done a great job throughout the year to optimise and benchmark different parts of CPython, and one of the things he came up with was a faster way to process function calls internally. For this, he added a new calling convention, METH_FASTCALL, which avoids tuple creation by passing positional arguments as a C array. I've added support for this to Cython and also ported a part of the CPython implementation to speed up calls to Python functions and PyCFunction functions from Cython code.

Further optimisations speed up f-string formatting, cython.inline() and some Python long integer operations (now also in Py2.7 and not only Py3).

The next big feature, one that has been on our list essentially forever, was finally added in 0.25. If you declare a special attribute __dict__ on a cdef class (a.k.a. extension type), if will have an instance dict that allows setting arbitrary attributes on the objects. Otherwise, extension types are limited to the declared attributes and trying to access undeclared ones would result in an AttributeError.

The C++ integration has received several minor improvements, including support for calling subclass methods from C++ classes implemented in Cython code, iterating over std::string with Cython's for-loop, typedef members in class declarations, or the typeid operator.

And a final little goodie is redundant declarations for pi and e in libc.math, which makes from libc cimport math pretty much a drop-in replacement for Python's import math.

Categories: FLOSS Project Planets

Akademy BoFs

Planet KDE - Tue, 2017-07-25 10:28

Here at Akademy in Almería, we have moved from the conference portion — two days of talks at the University — to the hacking week portion. The conference days were very busy; most of the talks were recorded and those recordings will be up when they’re done post-processing. I haven’t heard a date for that yet.

Cake is a theme. I don’t know who bakes them or where they come from, but the shout on IRC that there is fresh-baked, still-warm cake and coffee in the hall near the team room is an event to drop everything for.

I missed BoFs yesterday that I wanted to go to — I guess I was wrapped up in hacking and talking to people outside of the scheduled activities. Today I spotted a Calamares BoF — not one I scheduled, mind, but apparently people think I should be doing more work.

At the end of each hack-week day, there is a BoF wrap-up for those attendees who could not be everywhere at once, and also for people outside of Akademy who want to know what has been worked on. Yesterday’s wrap-up was lots of “we discussed this-and-that”, I imagine today is going to start moving into “we built something”. So, for everyone at Akademy: be in the big hall downstairs at 18:00; for others, catch the video later (yay, shameless self-promotion!)

Categories: FLOSS Project Planets

DrupalCon News: Reserve your room for DrupalCon Vienna

Planet Drupal - Tue, 2017-07-25 10:05

For DrupalCon Vienna, our partner hotel, Courtyard Vienna Prater, is located in the Trabrennstraße area, where you can explore St. Stephen's Cathedral and Vienna's famous Prater park. And, the hotels we chose are perfect hubs for connecting you to a rewarding DrupalCon experience.

The fun is where the Drupalers are. Stay with us at a partner hotel to network and socialize after sessions end.

Categories: FLOSS Project Planets

InternetDevels: How Drupal 8 saves time & money, or a few words about backwards compatibility

Planet Drupal - Tue, 2017-07-25 08:50

Migration to Drupal 8 will save your time, effort and money in the future. It’s a fact! Discover
the great news about easy upgrades and backwards compatibility.

Read more
Categories: FLOSS Project Planets

Bug parity achieved

Planet KDE - Tue, 2017-07-25 06:54

Riddell: “Hey, Ade, don’t touch your quassel for a minute, ok? Right, switch to a different channel.” Me: “Sure” Riddell: “OK, see the Quassel notification icon?” Me: “Yeah, it’s throbbing.”

He walks over, checks that the notification icon is throbbing in the systray, and that I’ve got a konsole window at hand. “So, run top.” I switch windows, type “top” and hit enter.

Silence. “Um, what operating system is this?”

So Plasma 5 on FreeBSD looks sufficiently indistinguishable from KDE Neon, that it can fool even the conoisseur. But top(1) is different enough. This makes me really happy, since it shows that packaging vanilla upstream KDE software is the right thing to do for FreeBSD. Even better, the KDE Neon bug Riddell was trying to illustrate to me, is also present on FreeBSD with Plasma 5 on Intel graphics (although I use the scfb driver for now). Achieving bug parity is quite a milestone.

Categories: FLOSS Project Planets

Plasma’s Vision

Planet KDE - Tue, 2017-07-25 06:52

Plasma — Durable, Usable, Elegant.Over the past weeks, we (KDE’s Plasma team) have distilled the reasons why we do what we do, and what we want to achieve into a vision statement. In this article, I’d like to present Plasma’s vision and explain a bit what is behind it. Let’s start with the statement itself, though:

Plasma is a cross-device work environment by the KDE Community where trust is put on the user’s capacity to best define her own workflow and preferences.

Plasma is simple by default, a clean work area for real-world usage which intends to stay out of your way.
Plasma is powerful when needed, enabling the user to create the workflow that makes her more effective to complete her tasks.

Plasma never dictates the user’s needs, it only strives to solve them. Plasma never defines what the user is allowed to do, it only ensures that she can.

Our motivation is to enable actual work to happen, across devices, across different platforms, using any application needed.

We build to be durable, we create to be usable, we design to be elegant.

I’ve marked a few bits which are especially important in a bold font, let’s get into a bit more detail:

Cross-device — Plasma is a work environment for different classes of devices, it adapts to the form-factor and offers a user interface which is suitable for the device’s characteristics (input methods such as touchscreen, mouse, keyboard) and constraints (screen size, memory and CPU capabilties, etc.).

Define the workflow — Plasma is a flexible tool that can be set up as the user wishes and needs to make her more effective, to get the job done. Plasma is not a purpose in itself, it rather enables and gets out of the way. This isn’t to say that we’ll introduce every single option one can think of, but we strive to serve many users’ use cases.

Simple by default means that Plasma is self-explanatory to new users, and that it offers a clean and sober interface in its default state. We don’t want to overwhelm the user, but present a serene and friendly environment.

Powerful when needed on the other hand means that under the hood, Plasma offers tremendous power that allow to get almost any job done efficiently and without flailing.

We build to be durable, we create to be usable, we design to be elegant. — The end result is a stable workspace that the user can trust, that is friendly and easy to use, and that is beautiful and elegant in how it works.

Categories: FLOSS Project Planets

Qt 3D Short Presentation

Planet KDE - Tue, 2017-07-25 06:04

Qt now provides a new module named Qt 3D. In this very short talk Giuseppe D’Angelo introduces some of the design ideas behind Qt 3D, discuss its use cases, and shows how simple it is to get 3D content in an application when using Qt 3D APIs.…

The post Qt 3D Short Presentation appeared first on KDAB.

Categories: FLOSS Project Planets

PyCharm: Interview: Paul Craven on Python Gaming and Teaching

Planet Python - Tue, 2017-07-25 05:38

Writing games in Python is fun, so how about using it to teach computer programming? Paul Craven is both a professor and the creator of Arcade, a 2d game library for Python. He’s doing a webinar with us next week, so we talked to him about teaching Python, using Python 3 type hints, why another Python game library, and how IDEs fit into teaching.

Thanks a bunch for doing the webinar next week. First, can you tell us a little bit about yourself?

I worked in the IT industry for 15 years before switching to teaching at Simpson College, a small 4-year college in Iowa. My main interest has been getting first-time programmers to realize that programming can be fun. That moment when a student cheers out loud because they finally figured out how to get sprites to move correctly on the screen? It is a beautiful thing to see.

You teach programming, and you created the Arcade game library to help. Before talking about Arcade, can you explain the motivation behind having a framework that you can teach?

Teaching is like engineering. Each semester you work to improve how you teach students. I had been using the Pygame library. But I wanted a library that I could improve on based on what I saw from students. For example:

Function and parameter names that students intuitively understand. Each year I had to teach them “set_mode” opens a window. Why not just have the function called “open_window”?
Support for functions students which ask for. In Pygame drawing ellipses with thick borders always had a moire pattern because of a bug in the library. And you can’t tilt ellipses when you draw them. Every year I have a student that wants to draw a football. And each year they were frustrated that it looked awful. I wanted a library where it just worked.
Students would download a graphic for their sprite. But it would be too large and there was no easy way to scale the image. That always resulted in hours of wasted time explaining how to do the scaling. What if the library just supported scaling?

After a while I collected such a long list of things like that, I decided to create a game library that where I wouldn’t have to teach “around” these issues.

Beyond using it for teaching, can you talk a bit about Arcade? What is it, how is it different, who might want to use it?

Arcade is great for sprite-based 2D games. It is simple and easy to get started with the library. There is no need to learn a complex framework. If you’ve wanted to create a game for fun, but Kivy, Unity, Unreal would just take more time to learn than what you’ve got, Arcade is a better choice. If you want to quickly create a scientific visualization without a lot of overhead code, Arcade can help you there too.

Arcade uses OpenGL and Pyglet. With OpenGL acceleration, it can draw a lot of sprites fast.

I use Arcade for my PyCharm tutorials and Arcade’s Python 3.6 type hinting is a big reason. Can you talk about your experience, as a library author and teacher, with type hinting?

New programmers often get confused when calling functions and methods. What data does it expect? And when the program doesn’t run, the students aren’t yet experts in reading stack traces. So they are stuck with a 500 line program that doesn’t work and they don’t know why. Frustrating.

Type hinting can sometimes tell students that they are passing unexpected data to the function. It does this before they run the program. Before they’ve even moved to writing the next line of code. It seems trivial, but it’s not. I found students able to create better, more complex programs because PyCharm and type hinting kept them from that error and allowed them to move on.

You also use PyCharm CE with your students. What’s been your experience having beginners start with an IDE?

I’ve taught students with an IDE and without an IDE. The biggest advantage is how the IDE can help students catch errors early. PyCharm’s built-in PEP-8 checking is huge. Also, built in spell-checking! Imagine trying to read program comments written without a spell-checker from today’s student. Students come up with some interesting ways to spell words.

Categories: FLOSS Project Planets

Clang Tidy, part 2: Integrate qmake and other build systems using Bear

Planet KDE - Tue, 2017-07-25 05:20
Introduction

This article is part of a blog series about Clang Tidy. In the previous article we learned about the general usage of Clang Tidy to automatically refactor source code for projects using the CMake build system. In this particular episode we’ll discuss using Clang Tooling on projects using different build systems with the help of Bear.

Motivation: So you want to use Clang Tooling on your project — but what if your particular project of interest is using …

The post Clang Tidy, part 2: Integrate qmake and other build systems using Bear appeared first on KDAB.

Categories: FLOSS Project Planets

ARREA-Systems: Usage of ClamAV in Drupal 8

Planet Drupal - Tue, 2017-07-25 05:17
Usage of ClamAV in Drupal 8 Tue, 07/25/2017 - 17:17

This is an example of anti-virus implementation with an Ubuntu server.

Our back office management solution allows users to upload files in various sections of the application for storage or file sharing. For this reason, checking of files for virus is an important advantage.

We use the ClamAV module integration from Drupal 8.

 

1) Install ClamAV on Ubuntu

Installation on Ubuntu server is straight forward.  However, it is better to install with clamav-daemon clamav-freshclam options for later settings

Categories: FLOSS Project Planets

The achievements during the second GSoC period in KStars

Planet KDE - Tue, 2017-07-25 05:11

In this month, I have been working on fixing bugs found by static code analyzers in KStars. All of the tools are open-source or they can be used for free by open-source projects:

- cppcheck: C++ source code analyzer.
- Clazy: Qt-oriented static code analyzer by KDE for C++ based on Clang.
Clang Static Analyzer: Clang Static Analyzer is a Clang-based static code analyzer for C++.
- Krazy: Code analyzer by KDE.
- Coverity: Coverity is a commercial C++ static code analyzer from Synopsys, but it is free for open-source projects.

If you want to give these analyzers a try, you can pick the build scripts from the tools directory of KStars Git repository and try in your project following our wiki page on the bottom.

Meanwhile I keep fixing memory handling bugs what I find when I test KStars built with runtime sanitizers and adding C++ smart pointers to certain places to make pointer-handling more safe. I also try to minimize the unnecessary includes over the codebase to cut the build time although sometimes I break the Jenkins CI build for FreeBSD or some other esoteric platforms. Ops. :) But I fix those problems as soon as possible.

More details are on our wiki page:
https://techbase.kde.org/Projects/Edu/KStars/C%2B%2B_developer_tools_with_KStars_on_Linux

Categories: FLOSS Project Planets

KStars 2.8.0 is released for Windows, MacOS, and Linux

Planet KDE - Tue, 2017-07-25 04:54


Here comes another KStars release: v2.8.0 For Windows, MacOS, and Linux.

This is a minor bugfix release to increase stability of KStars on all supported platforms. Nevertheless, there were few significant updates:

  • Default NGC/IC catalog is now replaced by OpenNGC project. Christian Dersch compiled this catalog from OpenNGC. KStars now include more objects with accurate positions!

  • Lots of C++11/C++14 code migration work by GSoC 2017 student Csaba Kertész. This also included memory management clean-up, usage of smart pointers, valgrind suppression file and other fixes.

  • Adding support to selection of telescopes from Ekos equipment profile directly. This facilitates having different profiles when utilizing different telescopes on the same mount.
  • Michael Fulbright, KStars & INDI latest developer,  added the ability to have a scheduled refocus every N minutes during a capture sequence. Michael also added dither size in pixels and dither timeout to UI.
Many small bugs were fixed, thanks to our users who are making use of KDE Bug Tracking facilities effectively.
Categories: FLOSS Project Planets

Agiledrop.com Blog: AGILEDROP: Top Drupal 8 Modules

Planet Drupal - Tue, 2017-07-25 01:55
Last time, we have looked at the most popular Drupal modules. There are around 12 000 modules available for Drupal 7 and 3 000 for Drupal 8, of whom only 1 000 are in a stable version. Not so much as some would perhaps expect. However a lot of them make our lives easier each day, so this time we will look at the top Drupal 8 modules. Firstly, we must point out that modules that were already used in the blog post Most popular Drupal modules will be left out. Namely, we already presented, which are available for Drupal 8 and their popularity makes them useful for the newest version of Drupal… READ MORE
Categories: FLOSS Project Planets

Python Insider: Python 3.5.4rc1 and Python 3.4.7rc1 are now available

Planet Python - Tue, 2017-07-25 00:40
Python 3.5.4rc1 and Python 3.4.7rc1 are now available for download.

You can download Python 3.5.4rc1 here, and you can download Python 3.4.7rc1 here.
Categories: FLOSS Project Planets

Gunnar Wolf: Getting ready for DebConf17 in Montreal!

Planet Debian - Mon, 2017-07-24 23:56


(image shamelessly copied from Noodles' Emptiness)

This year I will only make it to DebConf, not to DebCamp. But, still, I am very very happy and excited as the travel date looms nearer! I have ordered some of the delicacies for the Cheese and Wine party, signed up for the public bicycle system of Montreal, and done a fair share of work with the Content Team; finally today we sent out the announcement for the schedule of talks. Of course, there are several issues yet to fix, and a lot of things to do before traveling... But, no doubt about this: It will be an intense week!

Oh, one more thing while we are at it: The schedule as it was published today does not really look like we have organized stuff into tracks — But we have! This will be soon fixed, adding some color-coding to make tracks clearer on the schedule.

This year, I pushed for the Content Team to recover the notion of tracks as an organizative measure, and as something that delivers value to DebConf as a whole. Several months ago, I created a Wiki page for the DebConf tracks, asking interested people to sign up for them. We currently have the following tracks registered:

Blends
Andreas Tille
Debian Science
Michael Banck
Cloud and containers
Luca Filipozzi
Embedded
Pending
Systems administration, automation and orchestation
Pending
Security
Gunnar Wolf

We have two tracks still needing a track coordinator. Do note that most of the tasks mentioned by the Wiki have already been carried out; what a track coordinator will now do is to serve as some sort of moderator, maybe a recurring talkmeister, ensuring continuity and probably providing for some commentary, giving some unity to its sessions. So, the responsibilities for a track coordinator right now are quite similar to what is expected for video team volunteers — but to a set of contiguous sessions.

If you are interested in being the track coordinator/moderator for Embedded or for Systems administration, automation and orchestation or even to share the job with any of the other, registered, coordinators, please speak up! Mail content@debconf.org and update the table in the Wiki page.

See you very soon in Montreal!

Categories: FLOSS Project Planets

Peter Bengtsson: Find static files defined in django-pipeline but not found

Planet Python - Mon, 2017-07-24 20:12

If you're reading this you're probably familiar with how, in django-pipeline, you define bundles of static files to be combined and served. If you're not familiar with django-pipeline it's unlike this'll be of much help.

The Challenge (aka. the pitfall)

So you specify bundles by creating things in your settings.py something like this:

PIPELINE = { 'STYLESHEETS': { 'colors': { 'source_filenames': ( 'css/core.css', 'css/colors/*.css', 'css/layers.css' ), 'output_filename': 'css/colors.css', 'extra_context': { 'media': 'screen,projection', }, }, }, 'JAVASCRIPT': { 'stats': { 'source_filenames': ( 'js/jquery.js', 'js/d3.js', 'js/collections/*.js', 'js/aplication.js', ), 'output_filename': 'js/stats.js', } } }

You do a bit more configuration and now, when you run ./manage.py collectstatic --noinput Django and django-pipeline will gather up all static files from all Django apps installed, then start post processing then and doing things like concatenating them into one file and doing stuff like minification etc.

The problem is, if you look at the example snippet above, there's a typo. Instead of js/application.js it's accidentally js/aplication.js. Oh noes!!

What's sad is it that nobody will notice (running ./manage.py collectstatic will exit with a 0). At least not unless you do some careful manual reviewing. Perhaps you will notice later, when you've pushed the site to prod, that the output file js/stats.js actually doesn't contain the code from js/application.js.

Or, you can automate it!

A Solution (aka. the hack)

I started this work this morning because the error actually happened to us. Thankfully not in production but our staging server produced a rendered HTML page with <link href="/static/css/report.min.cd784b4a5e2d.css" rel="stylesheet" type="text/css" /> which was an actual file but it was 0 bytes.

It wasn't that hard to figure out what the problem was because of the context of recent changes but it would have been nice to catch this during continuous integration.

So what we did was add an extra class to settings.STATICFILES_FINDERS called myproject.base.finders.LeftoverPipelineFinder. So now it looks like this:

# in settings.py STATICFILES_FINDERS = ( 'django.contrib.staticfiles.finders.FileSystemFinder', 'django.contrib.staticfiles.finders.AppDirectoriesFinder', 'pipeline.finders.PipelineFinder', 'myproject.finders.LeftoverPipelineFinder', # the new hotness! )

And here's the class implementation:

from pipeline.finders import PipelineFinder from django.conf import settings from django.core.exceptions import ImproperlyConfigured class LeftoverPipelineFinder(PipelineFinder): """This finder is expected to come AFTER django.contrib.staticfiles.finders.FileSystemFinder and django.contrib.staticfiles.finders.AppDirectoriesFinder in settings.STATICFILES_FINDERS. If a path is looked for here it means it's trying to find a file that none of the regular staticfiles finders couldn't find. """ def find(self, path, all=False): # Before we raise an error, try to find out where, # in the bundles, this was defined. This will make it easier to correct # the mistake. for config_name in 'STYLESHEETS', 'JAVASCRIPT': config = settings.PIPELINE[config_name] for key in config: if path in config[key]['source_filenames']: raise ImproperlyConfigured( 'Static file {!r} can not be found anywhere. Defined in ' "PIPELINE[{!r}][{!r}]['source_filenames']".format( path, config_name, key, ) ) # If the file can't be found AND it's not in bundles, there's # got to be something else really wrong. raise NotImplementedError(path)

Now, if you have a typo or something in your bundles, you'll get a nice error about it as soon as you try to run collectstatic. For example:

▶ ./manage.py collectstatic --noinput Post-processed 'css/search.min.css' as 'css/search.min.css' Post-processed 'css/base.min.css' as 'css/base.min.css' Post-processed 'css/base-dynamic.min.css' as 'css/base-dynamic.min.css' Post-processed 'js/google-analytics.min.js' as 'js/google-analytics.min.js' Traceback (most recent call last): ... django.core.exceptions.ImproperlyConfigured: Static file 'js/aplication.js' can not be found anywhere. Defined in PIPELINE['JAVASCRIPT']['stats']['source_filenames'] Final Thoughts

This was a morning hack. I'm still not entirely sure if this the best approach, but there was none better and the result is pretty good.

We run ./manage.py collectstatic --noinput in our continous integration just before it runs ./manage.py test. So if you make a Pull Request that has a typo in bundles.py it will get caught.

Unfortunately, it won't find missing files if you use foo*.js or something like that. django-pipeline uses glob.glob to convert expressions like that into a list of actual files and that depends on the filesystem and all of that happens before the django.contrib.staticfiles.finders.find function is called.

If you have any better suggestions to solve this, please let me know.

Categories: FLOSS Project Planets
Syndicate content