FLOSS Project Planets

A. Jesse Jiryu Davis: Announcing PyMongo 3.0.3

Planet Python - Wed, 2015-07-01 10:06

Bernie Hackett, Luke Lovett, Anna Herlihy, and I are pleased to announce PyMongo 3.0.3. This release fixes bugs reported since PyMongo 3.0.2—most importantly, a bug that broke Kerberos authentication. We also fixed a TypeError if you try to turn off SSL hostname validation using an option in the MongoDB connection string, and an infinite loop reading certain kinds of corrupt GridFS files.

For the full list of bugs fixed in PyMongo 3.0.3, please see the release in Jira.

If you use PyMongo 3.0.x, upgrade.

If you are on PyMongo 2.8.x, you should probably wait to upgrade: we are about to make it easier for you. PyMongo 2.9, which will be released shortly, provides a smooth bridge for you to upgrade from the old API to the new one.

Let us know if you have any problems by opening a ticket in Jira, in the PYTHON project.

Image: Ian C. on Flickr.

Categories: FLOSS Project Planets

Agaric Collective: Performant bulk-redirection with Apache RewriteMap

Planet Drupal - Wed, 2015-07-01 09:47

When continuing development of a web site, big changes occur every so often. One such change that may occur, frequently as a result of another change, is a bulk update of URLs. When this is necessary, you can greatly improve the response time experienced by your users—as they are redirected from the old path to the new path—by using a handy directive offered in Apache's mod_rewrite called RewriteMap.

At Agaric we regularly turn to Drupal for it's power and flexibility, so one might question why we didn't leverage Drupal's support for handling redirects. When we see an opportunity for our software/system to respond "as early as it can", it is worth investigating how that is handled. Apache handles redirects itself, making it entirely unnecessary to hand-off to PHP, never mind Drupal bootstrapping and retrieving up a redirect record in a database, just to tell a browser (or a search engine) to look somewhere else.

There were two conditions that existed making use of RewriteMap a great candidate. For one, there will be no changes to the list of redirects once they are set: these are for historical purposes only (the old URLs are no longer exposed anywhere else on the site). Also, because we could make the full set of hundreds of redirects via a single RewriteRule—thanks to the substitution capability afforded by RewriteMap—this solution offered a fitting and concise solution.

So, what did we do, and how did we do it?

We started with an existing set of URLs that followed the pattern: http://example.com/user-info/ID[/tab-name]. Subsequently we implemented a module on the site that produced aliases for our user page URLs. The new patten to the page was then (given exceptions for multiple J Smiths, etc via the suffix): http://example.com/user-info/firstname-lastname[-suffix#][/tab-name]. The mapping of ID to firstname-lastname[-suffix#] was readily available within Drupal, so we used an update_hook to write out the existing mappings to a file (in the Drupal public files folder, since we know that's writable by Drupal) . This file (which I called 'staffmapping.txt') is what we used for a simple text-based rewrite map. Sample output of the update hook looked like this:

# User ID to Name mapping: 1 admin-admin 2 john-smith 3 john-smith-2 4 jane-smith

The format of this file is pretty straight-forward: comments can be started on any line with a #, and the mapping lines themselves are composed of {lookupValue}{whitespace}{replacementValue}.

To actually consume this mapping somewhere in our rules, we must let Apache know about the mapping file itself. This is done with a RewriteMap directive, which can be placed in the Server config or else inside a VirtualHost directive. The format of the RewriteMap looks like this: RewriteMap MapName MapType:MapSource. In our case, the file is a simple text file mapping, so the MapType is 'txt'. The resulting string added to our VirtualHost section is then: RewriteMap staffremap txt:/path/to/staffmapping.txt This directive makes this rewrite mapping file available under the name "staffremap" in our RewriteRules. There are other MapTypes, including ones that uses random selection for the replacement values from a text file, using a hash map rather than a text file, using an internal function, or even using an external program or script to generate replacement values.

Now it's time to actually change incoming URLs using this mapping file, providing the 301 redirect we need. The rewrite rule we used, looks like this:

RewriteRule ^user-detail/([0-9]+)(.*) /user-detail/${staffremap:$1}$2 [R=301,L]

The initial argument to the rewrite rule identifies what incoming URLs this rule applies to. This is the string: "^user-detail/([0-9]+)(.*)". This particular rule looks for URLs starting with (signified by the special character ^) the string "user-detail/", then followed by one or more numbers: ([0-9]+), and finally, anything else that might appear at the end of the string: "(.*)". There's a particular feature of regex being used here as well: each of search terms in parenthesis are captured (or tagged) by the regex processor which then provides some references that can be used in the replacement string portion. These are available with $<captured position>—so, the first value captured by parenthesis is available in "$1"—this would be the user ID, and the second in "$2"—which for this expression would be anything else appearing after the user ID.

Following the whitespace is our new target URL expression: "/user-detail/${staffremap:$1}$2". We're keeping the beginning of the URL the same, and then following the expression syntax "${rewritemap:lookupvalue}", which in our case is: "${staffremap:$1}" we find the new user-name URL. This section could be read as: take the value from the rewrite map called "staffremap", where the lookup value is $1 (the first tagged expression in the search: the numeric value) and return the substitution value from that map in place of this expression. So, if we were attempting to visit the old URL /user-detail/1/about, the staffremap provides the value "admin-admin" from our table. The final portion of the replacement URL (which is just $2) copies everything else that was passed on the URL through to the redirected URL. So, for example, /user-detail/1/about includes the /about portion of the URL in the ultimate redirect URL: /user-detail/admin-admin/about

The final section of the sample RewriteRule is for applying additional flags. In this case, we are specifying the response status of 301, and the L indicates to mod_rewrite that this is the last rule it should process.
That's basically it! We've gone from an old URL pattern, to a new one with a redirect mapping file, and only two directives. For an added performance perk, especially if your list of lookup and replacement values is rather lengthy, you can easily change your text table file (type txt) with a HashMap (type dbm) that Apache's mod_rewrite also understands using a quick command and directive adjustment. Following our example, we'll first run:

$> httxt2dbm -i staffrepam.txt -o staffremap.map

Now that we have a hashmap file, we can adjust our RewriteMap directive accordingly, changing the type to map, and of course updating the file name, which becomes:

RewriteMap staffremap dbm:/path/to/staffremap.map

RewriteMap substitutions provide a straight-forward, and high-performance method for pretty extensive enhancement of RewriteRules. If you are not familiar with RewriteRules generally, at some point you should consider reviewing the Apache documentation on mod_rewrite—it's worthwhile knowledge to have.

Categories: FLOSS Project Planets

Dries Buytaert: One year later: the Acquia Certification Program

Planet Drupal - Wed, 2015-07-01 09:31

A little over a year ago we launched the Acquia Certification Program for Drupal. We ended up the first year with close to 1,000 exams taken, which exceeded our goal of 300-600. Today, I'm pleased to announce that the Acquia Certification Program passed another major milestone with over 1,000 exams passed (not just taken).

People have debated the pros and cons of software certifications for years (including myself) so I want to give an update on our certification program and some of the lessons learned.

Acquia's certification program has been a big success. A lot of Drupal users require Acquia Certification; from the Australian government to Johnson & Johnson. We also see many of our agency partners use the program as a tool in the hiring process. While a certification exam can not guarantee someone will be great at their job (e.g. we only test for technical expertise, not for attitude), it does give a frame of reference to work from. The feedback we have heard time and again is how the Acquia Certification Program is tough, but fair; validating skills and knowledge that are important to both customers and partners.

We also made the Certification Magazine Salary Survey as having one of the most desired credentials to obtain. To be a first year program identified among certification leaders like Cisco and Red Hat speaks volumes on the respect our program has established.

Creating a global certification program is resource intensive. We've learned that it requires the commitment of a team of Drupal experts to work on each and every exam. We know have four different exams: developer, front-end specialist, backend specialist and site builder. It roughly takes 40 work days for the initial development of one exam, and about 12 to 18 work days for each exam update. We update all four of our exams several times per year. In addition to creating and maintaining the certification programs, there is also the day-to-day operations for running the program, which includes providing support to participants and ensuring the exams are in place for testing around the globe, both on-line and at test centers. However, we believe that effort is worth it, given the overall positive effect on our community.

We also learned that benefits are an important part to participants and that we need to raise the profile of someone who achieves these credentials, especially those with the new Acquia Certified Grand Master credential (those who passed all three developer exams). We have a special Grand Master Registry and look to create a platform for these Grand Masters to help share their expertise and thoughts. We do believe that if you have a Grand Master working on a project, you have a tremendous asset working in your favor.

At DrupalCon LA, the Acquia Certification Program offered a test center at the event, and we ended up having 12 new Grand Masters by the end of the conference. We saw several companies stepping up to challenge their best people to achieve Grand Master status. We plan to offer the testing at DrupalCon Barcelona, so take advantage of the convenience of the on-site test center and the opportunity to meet and talk with Peter Manijak, who developed and leads our certification efforts, myself and an Acquia Certified Grand Master or two about Acquia Certification and how it can help you in your career!

Categories: FLOSS Project Planets

Daniel Silverstone: Be careful what you ask for

Planet Debian - Wed, 2015-07-01 09:28
Date: Wed, 01 Jul 2015 06:13:16 -0000 From: 123-reg <noreply@123-reg.co.uk> To: dsilvers@digital-scurf.org Subject: Tell us what you think for your chance to win X-Mailer: MIME::Lite 3.027 (F2.74; T1.28; A2.04; B3.13; Q3.13) Tell us what you think of 123-reg! <!-- .style1 {color: #1996d8} -->

Well 123-reg mostly I think you don't know how to do email.

Categories: FLOSS Project Planets

Qt3D Technology Preview Released with Qt 5.5.0

Planet KDE - Wed, 2015-07-01 09:02

KDAB is pleased to announce that the Qt 5.5.0 release includes a Technology Preview of the Qt3D module. Qt3D provides a high-level framework to allow developers to easily add 3D content to Qt applications using either QML or C++ APIs. The Qt3D module is released with the Technology Preview status. This means that Qt3D will continue to see improvements across the API design, supported features and performance before release. It is provided to start collecting feedback from users and to give a taste of what is coming with Qt3D in the future. Please grab a copy of the Qt 5.5.0 release and give Qt3D a test drive and report bugs and feature requests.

Qt3D provides a lot of functionality needed for modern 3D rendering backed by the performance of OpenGL across the platforms supported by Qt with the exception of iOS. There is work under way to support Qt3D on iOS and we expect this to be available very shortly. Qt3D allows developers to not only show 3D content easily but also to totally customise the appearance of objects by using the built-in materials or by providing custom GLSL shaders. Moreover, Qt3D allows control over how the scene is rendered in a data-driven manner. This allows rapid prototyping of new or custom rendering algorithms. Integration of Qt3D and Qt Quick 2 content is enabled by the Scene3D Qt Quick item. Features currently supported by the Qt3D Technology Preview are:

  • A flexible and extensible Entity Component System with a highly threaded and scalable architecture
  • Loading of custom geometry (using built in OBJ parser or assimp if available)
  • Comprehensive material, effect, render pass system to customise appearance
  • Data-driven renderer configuration – change how your scene is rendered without touching C++
  • Support for many rendering techniques – forward, deferred, early z-fill, shadow mapping etc.
  • Support for all GLSL shader stages (excluding compute at present)
  • Good support for textures and render targets including high-dynamic range
  • Support for uniform buffer objects where available
  • Out of the box support for simple geometric primitives and materials
  • Keyboard input and simple camera mouse control
  • Integration with Qt Quick 2 user interfaces

Beyond rendering, Qt3D also provides a framework for adding additional functionality in the future for areas such as:

  • Physics simulation
  • Skeletal and morph target animation
  • 3D positional audio
  • Stereoscopic rendering
  • Artificial intelligence
  • Advanced input mechanisms

To learn more about the architecture and features of Qt3D, please read KDAB’s series of blogs and the Qt3D documentation.

KDAB and The Qt Company will continue to improve Qt3D over the coming months to improve support for more platforms, input handling and picking, import of additional 3D formats, instanced rendering, more materials and better integration points to the rest of Qt. If you wish to contribute either with code, examples, documentation or time then please contact us on the #qt-3d channel on freenode IRC or via the mailing lists.

The post Qt3D Technology Preview Released with Qt 5.5.0 appeared first on KDAB.

Categories: FLOSS Project Planets

Mike Driscoll: Python 101 Screencast: Episode #8 – File I/O

Planet Python - Wed, 2015-07-01 08:30

Check out my latest video from the Python 101 Screencast. In this episode we learn how to use Python’s file handling capabilities. You will learn how to read, write and append to a file.

Categories: FLOSS Project Planets

Andrew Savory: The cost of keeping your cool

Planet Apache - Wed, 2015-07-01 08:22

As the UK basks in record temperatures (“Scorched Britain”, “Havoc on hottest day for a DECADE”, “ travel disruptions and health fears as temperatures soar”, “Record-Baker”, “Heatwave UK”), I’ve been running the air conditioner at home rather more frequently in a desperate attempt to avoid melting.

I was intrigued to know what it costs. According to the label on the side of the AC, it uses 930 watts of power. UK Power has a handy running costs calculator, which does the work for me.

Assuming I use the AC for 6-7 hours a day for a month, it would cost me an extra £19.08 to keep cool*:

Not bad! Especially as it’s unlikely this heatwave will last very long at all (certainly not 30 days), so the cost will be significantly less than that. In fact, expenditure is still going to be lower than in the middle of winter, based on First Utility’s handy usage chart:

And since I use less electricity than the average house (even though I’m usually working from home), I can relax in the cool, cool, breeze coming from the AC.

Now all I need is a fridge full of cold beers.



* I am of course ignoring the environmental impact of this, but there’s little I can do until I am able to move to a house built on more sound ecological principles.

Categories: FLOSS Project Planets

Europython: EuroPython 2015 Keynote: Carrie Anne Philbin

Planet Python - Wed, 2015-07-01 07:51

We are pleased to introduce our next keynote speaker for EuroPython 2015: Carrie Anne Philbin. She will be giving her keynote on Thursday, July 23, to start the EuroPython Educational Summit.

About Carrie Anne Philbin

Carrie Anne is leading the education mission for the Raspberry Pi Foundation, but is also known as an award winning secondary Computing & ICT Teacher, Author, YouTuber:

  • Author of “Adventures in Raspberry Pi”, a computing book for teenagers wanting to get started with Raspberry Pi and programming. Winner of Teach Secondary magazine’s Technology & Innovation Best Author award 2014.
  • Creator of a YouTube video series for teenage girls called “The Geek Gurl Diaries“, which has won a Talk Talk Digital Hero Award. The episodes include interviews with women working in technology and hands on computer science based tutorials.
  • Vice chair of the Computing At Schools (CAS) initiative to get more girls and minority groups into computing, which created a workshop based hack day for teenagers concentrating on delivering good content to include all and ‘Hack the Curric’ bringing academics, educators and industry experts together to create inclusive resources for the new Computing curriculum.

In 2012, she became a Google Certified Teacher and KS3 ICT subject Leader at a school in East London. She has a blended and open approach to teaching as can be seen on her website ICT with Miss P. She became a Skype Moment Maker and ambassador for technology. She is an evangelist and often speaks at conferences like BETT, Raspberry Jamboree, YRS, PyCon UK and now EuroPython.

The Keynote: Designed for Education: A Python Solution

The problem of introducing children to programming and computer science has seen growing attention in the past few years. Initiatives like Raspberry Pi, Code Club, code.org, (and many more) have been created to help solve this problem. With the introduction of a national computing curriculum in the UK, teachers have been searching for a text based programming language to help teach computational thinking as a follow on from visual languages like Scratch.

The educational community has been served well by Python, benefiting from its straight-forward syntax, large selection of libraries, and supportive community. Education-focused summits are now a major part of most major Python Conferences. Assistance in terms of documentation and training is invaluable, but perhaps there are technical means of improving the experience of those using Python in education. Clearly the needs of teachers and their students are different to those of the seasoned programmer. Children are unlikely to come to their teachers with frustrations about the Global Interpreter Lock! But issues such as usability of IDEs or comprehensibility of error messages are of utmost

In this keynote, Carrie Anne will discuss existing barriers to Python becoming the premier language of choice for teaching computer science, and how learning Python could be helped immensely through tooling and further support from the Python developer community.

EuroPython Educational Summit

We will have Educational Summit focused talks, trainings, birds of a feather sessions to debate and also Educational Sprints for the building of education focused projects during the weekend.

EuroPython 2015 Educational Summit

Please see the summit announcement for more details.


EuroPython 2015 Team

Categories: FLOSS Project Planets

guix @ Savannah: Reproducible and User-Controlled Software Environments in HPC with Guix

GNU Planet! - Wed, 2015-07-01 06:05

Our paper entitled Reproducible and User-Controlled Software Environments in HPC with Guix was accepted for RepPar, a workshop on reproducibility in parallel computing:

Support teams of high-performance computing (HPC) systems often find themselves between a rock and a hard place: on one hand, they understandably administrate these large systems in a conservative way, but on the other hand, they try to satisfy their users by deploying up-to-date tool chains as well as libraries and scientific software. HPC system users often have no guarantee that they will be able to reproduce results at a later point in time, even on the same system—software may have been upgraded, removed, or recompiled under their feet, and they have little hope of being able to reproduce the same software environment elsewhere. We present GNU Guix and the functional package management paradigm and show how it can improve reproducibility and sharing among researchers with representative use cases.

The paper can be thought of as a followup to the recent experience report by Ricardo Wurmus.

We believe package management and reproducibility are key topics for HPC research. We are glad to have this opportunity to discuss the subject with researchers of the field.

About GNU Guix

GNU Guix is a functional package manager for the GNU system. The Guix System Distribution or GuixSD is an advanced distribution of the GNU system that relies on GNU Guix and respects the user's freedom.

In addition to standard package management features, Guix supports transactional upgrades and roll-backs, unprivileged package management, per-user profiles, and garbage collection. Guix uses low-level mechanisms from the Nix package manager, except that packages are defined as native Guile modules, using extensions to the Scheme language. GuixSD offers a declarative approach to operating system configuration management, and is highly customizable and hackable.

GuixSD can be used on an i686 or x86_64 machine. It is also possible to use Guix on top of an already installed GNU/Linux system, including on mips64el and armv7.

Categories: FLOSS Project Planets

Jim Birch: Essential Drupal: Stage File Proxy

Planet Drupal - Wed, 2015-07-01 05:20

We can easily checkout code from our git repositories for our local, development, and staging servers.  We can get a database from the live site through Backup and Migrate, drush, or a number of other ways.  But getting the files of the site, the images, pdfs, and everything else in /sites/default/files is not on the top of the list of most developers.  In recent versions of Backup and Migrate, you can export the files, but often times, this can be a huge archive file.  There is an easier way.

The Stage File Proxy module saves the day by sending requests for files to the live server if it does not exist yet in your local environment.  This saves you space on your non-production environment since it only grabs files from the pages you visit.  Great for us that have dozens of sites on our local. 

As simple as can be, it gets the files you need on your local server, as you need them.  No more navigating broken looking dev sites.  This will get your environment looking as it should so you can concentrate on your task at hand.

Read more

Categories: FLOSS Project Planets

Sponsor our digiKam team for Randa meeting

Planet KDE - Wed, 2015-07-01 04:57

Dear digiKam Users

One of our digiKam developers requires your kind support to organize his trip to "digiKam developers meetup 2015" to be held in Randa, Switzerland from 6th September'15 to 13th September'15. He seeks to raise 639 euros from fundraising to support his travel expenses to Randa.

read more

Categories: FLOSS Project Planets

Sooper Drupal Themes: YoutTube Video of Drupal Theme integrated Drag and Drop builder

Planet Drupal - Wed, 2015-07-01 04:22
After I posted a case study last week I had a number of readers ask me if they could try a demo and see how it works. There is no try-out demo yet but in the meanwhile I produced a video that demonstrates the basic controls: .embed-container { position: relative; padding-bottom: 56.25%; height: 0; overflow: hidden; max-width: 100%; } .embed-container iframe, .embed-container object, .embed-container embed { position: absolute; top: 0; left: 0; width: 100%; height: 100%; } If you have any questions about integration and the open source library that powers it feel free to contact or comment! Tags planet drag and drop glazed planet drag and drop glazed Drupal 7.x
Categories: FLOSS Project Planets

FSF Events: Richard Stallman to speak at SeaGL (Seattle, WA)

GNU Planet! - Wed, 2015-07-01 02:30

Richard Stallman will be delivering the keynote speech at this year's SeaGL, the 2015 Seattle GNU/Linux Conference. His speech will be nontechnical, admission is gratis, and the public is encouraged to attend.

Speech topic and start time to be determined.

Please fill out our contact form, so that we can contact you about future events in and around Seattle.

Categories: FLOSS Project Planets

Christian Perrier: [LIFE] Running activities

Planet Debian - Wed, 2015-07-01 00:37
Hello dear readers,

It has been quite some time since I blogged on Planet Debian,so today, I just want to give some news to fellow Debian pals.

My involvment in Debian is still there. I'm probably less visible nowadays, but I'm still actively working on some packages, monotiring some i18n activities and doing work on D-I.

But, as you know, running has taken precedence nowadays and is still becoming a growing part of my life (along with my family, of course).

This year, I had a first "summit" running the "Vulcain" trail race in French "Massif Central" (mountains in Central France), which was 80km and 3000m positive climb race. It was run mostly in snow and with quite bad weather conditions, a good training for more difficult races. I completed it in about more than 12 hours, for a race that finally had less than 60% finishers.

Later on, most races were preparation races for the summer moutain races : I mostly ran three 50km trail races in the Paris and neighbourhood area. All of them were very good results with a good feeling. Some were run along with friends from the Kikourou.net web community, where I am now very active.

My training was also strongly increased wrt former years (yes that *is* possible), peaking at more than 500km during May, where I was mostly on holidays all month long (lucky man).

And now, the first Great Great Thing of the year is coming : La Montagn'hard, 110 kilometers, about 9000 meters positive climb, around Les Contamines, close to Mont-Blanc in French Alps.

That is a Big One, indeed. Technically more difficult than the TDS race I ran last August, during DebConf (120km, but "only" 7000 meters climb). Montagn'hard is indeed known as one of the most difficult moutain trail races in France.

I plan to complete it in about 29 hours....but that can indeed be 30, 32 or even 35, who knows what can happen? Given the very high temperatures over Europe this week (they'll peak at about 38°C on Saturday in the Alps), that will be an incredibly difficult challenge and we expect about only 40% finishers.

A live tracking will be available for thos who care at http://chrono.geofp.com/montagnhard2015/v3/. Wish me luck !

Next challenge will be end of August, with the "Echappee Belle" race : 144km and 10.000 meters positive climb, still in French Alps (Belledonne range, this time). About 48 hours, or even up to 55, two nights out.....harder and hopefully better, faster, stronger...:-)

Categories: FLOSS Project Planets

Mukul Gandhi: www server host &lt;--&gt; intermediate network BLOB &lt;--&gt; mobile device, "interface conceptual architecture"

Planet Apache - Tue, 2015-06-30 22:53
I'm tempted to write here about the conceptual "IP / mobile network" interface architecture between a mobile software device and the information content serving host. There are few reasons why I'm writing about this topic here, but I'm not in a position to write about those reasons here. So here goes my description about this topic in this post..

The provided pictorial representation as below,

of the mentioned architectural concept in this post, is given in such a way so that it does appear quite self-explanatory to any of the readers of this post. In the diagrammatic illustration it seems to me that the use of computing node (C) i.e. the data cache node hosted by a mobile network operator deserves a little more textual explanation in the body of this post. This node is thought by me to be optional in this architecture, but if present in the real such physical network it would be able to improve the user's information retrieval performance. In-fact the nodes (C) and (D) can be thought of as one physical computing node / an IT computer (since the tasks performed by nodes (C) and (D) are tiny with-respect-to the compute sizing capabilities of node C & D's hardware, and also the tasks performed by nodes C & D should logically belong as one node and should exist within the mobile operator's network domain).

The conceptual GSM / CDMA network BLOB shown as "A specific GSM/CDMA Mobile Network", has following elements which I'm sure makes sense;

1) Mobile Device: As an author of this blog post, I know that this will be a real device that a user has to use.

2) GSM / CDMA Switch (network switches route analog / digital "data or information" traffic, from one part of network to another based on certain rules): I know for sure that GSM network switching devices exist that do the kind of routing as imagined in this post. CDMA switches would be there similary (since GSM & CDMA are different only by protocol, but they play same role in the overall scheme of things).

3) Mobile network cloud: This is thought to be an imaginary mobile network BLOB (a BLOB since it's internal details are left open as-far-as the architectural representation).

I feel that after looking at the provided diagram, and reading the text of this post it may be clear what I wish to convey. Just some food for thought, and for any comments.
Categories: FLOSS Project Planets

End Point: Python string formatting and UTF-8 problems workaround

Planet Python - Tue, 2015-06-30 22:52
Recently I worked on a program which required me to filter hundred of lines of blog titles. Throughout the assignment I stumbled upon a few interesting problems, some of which are outlined in the following paragraphs.

Non Roman characters issueDuring the testing session I missed one title and investigating why it happened, I found that it was simply because the title contained non-Roman characters.

Here is the code's snippet that I was previously using:

for e in results: simple_author=e['author'].split('(')[1][:-1].strip() if freqs.get(simple_author,0) < 1: print parse(e['published']).strftime("%Y-%m-%d") , "--",simple_author, "--", e['title']
And here is the fixed version

for e in results: simple_author=e['author'].split('(')[1][:-1].strip().encode('UTF-8') if freqs.get(simple_author,0) < 1: print parse(e['published']).strftime("%Y-%m-%d") , "--",simple_author, "--", e['title'].encode('UTF-8')
To fix the issue I faces I added .encode('UTF-8') in order to encode the characters with the UTF-8 encoding. Here is an example title that would have been otherwise left out:

2014-11-18 -- Unknown -- Novo website do Liquid Galaxy em Português!
Python 2.7 uses ASCII as its default encoding but in our case that wasn't sufficient to scrap web contents which often contains UTF-8 characters. To be more precise, this program fetches an RSS feed in XML format and in there it finds UTF-8 characters. So when the initial Python code I wrote met UTF-8 characters, while using ASCII encoding as the default sets, it was unable to identify them and returned an error.

Here is an example of the parsing error it gave us while fetching non-roman characters while using ASCII encoding:
UnicodeEncodeError: 'ascii' codec can't encode character u'\xea' in position 40: ordinal not in range(128)
Right and Left text alignment
In addition to the error previously mentioned, I also had the chance to dig into several ways of formatting output.
The following format is the one I used as the initial output format:

print("Name".ljust(30)+"Age".rjust(30)) Name Age
Using "ljust" and "rjust" method
I want to improve the readability in the example above by left-justify "Name" by 30 characters and "Age" by another 30 characters distance.

Let's try with the '*' fill character. The syntax is str.ljust(width[, fillchar])

print("Name".ljust(30,'*')+"Age".rjust(30)) Name************************** Age
And now let's add .rjust:

print("Name".ljust(30,'*')+"Age".rjust(30,'#')) Name**************************###########################Age
By using str, it counts from the left by 30 characters including the word "Name" which has four characters
and then another 30 characters including "Age" which has three letters, by giving us the desired output.

Using "format" method
Alternatively, it is possible to use the same indentation approach with the format string method:

print("{!s:{fill}}{!s:>{fill}}".format("Name", "Age",fill=30)) Name Age
And with the same progression, it is also possible to do something like:

print("{!s:*{fill}}{!s:>{fill}}".format("Name", "Age",fill=30)) Name************************** Age print("{!s:*{fill}}{!s:#>{fill}}".format("Name", "Age",fill=30)) Name**************************###########################Age
"format" also offers a feature to indent text in the middle. To put the desired string in the middle of the "fill" characters trail, simply use the ^ (caret) character:
print("{!s:*^{fill}}{!s:#^{fill}}".format("Age","Name",fill=30)) *************Age**************#############Name#############
Feel free to refer the Python's documentation on Unicode here:
And for the "format" method it can be referred here:
Categories: FLOSS Project Planets

KatteKrab: Comparing D7 and D8 outta the box

Planet Drupal - Tue, 2015-06-30 21:53
Wednesday, July 1, 2015 - 11:53

I did another video the other day. This time I've got a D7 and D8 install open side by side, and compare the process of adding an article.

Categories: FLOSS Project Planets

Junichi Uekawa: My thermometer (Fplug) is no longer returning temperature.

Planet Debian - Tue, 2015-06-30 20:44
My thermometer (Fplug) is no longer returning temperature. It does give me humidity. The values don't really look sane either, maybe it's not a great product.

Categories: FLOSS Project Planets

Steve McIntyre: Quick trip to Sweden

Planet Debian - Tue, 2015-06-30 20:06

Jo and I spent a few days in Sweden and had an awesome time! The main reason for being there was Leif and Maria's wedding way up north in Skellefteå. They cunningly organised their ceremony for the Midsummer weekend, which was an excellent plan - we had a full weekend of partying while we were there. :-)

We had some time to ourselves while we were there, so we wandered about a little and got to see some of the beautiful coastal countryside.

Then on the way home we stopped off in Umeå to visit Mattias Wadenstein (maswan) and his wife Melanie, and he showed me around some of the machines that he's been admining on behalf of Debian. Maybe I'm a sad geek, but I feel quite a bond with one of the machines there, pettersson.debian.org. It's the official CD build machine for Debian, and I've been responsible for thrashing it really hard for the last 5 years or so... :-)

Massive thanks to the University of Umeå and their Academic Computer Club for hosting Debian machines and serving all the CD images for us!

The only downsides from the trip were the massive tiredness (midnight sun is pretty, but notconducive to sleep!) the mosquito bites and the nasty plague^Wcold that we picked up while we were there... Ah well. :-)

Categories: FLOSS Project Planets

Pixelite: Debugging Drupal performance with Cache Debug module

Planet Drupal - Tue, 2015-06-30 20:00

This blog post is for developers, not site builders, as the analysis for cache debugging requires knowledge about the runtime stack of Drupal.

The Problem with Caching in Drupal 7

To obtain performance in Drupal 7, Drupal relies heavily on caching. That is, to process something and cache the end result so that same work doesn’t need to be repeated. Conditions also have to be created for when that cache expires or is invalidated. Drupal has a caching layer to help with this. When you want to store something in cache, you use cache_set, to retrieve it, you use cache_get and to wipe the cache bin clean, you can use cache_clear_all.

Often, modules can implicitly clear or set cache unintentionally. This can lead to more caching overhead than you need. For example, theme registry clearing, use of the variable_set function or calls to other modules that call cache_clear_all. The problem is, how do you track down culprits to fix the issue?

Enter Cache Debug

Cache Debug is a module that wraps around the caching layer and adds logging. Including stacktrace information. It means when a cache_set or cache_clear_all is called, you can trace back to what called it - understand the problem and fix it. Very quickly.

It comes with three logging options:

  • watchdog - good if you’re using syslog module but deadly if you’re using dblog.
  • error_log - logs to your php error log
  • arbitrary file - specify your own log file to log to
Configuring Cache Debug

Because the caching system is so highly utilized, cache logging can be incredibly verbose. Perhaps this is why there is no logging around this in Drupal core. Fortunately, Cache Debug is highly configurable to control what to log.

NOTE: Because the caching system is loaded and used before Drupal’s variable system which manages configuration, it is best to set configuration in settings.php rather than in the database. However, there is a web UI that does set configuration in the database for ease of use.

Basic configuration

If you’ve used the memcache module before, this should feel familiar. In order to use Cache Debug, you need to set it as the cache handler:

<?php $conf['cache_backends'][] = 'sites/all/modules/cache_debug/cache_debug.inc'; $conf['cache_default_class'] = 'DrupalDebugCache'; ?>

This tells Drupal that there is a cache backend located in the path provided (make sure its correct for your Drupal site!) and that the default class for all cache bins is the DrupalDebugCache class. If you only want to monitor a single bin you may want to omit this option.

Since Cache Debug is a logger and not an actual caching system, it needs to pass cache requests onto a real cache system. By default, Debug Cache will use Drupal core’s Database Cache system for cache storage, but if you’re using memcache, redis or similar, you may want to set that as the handler for Cache Debug:

<?php $conf['cache_debug_class'] = 'MemCacheDrupal'; $conf['cache_cache_debug_form'] = 'DrupalDatabaseCache'; ?>

You need to also configure those modules accordingly.

At this point, you’ll be logging all cache calls and stack traces to set and clear calls to the php error log.

Configure the logging location

You may want to choose your own logging location. For example, if you use dblog, then you won’t want to log to watchdog because it will bloat your database. Likewise, if you don’t want to bloat our php error log, then you may want to log to an arbitrary file. You can choose your logging location by setting cache_debug_logging_destination to error_log (default), watchdog or file. For file you will also need to provide the location:

<?php $conf['cache_debug_logging_destination'] = 'file'; $conf['cache_debug_log_filepath'] = '/tmp/cachedebug.log'; ?> Configuring logging options

You can choose to log calls to cache get, getMulti, set and clear. You can also choose to log a stacktrace of these calls to show the stack that triggered the call. This is most useful for calls to SET and CLEAR. For a minimal logging option with the most about of insight, you might want to try this:

<?php $conf['cache_debug_log_get'] = FALSE; $conf['cache_debug_log_getMulti'] = FALSE; $conf['cache_debug_log_set'] = TRUE; $conf['cache_debug_log_clear'] = TRUE; $conf['cache_debug_stacktrace_set'] = TRUE; $conf['cache_debug_stacktrace_clear'] = TRUE; ?> Logging per cache bin

You don’t have to log the entire caching layer if you know which bin to look at for the caching issue you’re observing. For example, if you’re looking for misuse of variable_set, you only need to log the cache_bootstrap bin. In which case you could do this:

<?php # Do not log to all cache bins so ensure this line is removed (from above): # $conf['cache_default_class'] = 'DrupalDebugCache'; $conf['cache_bootstrap_class'] = 'DrupalDebugCache'; ?> Configure for common issues

Variable set calls and theme registry rebuilds are the two most common issues and so Cache Debug has use cases for these issues built in. So long as Cache Debug is the cache handler for the bin, you can turn off logging and turn on these features and Cache Debug will only log when these issues occur:

<?php $conf['cache_default_class'] = 'DrupalDebugCache'; $conf['cache_debug_common_settings'] = array( 'variables' => TRUE, 'theme_registry' => TRUE, ); // Turn off logging $conf['cache_debug_log_get'] = FALSE; $conf['cache_debug_log_getMulti'] = FALSE; $conf['cache_debug_log_set'] = FALSE; $conf['cache_debug_log_clear'] = FALSE; ?> Analysing the logged data

Cache debug logs to a log file like the example below:

In this snapshot of log output you can see both how cache debug logs cache calls and the stacktracing in action.

Log format structure

A log line starts with a value that describes the cache bin, the cache command and the cache id. E.g. cache_bootstrap->set->variables would bet a cache_set call to the cache_bootstrap cache bin to set the variables cache key. Some calls also log additional data, for example, cache clear also indicates if the call was a wildcard clear. Set calls also log how much data (length) was set.

Stack trace logs

When stack tracing is enabled for specific commands, a stack trace will be logged immediately after the log event that triggered it. The trace rolls back through each function that led to the current cache command being triggered. In the example above you can see that cache_clear_all was called by drupal_theme_rebuild which was called by an include from phptemplate_init. If you look at the source code in phptemplate_init, you’ll see that this means a cache rebuild was triggered from including template.php. In this case it was that Zen base theme had the theme registry rebuild left on.

Categories: FLOSS Project Planets
Syndicate content