FLOSS Project Planets

Vincent Bernat: Testing infrastructure with serverspec

Planet Debian - Wed, 2014-01-01 08:05

Checking if your servers are configured correctly can be done with IT automation tools like Puppet, Chef, Ansible or Salt. They allow an administrator to specify a target configuration and ensure it is applied. They can also run in a dry-run mode and report servers not matching the expected configuration.

On the other hand, serverspec is a tool to bring the well known RSpec, a testing tool for the Ruby programming language frequently used for test-driven development, to the infrastructure world. It can be used to remotely test server state through an SSH connection.

Why one would use such an additional tool? Many things are easier to express with a test than with a configuration change, like for example checking that a service is correctly installed by checking it is listening to some port.

Getting started

Good knowledge of Ruby may help but is not a prerequisite to the use of serverspec. Writing tests feels like writing what we expect in plain English. If you think you need to know more about Ruby, here are two short resources to get started:

serverspec’s homepage contains a short and concise tutorial on how to get started. Please, read it. As a first illustration, here is a test checking a service is correctly listening on port 80:

describe port(80) do it { should be_listening } end

The following test will spot servers still running with Debian Squeeze instead of Debian Wheezy:

describe command("lsb_release -d") do it { should return_stdout /wheezy/ } end

Conditional tests are also possible. For example, we want to check the miimon parameter of bond0, but only when the interface is present:

has_bond0 = file('/sys/class/net/bond0').directory? # miimon should be set to something other than 0, otherwise, no checks # are performed. describe file("/sys/class/net/bond0/bonding/miimon"), :if => has_bond0 do it { should be_file } its(:content) { should_not eq "0\n" } end

serverspec comes with a complete documentation of available resource types (like port and command) that can be used after the keyword describe.

When a test is too complex to be expressed with simple expectations, it can be specified with arbitrary commands. In the below example, we check if memcached is configured to use almost all the available system memory:

# We want memcached to use almost all memory. With a 2GB margin. describe "memcached" do it "should use almost all memory" do total = command("vmstat -s | head -1").stdout # ➊ total = /\d+/.match(total)[0].to_i total /= 1024 args = process("memcached").args # ➋ memcached = /-m (\d+)/.match(args)[1].to_i (total - memcached).should be > 0 (total - memcached).should be < 2000 end end

A bit more arcane, but still understandable: we combine arbitrary shell commands (in ➊) and use of other serverspec resource types (in ➋).

Advanced use

Out of the box, serverspec provides a strong fundation to build a compliance tool to be run on all systems. It comes with some useful advanced tips, like sharing tests among similar hosts or executing several tests in parallel.

I have setup a GitHub repository to be used as a template to get the following features:

  • assign roles to servers and tests to roles;
  • parallel execution;
  • report generation & viewer.
Host classification

By default, serverspec-init generates a template where each host has its own directory with its unique set of tests. serverspec only handles test execution on remote hosts: the test execution flow (which tests are executed on which servers) is delegated to some Rakefile1. Instead of extracting the list of hosts to test from a directory hiearchy, we can extract it from a file (or from an LDAP server or from any source) and attach a set of roles to each of them:

hosts = File.foreach("hosts") .map { |line| line.strip } .map do |host| { :name => host.strip, :roles => roles(host.strip), } end

The roles() function should return a list of roles for a given hostname. It could be something as simple as this:

def roles(host) roles = [ "all" ] case host when /^web-/ roles << "web" when /^memc-/ roles << "memcache" when /^lb-/ roles << "lb" when /^proxy-/ roles << "proxy" end roles end

In the snippet below, we create a task for each server as well as a server:all task that will execute the tests for all hosts (in ➊). Pay attention, in ➋, at how we attach the roles to each server.

namespace :server do desc "Run serverspec to all hosts" task :all => hosts.map { |h| h[:name] } # ➊ hosts.each do |host| desc "Run serverspec to host #{host[:name]}" ServerspecTask.new(host[:name].to_sym) do |t| t.target = host[:name] # ➋: Build the list of tests to execute from server roles t.pattern = './spec/{' + host[:roles].join(",") + '}/*_spec.rb' end end end

You can check the list of tasks created:

$ rake -T rake check:server:all # Run serverspec to all hosts rake check:server:web-10 # Run serverspec to host web-10 rake check:server:web-11 # Run serverspec to host web-11 rake check:server:web-12 # Run serverspec to host web-12

Then, you need to modify spec/spec_helper.rb to tell serverspec to fetch the host to test from the environment variable TARGET_HOST instead of extracting it from the spec file name.

Parallel execution

By default, each task is executed when the previous one has finished. With many hosts, this can take some time. rake provides the -j flag to specify the number of tasks to be executed in parallel and the -m flag to apply parallelism to all tasks:

$ rake -j 10 -m check:server:all Reports

rspec is invoked for each host. Therefore, the output is something like this:

$ rake spec env TARGET_HOST=web-10 /usr/bin/ruby -S rspec spec/web/apache2_spec.rb spec/all/debian_spec.rb ...... Finished in 0.99715 seconds 6 examples, 0 failures env TARGET_HOST=web-11 /usr/bin/ruby -S rspec spec/web/apache2_spec.rb spec/all/debian_spec.rb ...... Finished in 1.45411 seconds 6 examples, 0 failures

This does not scale well if you have dozens or hundreds of hosts to test. Moreover, the output is mangled with parallel execution. Fortunately, rspec comes with the ability to save results in JSON format. Those per-host results can then be consolidated into a single JSON file. All this can be done in the Rakefile:

  1. For each task, set rspec_opts to --format json --out ./reports/current/#{target}.json. This is done automatically by the subclass ServerspecTask which also handles passing the hostname in an environment variable and a more concise and colored output.

  2. Add a task to collect the generated JSON files into a single report. The test source code is also embedded in the report to make it self-sufficient. Moreover, this task is executed automatically by adding it as a dependency of the last serverspec-related task.

Have a look at the complete Rakefile for more details on how this is done.

A very simple web-based viewer can handle those reports2. It shows the test results as a matrix with failed tests in red:

Clicking on any test will display the necessary information to troubleshoot errors, including the test short description, the complete test code, the expectation message and the backtrace:

I hope this additional layer will help making serverspec another feather in the “IT” cap, between an automation tool and a supervision tool.

  1. A Rakefile is a Makefile where tasks and their dependencies are described in plain Ruby. rake will execute them in the appropriate order. 

  2. The viewer is available in the GitHub repository in the viewer/ directory. 

Categories: FLOSS Project Planets

Season off, code on!

Planet KDE - Wed, 2014-01-01 07:21

Hello planet! Happy New Year 2014! With the year 2013 Season of KDE has come to an end. This was great experience for me. Under guidance of Sebastian Kügler (sebas), I ported some of the DataEngines to KDE Frameworks 5 and libplasma2, Some applets to Plasma2 and QML2 and also killed some bugs.

List of things that I ported to Plasma2 includes Icon (Generic Icon), Device notifier, ksgrd module from libksysguard, Activity bar and much more. I got to know many things about opensource world.

Patience

Okay that’s first rule of life, Keep patience. When you ask a question or ping someone on IRC, just stick to IRC, don’t disappear or panic! If you don’t get reply in expected time that doesn’t mean that you are being ignored. You will get reply for sure. Just wait!

Double check whatever you do

When you work on some opensource project and you have commit access to that project, it doesn’t mean you can commit anything. If you are going to commit something without someone reviewing, please make sure that commit will not break build on CI system. Keep eye on Jenkins. During my SoK once I commited some changes that were not ready to commit and result was broken build of kde-workspace on Jenkins, I fixed it after 30 minutes and 5 commits. :(

Stick to the plan

That’s what movies tell us! (Right?) Sometimes in opensource world it is not recommanded to do work which is out of plan. I started to port some of applets in kdeplasma-addons during the end of SoK but then I realized porting stuffs from kdeplasma-addons was out of scope to meet the requirements of Plasma 2 release schedule. It’s not that you are forbidden to do stuffs out of plan but its better if you stick to the plan.

That’s what I learnt during SoK. I plan to make my contributions to KDE this year also in spair time. Sebastian Kügler and everyone else from Plasma team, thanks again for mentoring me through this season.

Categories: FLOSS Project Planets

Pierre Carbonnelle: PySonar + Cython = pure-python compiler ?

Planet Python - Wed, 2014-01-01 06:55
PySonar + Cython = pure-python compiler ?
The recent buzz on PySonar made me think about a possible way to create a pure-python compiler.
The idea would be to statically analyze your pure-python code with PySonar, and to format the result of the analysis into an "augmenting .pxd" file containing the static type definitions that Cython can use for its compilation. You would then compile your .py files with Cython, with the help of the .pxd files, resulting in a fast, efficient executable.  
Wouldn't that be neat ? Is it feasible ? Is anybody working on it ? Please discuss on reddit.
Categories: FLOSS Project Planets

Vincent Fourmond: Rubyforge is dead, but ctioga2 goes on...

Planet Debian - Wed, 2014-01-01 06:27
In the light of the frequent recent downtimes of rubyforge, and thanks to the information from @copiousfreetime, I finally decided to move the hosting of ctioga2 away from rubyforge, to sourceforge. Transition went smooth, git is now the only VCS. Code is hosted at sourceforge and mirrored at github.

Work goes on on ctioga2. I've recently implemented a decent support for histograms. It is already much more powerful than the one in the old ctioga, but it is far from being feature-full. Here's a preview.

I'm slowly preparing a new release for ctioga2, that would incorporate quite some fixes since the last time, and quite a few nice features in addition to the histograms. Stay tuned !

Categories: FLOSS Project Planets

News from Marble: Introducing Sun and the Moon

Planet KDE - Wed, 2014-01-01 06:18

Happy New Year, folks!

Here, in Marble we work a lot on new cool features, so you can play around with the Globe and explore the world more and more. Not a secret, that each minor release Marble shows great improvements in performance and features. Today I want to show you our new previously missing feature - enhanced Sun and absolutely cool support of the Moon.

For lazy people, who don’t really like reading all this news posts here is a quick overview of added stuff:

  • Gerhard Holtkamp’s astrolib is bundled to Marble core
  • Sun texture is scaled according to its real size (thanks, astrolib!)
  • Moon is rendered, moon phases are shown

Marble provides a nice opportunity to observe eclipses without even standing up from your sofa. You just go for View -> Eclipses and choose the right one (only works in master now, sorry). Here is an example of Tue Apr 29 2014:

A quick example of moon phases support in the Marble is 23 Jan 2014, when we have the Moon, Spica and Mars around!

You would probably like to compare this to what KStars gives as output:

Note: distances to Mars are different, because in Marble, the Moon is 4x times bigger than it should be (otherwise it’s too small and makes no sense, however this can be changed in preferences).

**Few words about moon phases (bonus) **

Moon rendering becomes not so trivial as soon as you need to reflect moon phases. It should be rotated according to its actual position also, so that’s where comes astrolib (thanks, Gerhard Holtkamp!). Final rendering process looks similar to this:

  • Rotation (Moon is rotated relatively to its current position to the Earth)
  • Scaling (by default, in Marble we use 4x aspect ratio to “real” Moon size, but you possibly would like to turn it off)
  • Moon phases overlay rendering

Let’s stop on the last step. Astrolib provides API to get information about the Moon phase in the specific moment of time:

void SolarSystem::getLunarPhase(qreal &phase, qreal &ildisk, qreal &amag)

Outputs are actual phase, illuminated disk and the apparent magnitude of the Moon reffered to the center of the Earth.

There are 8 different phases (from 1 to 8 on the picture): New Moon, Waxing Crescent, First Quarter, Waxing Gibbous, Full Moon, Waning Gibbous, Last Quarter, Wanning Crescent and then it’s New Moon again. Actual rendering process is about drawing two arcs, which is not complicated, but still needs to cover all cases and calculate angles neatly. An exported paste is published here. You also can observe it right in the source code (StarsPlugin.cpp:1130; line may be different later).

Find your way and explore the world!

Categories: FLOSS Project Planets

GNU Hackers Meeting: Next European GNU Hackers Meeting to take place in Munich, August 15 to August 17 2014

GNU Planet! - Wed, 2014-01-01 05:01

We are happy to announce the eighth GNU Hackers' Meeting, which will take place from August 15 to August 17 2014 in Munich, Germany.

The GNU Hackers' Meetings are an organized, yet friendly and informal opportunity to discuss technical, social and organizational issues concerning GNU and free software in general.

We have set up an official meeting page containing practical information at http://www.gnu.org/ghm/2014/munich

As we are still in the process of defining the program, we welcome your talk proposals. We are particularly interested in new GNU programs and recent developments in existing packages, as well as any GNU related topic. In the past, the audience has been technically competent, and very technical topics are usually well received. However, topics of a non-technical nature are also very welcome.

GNU developers, users and other interested parties are invited to address the meeting. This year, presentations will fall into two categories: Short Talks, and Full Presentations. Short talks will be allocated 30 minutes including time for questions (if any). The longer presentations will be 1 hour including question time. We will try to schedule similar talks together in the same morning or afternoon session.

Registration is necessary but free of charge. There will be the opportunity to make a donation to the cost of future meetings.

The FSF has kindly set up a small fund to help finance (documented) travel expenses for GNU maintainers and contributors, past and present. In case you qualify and you need financial assistance to attend, please mention this fact in your registration e-mail, along with all the relevant details including the minimum amount that would permit you to attend the event, no later than June 13 2014.

You are welcome to join the public mailing list ghm-discuss@gnu.org (https://lists.gnu.org/mailman/listinfo/ghm-discuss). Talk proposals and registration e-mails should also be directed to ghm-reg@gnu.org

Categories: FLOSS Project Planets

cgicc @ Savannah: GNU Cgicc 3.2.11

GNU Planet! - Wed, 2014-01-01 04:24

Hello

I am happy to announce the next release of GNU Cgicc.
This release is numbered 3.2.11 .

Description
===========
GNU cgicc is an ANSI C++ compliant class library that greatly
simplifies the creation of CGI applications for the World Wide Web.
cgicc performs the following functions:

  • Parses both GET and POST form data transparently.
  • Provides string, integer, floating-point and single- and

multiple-choice retrieval methods for form data.

  • Provides methods for saving and restoring CGI environments to aid in

application debugging.

  • Provides full on-the-fly HTML/XHTML generation capabilities, with

support for cookies.

  • Supports HTTP file upload.
  • Compatible with FastCGI.

Changes in 3.2.11: 31 Dec 2013
==========================

  • Add Mac Osx compatibility

Download
========
Gzipped tarballs are available from

ftp://ftp.gnu.org/gnu/cgicc/
or
http://ftp.gnu.org/gnu/cgicc/

and its mirror sites. SHA1 sums may be found there as well.

The web site of GNU cgicc is at:

http://www.gnu.org/software/cgicc/

Sébastien DIAZ <sebastien.diaz@gmail.com>
GNU Cgicc Maintainer

Categories: FLOSS Project Planets

Junichi Uekawa: A Happy New Year.

Planet Debian - Wed, 2014-01-01 04:19
A Happy New Year.

Categories: FLOSS Project Planets

Google Nexus 5 Review

LinuxPlanet - Mon, 2013-12-30 15:49

This review was originally done for Bad Voltage, but I figured it may also be of interest to my general readers.

In this episode I’m going to review the recently released Nexus 5 phone, manufactured by LG. While the 5 in the product name is a reference to the device’s nearly 5 inch screen, it’s also the 5th iteration of the Google Nexus line (the predecessors being the HTC Nexus One, Samsung Nexus S, Samsung Galaxy Nexus and the LG Nexus 4). The exterior of the Nexus 5 is made from a polycarbonate shell, unlike the Nexus 4, which used a glass-based construction. At 5.42 inches tall and 2.7 inches wide, it’s a big phone but is shaped to feel smaller than it looks. It’s surprisingly light for its size, at only 4.6oz, and is 8.6 millimeters thick. The phone feels a bit more solid than a Samsung Galaxy S4, but sitting next to an HTC One it looks a bit, well, plain. But being flashy or ostentatious was never Google’s goal with the Nexus line. It was to showcase the unbridled, unadulterated and bloatware free vanilla Google Android experience. And the phone’s 445 pixel per inch, 4.95-inch, 1080p IPS screen helps a great deal in doing that. At the time of this review the Nexus 5 was the only phone officially running Android’s latest version: Kit Kat. And that’s a big part of the Nexus experience and something no other phone is going to offer. Manufacturers often take many months to port new versions of Android to existing handsets and in some cases ports you think will come never do. Even the new Google Play edition of phones will likely never receive updates as quickly as the Nexus line. If that’s important to you, most of this review probably doesn’t matter. Get yourself a Nexus 5. It’s hands down the best Nexus phone to date. On that note, Kit Kat is the best Android version to date as well, and is a fairly significant change from previous versions of the software. It’s sleeker, cleaner, more refined and more modern looking while being considerably more responsive. Google Search and Google Now are integrated much more seamlessly than in previous versions. And while I’m not personally a fan of Hangouts replacing SMS and MMS, one nice thing about Android is that you can easily change that.

Now, back to the phone itself. Some good: The quad-core Snapdragon 800 processor with 2G of RAM means that the phone is astonishingly fast. By far the fastest phone I’ve used to date. The display is absolutely gorgeous. The battery life has also been better than most Android phones I’ve used. The  overall build quality of the phone is high and the form factor is extremely usable. The Nexus experience is also difficult to beat. Some bad: While battery life has been better, it’s still fairly unpredictable at times. The camera is probably the weakest part of the phone and is considerably worse than other flagship offerings. That said, Google claims that much of the issue is software related so we may see some marked improvement here. The speaker, while fairly loud, is also frustratingly distorted at times. While I like the overall form factor of the phone, it is quizzical that they chose to give it such a large bottom bezel, especially considering the phone has only software buttons. The lack of an SD card slot is also disappointing.

So, what’s the Verdict? If you want the Nexus experience or would like to buy an off contract phone, at $349 for the 16GB model and $399 for the 32GB model I think the Nexus 5 is going to be impossible to beat. I’m certainly extremely happy with the device myself. That said if you’re in a position where you have to buy a phone on contract, the HTC One (which I’ve seen as low as $75 with a 2 year contract) or possibly the Samsung Galaxy S4 are probably better options.

–jeremy
Google+


Categories: FLOSS Project Planets
Syndicate content