Planet Debian

Syndicate content
Planet Debian -
Updated: 6 hours 1 min ago

Russell Coker: SE Linux Play Machine Over Tor

10 hours 38 min ago

I work on SE Linux to improve security for all computer users. I think that my work has gone reasonably well in that regard in terms of directly improving security of computers and helping developers find and fix certain types of security flaws in apps. But a large part of the security problems we have at the moment are related to subversion of Internet infrastructure. The Tor project is a significant step towards addressing such problems. So to achieve my goals in improving computer security I have to support the Tor project. So I decided to put my latest SE Linux Play Machine online as a Tor hidden service. There is no real need for it to be hidden (for the record it’s in my bedroom), but it’s a learning experience for me and for everyone who logs in.

A Play Machine is what I call a system with root as the guest account with only SE Linux to restrict access.

Running a Hidden Service

A Hidden Service in TOR is just a cryptographically protected address that forwards to a regular TCP port. It’s not difficult to setup and the Tor project has good documentation [1]. For Debian the file to edit is /etc/tor/torrc.

I added the following 3 lines to my torrc to create a hidden service for SSH. I forwarded port 80 for test purposes because web browsers are easier to configure for SOCKS proxying than ssh.

HiddenServiceDir /var/lib/tor/hidden_service/
HiddenServicePort 22
HiddenServicePort 80

Generally when setting up a hidden service you want to avoid using an IP address that gives anything away. So it’s a good idea to run a hidden service on a virtual machine that is well isolated from any public network. My Play machine is hidden in that manner not for secrecy but to prevent it being used for attacking other systems.

SSH over Tor

Howtoforge has a good article on setting up SSH with Tor [2]. That has everything you need for setting up Tor for a regular ssh connection, but the tor-resolve program only works for connecting to services on the public Internet. By design the .onion addresses used by Hidden Services have no mapping to anything that reswemble IP addresses and tor-resolve breaks it. I believe that the fact that tor-resolve breaks thins in this situation is a bug, I have filed Debian bug report #776454 requesting that tor-resolve allow such things to just work [3].

Host *.onion
ProxyCommand connect -5 -S localhost:9050 %h %p

I use the above ssh configuration (which can go in ~/.ssh/config or /etc/ssh/ssh_config) to tell the ssh client how to deal with .onion addresses. I also had to install the connect-proxy package which provides the connect program.

ssh root@zp7zwyd5t3aju57m.onion
The authenticity of host ‘zp7zwyd5t3aju57m.onion ()
ECDSA key fingerprint is 3c:17:2f:7b:e2:f6:c0:c2:66:f5:c9:ab:4e:02:45:74.
Are you sure you want to continue connecting (yes/no)?

I now get the above message when I connect, the ssh developers have dealt with connecting via a proxy that doesn’t have an IP address.

Also see the general information page about my Play Machine, that information page has the root password [4].

Related posts:

  1. Trust and My SE Linux Play Machine When discussing the machine there are two common comments I...
  2. New SE Linux Play Machine Online After over a year I have finally got a SE...
  3. Play Machine Online Again I have returned from the US and my SE Linux...
Categories: FLOSS Project Planets

Laura Arjona: Upgrading my computers to Debian Jessie: Husband’s laptop (Acer Aspire 5250)

Tue, 2015-01-27 18:39

This is an old laptop, with AMD E-300 processor, 6 GB RAM, Radeon HD 6310 VGA and Atheros AR9485 wireless network adapter.

It was running Windows 7 (preinstalled). The hard disk failed, and I put the hard disk of another laptop (a broken Acer Aspire One D255) on it. Surprisingly, the Windows 7 on it booted (after some self-configuration that took quite long), but it was a Windows 7 Home 32bits, so it was only recognizing 4 GB RAM. That was the perfect excuse to convince my husband to install Debian in the laptop and begin his transition to a free OS. Yay!

I installed Debian Jessie from scratch last summer. Everything went well (the installer went fine, 8 months before than its RC1 release, congrats Debian-boot team!).

I needed the non-free radeon driver for the graphical display :/

Jessie is running GNOME3 desktop, and I’ve been seeing all these months the transition to 3.14 version, and later, the integration of the “Lines” theme (by Juliette Belin), which I like very much.

I have problems to watch high quality videos, in every player that I tried (VLC, Totem, mplayer) the audio and video are not synced, and video sometimes freezes. I’m almost sure that the problem is what mplayer says: “Your system is too SLOW to play this!”.

I tried to install the ATI non-free driver for better performance, but after successfully install it and reboot, GNOME was not starting (I got a black screen, no gdm greeting me). I could log in tty2, though. I don’t know if I did something wrong, how to solve the problem, and I don’t wanted to waste time, so I uninstalled it and returned to the non-free firmware that goes to the Linux kernel. For now, when I need to watch a video that gives those problems, I upload the file to my GNU MediaGoblin site, or use WinFF to reduce size/quality.

Overall impression

Fine! Both my husband and me are very happy.

The installation went really well.

I’m not a GNOME expert user but I find it easy, intuitive, and he found it easy too.

My husband uses the computer to surf the web, watch some videos and online series (we had to install non-free flash plugin from Adobe #grr), read mail from the browser, write something in LibreOffice and print it (hey! we just plug the printer/scanner and it works, no need to install drivers!), scan some image and send it by email… I set Debian as default in GRUB, and the switch from Windows has been very natural for him (he was already using Firefox and LibreOffice in Windows. He still says “I’m a Windows user” although he is just using Debian for months!).

He bought an IPhone 4S (#grr!) and I tried to connect it as shown in the corresponding Debian wiki page, but it didn’t work (I got “segmentation fault” when connecting the phone). However, it is recognized by Shotwell and we can copy all the photos and videos to the computer, which is what we wanted to do. So no problem on that side, either.

In conclussion, one more computer at home running Debian (“future stable”), and we don’t run Windows at home anymore :)

Filed under: My experiences and opinion Tagged: Debian, English, Moving into free software
Categories: FLOSS Project Planets

Laura Arjona: Upgrading my computers to Debian Jessie

Tue, 2015-01-27 17:40

Until now, I usually run Debian stable at work (in my desktop PC) and stable or testing at home in my laptop. I was upgrading to testing during the freeze, and then, stay in testing (future stable) or stable (when it’s published) until the next freeze.

I have changed this ‘conservative’ pattern. I’ve been running Jessie for many months now, and here I’ll document the different experiences in the computers that I use.

Upgrade or clean install?

I decided to upgrade my computers instead of making a clean install (except in the ones  that were not running Debian).

Although the upgrade process have been fine, I’m still not sure which is the best for my needs. Installing from the beginning forces me to re-read the feature list of the different pieces of software and choose the one that fits best (not the one that I was using some years ago). And maybe I just don’t need that non-free driver anymore because there’s free replacement already, the installer is wise. OTOH, upgrading is easier and quicker, and I got all my software and configurations (and my rubbish) there, nothing is lost.

The computers

Here I will link the blog posts of each computer that I upgrade, when I finish writing the corresponding articles:

  • Husband’s laptop (Acer 5250): Clean install – Done, and OK!
  • My laptop (Compaq Mini 110c): Upgrade – Done and OK!
  • Home server (HP Microserver N54L G7): Upgrade – Done and OK!
  • PC at work (motherboard Asus P5KPL-AM-SE): Upgrade – Done, some issues.
  • Mini-laptop Airis Kira N7000 (ARM board, 128MB RAM) – Clean install – Pending

Filed under: My experiences and opinion Tagged: Debian, English, Moving into free software
Categories: FLOSS Project Planets

Matthias Klumpp: AppStream 0.8 released!

Tue, 2015-01-27 11:48

Yesterday I released version 0.8 of AppStream, the cross-distribution standard for software metadata, that is currently used by GNOME-Software, Muon and Apper in to display rich metadata about applications and other software components.

 What’s new?

The new release contains some tweaks on AppStreams documentation, and extends the specification with a few more tags and refinements. For example we now recommend sizes for screenshots. The recommended sizes are the ones GNOME-Software already uses today, and it is a good idea to ship those to make software-centers look great, as others SCs are planning to use them as well. Normal sizes as well as sizes for HiDPI displays are defined. This change affects only the distribution-generated data, the upstream metadata is unaffected by this (the distro-specific metadata generator will resize the screenshots anyway).

Another addition to the spec is the introduction of an optional <source_pkgname/> tag, which holds the source package name the packages defined in <pkgname/> tags are built from. This is mainly for internal use by the distributor, e.g. it can decide to use this information to link to internal resources (like bugtrackers, package-watch etc.). It may also be used by software-center applications as additional information to group software components.

Furthermore, we introduced a <bundle/> tag for future use with 3rd-party application installation solutions. The tag notifies a software-installer about the presence of a 3rd-party application bundle, and provides the necessary information on how to install it. In order to do that, the software-center needs to support the respective installation solution. Currently, the Limba project and Xdg-App bundles are supported. For software managers, it is a good idea to implement support for 3rd-party app installers, as soon as the solutions are ready. Currently, the projects are worked on heavily. The new tag is currently already used by Limba, which is the reason why it depends on the latest AppStream release.

How do I get it?

All AppStream libraries, libappstream, libappstream-qt and libappstream-glib, are supporting the 0.8 specification in their latest version – so in case you are using one of these, you don’t need to do anything. For Debian, the DEP-11 spec is being updated at time, and the changes will land in the DEP-11 tools soon.

Improve your metadata!

This call goes especilly to many KDE projects! Getting good data is partly a task for the distributor, since packaging issues can result in incorrect or broken data, screenshots need to be properly resized etc. However, flawed upstream data can also prevent software from being shown, since software with broken data or missing data will not be incorporated in the distro XML AppStream data file.

Richard Hughes of Fedora has created a nice overview of software failing to be included. You can see the failed-list here – the data can be filtered by desktop environment etc. For KDE projects, a Comment= field is often missing in their .desktop files (or a <summary/> tag needs to be added to their AppStream upstream XML file). Keep in mind that you are not only helping Fedora by fixing these issues, but also all other distributions cosuming the metadata you ship upstream.

For Debian, we will have a similar overview soon, since it is also a very helpful tool to find packaging issues.

If you want to get more information on how to improve your upstream metadata, and how new metadata should look like, take a look at the quickstart guide in the AppStream documentation.

Categories: FLOSS Project Planets

Jingjie Jiang: Yet another post.

Tue, 2015-01-27 09:21
In the middle of OPW internship

I originally thought taking part in FOSSOPW is a great chance to lift my coding skills and I shouldn’t miss it. As time passes by, I now have a new thought towards it.

# 1 It do improves your coding skill.

Zack, Matthieu and I often have discussions on coding style. For example, once Zack said, “For code like this, you should explicitly use an if/else clause, not if-return.” I was totally unaware of this sort of issue. Actually I even didn’t know how I should call this problem. Matthieu gave me a detailed FYI link on this in no time.

Besides, my completeness of thinking is also trained. Recently, I was fixing a “HTTP GET Method ?suite=suite-name” issue. It’s a trivial task. And you know most trivial tasks require lots of scattered modification on the source code base. I did have fixed most places, such as the pages of “/src/packagename” and “/search/”. Zack did a thorough review, and pointed out that the pages rendered by “/prefix” has some malformed urls in the HTML. Waiiiit, I should have noticed it. But somehow I missed it. Maybe because my mind was wandering at that time? This made me think, I shall have a thorough view of what I should do before getting my hands dirty. Or more preferably, if I could write down what I exactly want to achieve before coding, then silly problems definitely wouldn’t occur. This may sound a little bit like TDD. ;).

# 2 It makes you look like a (not-that-good) ninja.

I use a macbook. It’s not my fault! I’ve tried several times, but I never successfully find a laptop that is not capable of boiling eggs when running Debian. (and especially KDE+Debian). So I have no way but switched to OSX. The development of Debsources happens on a remote Ubuntu LTS (now Debian SID, haha) virtual machine. Of course I have to install all the dependency on my own, e.g., Postgres, set up port-forwarding, e.g., ssh -D, write automate shell scripts, e.g., dash, but more importantly, I am forced to live under the dark terminal with no GUI. You know the feeling when pain hurts? Yes, exactly! But I survived. How shall I call myself now? A dedicated with-a-lot-of-useless-plugin-installed vimmer? A fond-of-fancy-window tmux-er? Yep, both. I finally found a comfort zone under the black-white-blinking screen. I wonder how people feel when they see a girl hanging out in the library, facing a full-screened black console, typing at a speed of 140wpm (Yeah, I am kidding). I don’t know, but please don’t call me a geek. Show me your respect, I am a ninja!

# 3 It tells you communication is the most important.

I bet anyone who has participated in a group-based project would understand what I mean. For one perspective, communication helps to eliminate misunderstanding. So I won’t doing some useless stuff for all day and finally find out that it totally doesn’t meet the requirement. On the other hand, it speeds up your learning process. I often have problems on git. So in the email I will complain if I mess up with the git repo. After a short while, my dear mentors will reply in detail on how to correctly do the git stuff.

My OPW journey is cool! ;).
Categories: FLOSS Project Planets

Thomas Goirand: OpenStack debian image available from

Tue, 2015-01-27 07:30

About a year and a half after I started writing the openstack-debian-images package, I’m very happy to announce to everyone that, thanks to Steve McIntyre’s help, the official OpenStack Debian image is now generated at the same time as the official Debian CD ISO images. If you are a cloud user, if you use OpenStack on a private cloud, or if you are a public cloud operator, then you may want to download the weekly build of the OpenStack image from here:

Note that for the moment, there’s only the amd64 arch available, but I don’t think this is a problem: so far, I haven’t found any public cloud provider offering anything else than Intel 64 bits arch. Maybe this will change over the course of this year, and we will need arm64, but this can be added later on.

Now, for later plans: I still have 2 bugs to fix on the openstack-debian-images package (the default 1GB size is now just a bit too small for Jessie, and the script exits with zero in case of error), but nothing that prevents its use right now. I don’t think it will be a problem for the release team to accept these small changes before Jessie is out.

When generating the image, Steve also wants to generate a sources.tar.gz containing all the source packages that we include on the image. He already has the script (which is used as a hook script when running the build-openstack-debian-image script), and I am planning to add it as a documentation in /usr/share/doc/openstack-debian-images.

Last, probably it would be a good idea to install grub-xen, just as Ian Campbell suggested to make it possible for this image to run in AWS or other Xen based clouds. I would need to be able to test this though. If you can contribute with this kind of test, please get in touch.

Feel free to play with all of this, and customize your Jessie images if you need to. The script is (on purpose) very small (around 400 lines of shell script) and easy to understand (no function, it’s mostly linear from top to bottom of the file), so it is also very easy to hack, plus it has a convenient hook script facility where you can do all sorts of things (copying files, apt-get install stuff, running things in the chroot, etc.).

Again, thanks so much to Steve for working on using the script during the CD builds. This feels me with joy that Debian finally has official images for OpenStack.

Categories: FLOSS Project Planets

Steve Kemp: Recording gym-visits on Linux.

Mon, 2015-01-26 19:00

I go to the gym every couple of days. I lift things up, then put them down, and sometimes I repeat this process another 30 times. When I'm done I write down what I've done, how many times I did the lifty-droppy thing, and so on.

I want to see pretty graphs. I want to have records of different things. I guess I just need some simple text-boxes:

deadlift 3 x 7 @ 210lbs.

etc. Sometimes I use machines so I'd say instead:

converging seated-row 3 x 8 @ 150lbs

Anyway that's it. I want a simple GUI, a bit like a spreadsheet where I can easily add rows of each session. (A session might have 10-15 exercises in it, so not many.) I imagine some kind of SQLite database for the back-end. Or CSV. Either works.

Writing GUI software is hard. I guess I should look at GtK or Qt over the next few days and see if it is going to be easier to do it online via a jQuery + CGI system instead. To be honest I expect doing it "online" is liable to be more popular, but I think a desktop toy-application is just as useful.

Categories: FLOSS Project Planets

Daniel Pocock: Get your Nagios issues as an iCalendar feed

Mon, 2015-01-26 16:37

The other day I demonstrated how to get your Github issues/bugs as an iCalendar feed.

I'm planning to take this concept further and I just whipped up another Python script, exposing Nagios issues as an iCalendar feed.

The script is nagios-icalendar. Usage is explained concisely in the README file, it takes just minutes to get up and running.

One interesting feature is that you can append a contact name to the URL and just get the issues for that contact, e.g.: Screenshots

Here I demonstrate using Mozilla Lightning / Iceowl-extension to aggregate issues from Nagios, the Fedora instance of Bugzilla and Lumicall's Github issues into a single to-do list.

Categories: FLOSS Project Planets

Vincent Fourmond: Linux kernels for a macbook pro retina

Mon, 2015-01-26 15:47
I was unhappy about the recent Linux (3.14-3.16, and I think 3.17 too) kernels on my Macbook Pro Retina (15'), for a few reasons:

  • the nouveau graphics driver was not handling the graphics card very well (hangs when using the DRM after putting the computer to sleep once, garbage screen on various apps, slow 3D rendering), and I could never get the proprietary nvidia drivers to work (would give blank screen at boot time)
  • very unstable wireless (at least with my box home, but not with all the ones I've tried)
  • and the most painful was the need to recompile the kernel by hand, with the following modifications from stock debian kernel:
    -CONFIG_X86_SYSFB=y +# CONFIG_X86_SYSFB is not set -CONFIG_FB_SIMPLE=y +# CONFIG_FB_SIMPLE is not set Without these modifications, the screen would be garbled some 5-6 seconds after boot (but SSH would still work, as far as I remember).
The latest 3.18-trunk kernel fixes essentially all the above problems, which is just great. Kudos to everyone involved ! Hope it helps...
Categories: FLOSS Project Planets

Tanguy Ortolo: Scale manufacturers…

Mon, 2015-01-26 08:54

Dear manufacturers of kitchen scales, could you please stop considering your clients as idiots, and start developing useful features?

Liquid measurement: this is one feature that is available on almost every electronic scale available. Except it is completely useless to people that use the metric system, as all it does is replace the usual display in grammes by centilitres and divide the number on display by ten. Thank you, but no person that has been to school in a country that uses the metric system needs electronic assistance to determine the volume corresponding to a given weight of water, and for people that have not, a simple note written on the scale, stating that “for water or milk, divide the weight in grammes by ten to get the volume in centilitres” should be enough.

Now, there is still one thing that an electronic scale could be useful for, which is determining the volume of liquids other than water (density 1 g/ml) or milk (density approx. equal to 1 g/ml), most importantly: oil (density approx. equal to .92 g/ml for edible oils like sunflower, peanut, olive and canola).

Categories: FLOSS Project Planets

Francois Marier: Using unattended-upgrades on Rackspace's Debian and Ubuntu servers

Mon, 2015-01-26 03:25

I install the unattended-upgrades package on almost all of my Debian and Ubuntu servers in order to ensure that security updates are automatically applied. It works quite well except that I still need to login manually to upgrade my Rackspace servers whenever a new rackspace-monitoring-agent is released because it comes from a separate repository that's not covered by unattended-upgrades.

It turns out that unattended-upgrades can be configured to automatically upgrade packages outside of the standard security repositories but it's not very well documented and the few relevant answers you can find online are still using the old whitelist syntax.

Initial setup

The first thing to do is to install the package if it's not already done:

apt-get install unattended-upgrades

and to answer yes to the automatic stable update question.

If you don't see the question (because your debconf threshold is too low -- change it with dpkg-reconfigure debconf), you can always trigger the question manually:

dpkg-reconfigure -plow unattended-upgrades

Once you've got that installed, the configuration file you need to look at is /etc/apt/apt.conf.d/50unattended-upgrades.

Whitelist matching criteria

Looking at the unattended-upgrades source code, I found the list of things that can be used to match on in the whitelist:

  • origin (shortcut: o)
  • label (shortcut: l)
  • archive (shortcut: a)
  • suite (which is the same as archive)
  • component (shortcut: c)
  • site (no shortcut)

You can find the value for each of these fields in the appropriate _Release file under /var/lib/apt/lists/.

Note that the value of site is the hostname of the package repository, also present in the first part these *_Release filenames ( in the example below).

In my case, I was looking at the following inside /var/lib/apt/lists/stable.packages.cloudmonitoring.rackspace.com_debian-wheezy-x86%5f64_dists_cloudmonitoring_Release:

Origin: Rackspace Codename: cloudmonitoring Date: Fri, 23 Jan 2015 18:58:49 UTC Architectures: i386 amd64 Components: main ...

which means that, in addition to site, the only things I could match on were origin and component since there are no Suite or Label fields in the Release file.

This is the line I ended up adding to my /etc/apt/apt.conf.d/50unattended-upgrades:

Unattended-Upgrade::Origins-Pattern { // Archive or Suite based matching: // Note that this will silently match a different release after // migration to the specified archive (e.g. testing becomes the // new stable). // "o=Debian,a=stable"; // "o=Debian,a=stable-updates"; // "o=Debian,a=proposed-updates"; "origin=Debian,archive=stable,label=Debian-Security"; "origin=Debian,archive=oldstable,label=Debian-Security"; + "origin=Rackspace,component=main"; }; Testing

To ensure that the config is right and that unattended-upgrades will pick up rackspace-monitoring-agent the next time it runs, I used:

unattended-upgrade --dry-run --debug

which should output something like this:

Initial blacklisted packages: Starting unattended upgrades script Allowed origins are: ['origin=Debian,archive=stable,label=Debian-Security', 'origin=Debian,archive=oldstable,label=Debian-Security', 'origin=Rackspace,component=main'] Checking: rackspace-monitoring-agent (["<Origin component:'main' archive:'' origin:'Rackspace' label:'' site:'' isTrusted:True>"]) pkgs that look like they should be upgraded: rackspace-monitoring-agent ... Option --dry-run given, *not* performing real actions Packages that are upgraded: rackspace-monitoring-agent Making sure that automatic updates are happening

In order to make sure that all of this is working and that updates are actually happening, I always install apticron on all of the servers I maintain. It runs once a day and emails me a list of packages that need to be updated and it keeps doing that until the system is fully up-to-date.

The only thing missing from this is getting a reminder whenever a package update (usually the kernel) requires a reboot to take effect. That's where the update-notifier-common package comes in.

Because that package will add a hook that will create the /var/run/reboot-required file whenever a kernel update has been installed, all you need to do is create a cronjob like this in /etc/cron.daily/reboot-required:

#!/bin/sh cat /var/run/reboot-required 2> /dev/null || true

assuming of course that you are already receiving emails sent to the root user (if not, add the appropriate alias in /etc/aliases and run newaliases).

Categories: FLOSS Project Planets

NOKUBI Takatsugu: Weak ssh public keys in github

Mon, 2015-01-26 00:15

A presentation slide, named ”Attacking against 5 millions SSH public keys – 偶然にも500万個のSSH公開鍵を手に入れた俺たちは” is published, it is a lightning talk in “Edomae security seminar” in Jan 24, 2015.

 He grabbed ssh public keys with  GitHub API (${user}.key), the API is obsoleted, but not closed.

He found short (<= 512 bit) DSA/RSA keys and can solve prime decomposition 256bit RSA key in 3 seconds.

And he repoted there are 208 weak ssh keys generated by Debian/Ubuntu (CVE-2008-0166). It was already announced  by GitHub.

On the other hand, such ssh keys couldn’t solve prime decomposition with fastgcd. It means almost ssh keys in GitHub has no bias in almost random number generators implementations, it is a good news.

Categories: FLOSS Project Planets

Richard Hartmann: KDE battery monitor

Sun, 2015-01-25 16:11

Dear lazyweb,

using a ThinkPad X1 Carbon with Debian unstable and KDE 4.14.2, I have not had battery warnings for a few weeks, now.

The battery status can be read out via acpi -V as well as via the KDE widget. Hibernation via systemctl hibernate works as well.

What does not work is the warning when my battery is low, or automagic hibernation when shutting the lid or when the battery level is critical.

From what I gather, something in the communication between upower and KDE broke down, but I can't find what it is. I have also been told that Cinnamon is affected as well, so this seems to be a more general problem

Sadly, me and anyone else who's affected has been unable to fix this.

So, dear lazyweb, please help.

In loosely related news, this old status is still valid. UMTS is stable-ish now but even though I saved the SIM's PIN, KDE always displays a "SIM PIN unlock request" prompt after booting or hibernating. Once I enter that PIN, systemd tells me that a system policy prevents the change and wants my user password. If anyone knows how to get rid of that, I would also appreciate any pointers.

Categories: FLOSS Project Planets

Chris Lamb: Recent Redis hacking

Sun, 2015-01-25 15:52

I've done a bunch of hacking on the Redis key/value database server recently:

  • Lua-based maxmemory eviction scripts. (#2319)

    (This changeset was sponsored by an anonymous client.)

    Redis typically stores the entire data set in memory, using the operating system's virtual memory facilities if required. However, one can use Redis more like a cache or ring buffer by enabling a "maxmemory policy" where a RAM limit is set and then data is evicted when required based on a predefined algorithm.

    This change enables entirely custom control over exactly what data to remove from RAM when this maxmemory limit is reached. This is an advantage over the existing policies of, say, removing entire keys based on the existing TTL, Least Recently Used (LRU) or random eviction strategies as it permits bespoke behaviours based on application-specific requirements, crucially without maintaining a private fork of Redis.

    As an example behaviour of what is possible with this change, to remove the lowest ranked member of an arbitrary sorted set, you could load the following eviction policy script:

    local bestkey = nil local bestval = 0 for s = 1, 5 do local key ="RANDOMKEY") local type_ ="TYPE", key) if type_.ok == "zset" then local tail ="ZRANGE", key, "0", "0", "WITHSCORES") local val = tonumber(tail[2]) if not bestkey or val < bestval then bestkey = key bestval = val end end end if not bestkey then -- We couldn't find anything to remove, so return an error return false end"ZREMRANGEBYRANK", bestkey, "0", "0") return true
  • TCP_FASTOPEN support. (#2307)

    The aim of TCP_FASTOPEN is to eliminate one roundtrip from a TCP conversation by allowing data to be included as part of the SYN segment that initiates the connection. (More info.)

  • Support infinitely repeating commands in redis-cli. (#2297)

  • Add --failfast option to testsuite runner. (#2290)

  • Add a -q (quiet) argument to redis-cli. (#2305)

  • Making some Redis Sentinel defaults a little saner. (#2292)

I also made the following changes to the Debian packaging:

  • Add run-parts(8) directories to be executed at various points in the daemon's lifecycle. (e427f8)

    This is especially useful for loading Lua scripts as they are not persisted across restarts.

  • Split out Redis Sentinel into its own package. (#775414, 39f642)

    This makes it possible to run Sentinel sanely on Debian systems without bespoke scripts, etc.

  • Ensure /etc/init.d/redis-server start idempotency with --oknodo (60b7dd)

    Idempotency in initscripts is especially important given the rise of configuration managment systems.

  • Uploaded 3.0.0 RC2 to Debian experimental. (37ac55)

  • Re-enabled the testsuite. (7b9ed1)

Categories: FLOSS Project Planets

Dirk Eddelbuettel: RcppArmadillo 0.4.600.4.0

Sun, 2015-01-25 15:19

Conrad put up a maintenance release 4.600.4 of Armadillo a few days ago. As in the past, we tested this with number of pre-releases and test builds against the now over one hundred CRAN dependents of our RcppArmadillo package. The tests passed fine as usual, and results are as always in the rcpp-logs repository.

Changes are summarized below based on the NEWS.Rd file.

Changes in RcppArmadillo version 0.4.600.4.0 (2015-01-23)
  • Upgraded to Armadillo release Version 4.600.4 (still "Off The Reservation")

    • Speedups in the transpose operation

    • Small bug fixes

Courtesy of CRANberries, there is also a diffstat report for the most recent release. As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Jonathan Dowland: Frontier: First Encounters

Sun, 2015-01-25 08:18

Cobra mk. 3

Four years ago, whilst looking for something unrelated, I stumbled across Tom Morton's port of "Frontier: Elite II" for the Atari to i386/OpenGL. This took me right back to playing Frontier on my Amiga in the mid-nineties. I spent a bit of time replaying Frontier and its sequel, First Encounters, for which there exists an interesting family of community-written game engines based on a reverse-engineering of the original DOS release.

I made some scrappy notes about engines, patches etc. at the time, which are on my frontier page.

With the recent release of Elite: Dangerous, I thought I'd pick up where I left in 2010 and see if I could get the Thargoid ship. I'm nowhere near yet, but I've spent some time trying to maximize income during the game's initial Soholian Fever period. My record in a JJFFE-derived engine (and winning the Wiccan Ware race during the same period) is currently £727,800. Can you do better?

Categories: FLOSS Project Planets

Joey Hess: making propellor safer with GADTs and type families

Sat, 2015-01-24 22:54

Since July, I have been aware of an ugly problem with propellor. Certain propellor configurations could have a bug. I've tried to solve the problem at least a half-dozen times without success; it's eaten several weekends.

Today I finally managed to fix propellor so it's impossible to write code that has the bug, bending the Haskell type checker to my will with the power of GADTs and type-level functions.

the bug

Code with the bug looked innocuous enough. Something like this:

foo :: Property foo = property "foo" $ unlessM (liftIO $ doesFileExist "/etc/foo") $ do bar <- liftIO $ readFile "/etc/foo.template" ensureProperty $ setupFoo bar

The problem comes about because some properties in propellor have Info associated with them. This is used by propellor to introspect over the properties of a host, and do things like set up DNS, or decrypt private data used by the property.

At the same time, it's useful to let a Property internally decide to run some other Property. In the example above, that's the ensureProperty line, and the setupFoo Property is run only sometimes, and is passed data that is read from the filesystem.

This makes it very hard, indeed probably impossible for Propellor to look inside the monad, realize that setupFoo is being used, and add its Info to the host.

Probably, setupFoo doesn't have Info associated with it -- most properties do not. But, it's hard to tell, when writing such a Property if it's safe to use ensureProperty. And worse, setupFoo could later be changed to have Info.

Now, in most languages, once this problem was noticed, the solution would probably be to make ensureProperty notice when it's called on a Property that has Info, and print a warning message. That's Good Enough in a sense.

But it also really stinks as a solution. It means that building propellor isn't good enough to know you have a working system; you have to let it run on each host, and watch out for warnings. Ugh, no!

the solution

This screams for GADTs. (Well, it did once I learned how what GADTs are and what they can do.)

With GADTs, Property NoInfo and Property HasInfo can be separate data types. Most functions will work on either type (Property i) but ensureProperty can be limited to only accept a Property NoInfo.

data Property i where IProperty :: Desc -> ... -> Info -> Property HasInfo SProperty :: Desc -> ... -> Property NoInfo data HasInfo data NoInfo ensureProperty :: Property NoInfo -> Propellor Result

Then the type checker can detect the bug, and refuse to compile it.


Except ...

Property combinators

There are a lot of Property combinators in propellor. These combine two or more properties in various ways. The most basic one is requires, which only runs the first Property after the second one has successfully been met.

So, what's it's type when used with GADT Property?

requires :: Property i1 -> Property i2 -> Property ???

It seemed I needed some kind of type class, to vary the return type.

class Combine x y r where requires :: x -> y -> r

Now I was able to write 4 instances of Combines, for each combination of 2 Properties with HasInfo or NoInfo.

It type checked. But, type inference was busted. A simple expression like

foo `requires` bar

blew up:

No instance for (Requires (Property HasInfo) (Property HasInfo) r0) arising from a use of `requires' The type variable `r0' is ambiguous Possible fix: add a type signature that fixes these type variable(s) Note: there is a potential instance available: instance Requires (Property HasInfo) (Property HasInfo) (Property HasInfo) -- Defined at Propellor/Types.hs:167:10

To avoid that, it needed ":: Property HasInfo" appended -- I didn't want the user to need to write that.

I got stuck here for an long time, well over a month.

type level programming

Finally today I realized that I could fix this with a little type-level programming.

class Combine x y where requires :: x -> y -> CombinedType x y

Here CombinedType is a type-level function, that calculates the type that should be used for a combination of types x and y. This turns out to be really easy to do, once you get your head around type level functions.

type family CInfo x y type instance CInfo HasInfo HasInfo = HasInfo type instance CInfo HasInfo NoInfo = HasInfo type instance CInfo NoInfo HasInfo = HasInfo type instance CInfo NoInfo NoInfo = NoInfo type family CombinedType x y type instance CombinedType (Property x) (Property y) = Property (CInfo x y)

And, with that change, type inference worked again! \o/

(Bonus: I added some more intances of CombinedType for combining things like RevertableProperties, so propellor's property combinators got more powerful too.)

Then I just had to make a massive pass over all of Propellor, fixing the types of each Property to be Property NoInfo or Property HasInfo. I frequently picked the wrong one, but the type checker was able to detect and tell me when I did.

A few of the type signatures got slightly complicated, to provide the type checker with sufficient proof to do its thing...

before :: (IsProp x, Combines y x, IsProp (CombinedType y x)) => x -> y -> CombinedType y x before x y = (y `requires` x) `describe` (propertyDesc x) onChange :: (Combines (Property x) (Property y)) => Property x => Property y => CombinedType (Property x) (Property y) onChange = -- 6 lines of code omitted fallback :: (Combines (Property p1) (Property p2)) => Property p1 -> Property p2 -> Property (CInfo p1 p2) fallback = -- 4 lines of code omitted

.. This mostly happened in property combinators, which is an acceptable tradeoff, when you consider that the type checker is now being used to prove that propellor can't have this bug.

Mostly, things went just fine. The only other annoying thing was that some things use a [Property], and since a haskell list can only contain a single type, while Property Info and Property NoInfo are two different types, that needed to be dealt with. Happily, I was able to extend propellor's existing (&) and (!) operators to work in this situation, so a list can be constructed of properties of several different types:

propertyList "foos" $ props & foo & foobar ! oldfoo conclusion

The resulting 4000 lines of changes will be in the next release of propellor. Just as soon as I test that it always generates the same Info as before, and perhaps works when I run it. (eep)

These uses of GADTs and type families are not new; this is merely the first time I used them. It's another Haskell leveling up for me.

Anytime you can identify a class of bugs that can impact a complicated code base, and rework the code base to completely avoid that class of bugs, is a time to celebrate!

Categories: FLOSS Project Planets

Daniel Pocock: Get your Github issues as an iCalendar feed

Sat, 2015-01-24 18:07

I've just whipped up a Python script that renders Github issue lists from your favourite projects as an iCalendar feed.

The project is called github-icalendar. It uses Python Flask to expose the iCalendar feed over HTTP.

It is really easy to get up and running. All the dependencies are available on a modern Linux distribution, for example:

$ sudo apt-get install python-yaml python-icalendar python-flask python-pygithub

Just create an API token in Github and put it into a configuration file with a list of your repositories like this:

api_token: 6b36b3d7579d06c9f8e88bc6fb33864e4765e5fac4a3c2fd1bc33aad bind_address: ::0 bind_port: 5000 repositories: - repository: your-user-name/your-project - repository: your-user-name/another-project

Run it from the shell:

$ ./github_icalendar/ github-ics.cfg

and connect to it with your favourite iCalendar client.

Consolidating issue lists from Bugzilla, Github, Debian BTS and other sources

A single iCalendar client can usually support multiple sources and thereby consolidate lists of issues from multiple bug trackers.

This can be much more powerful than combining RSS bug feeds because iCalendar has built-in support for concepts such as priority and deadline. The client can use these to help you identify the most critical issues across all your projects, no matter which bug tracker they use.

Bugzilla bugtrackers already expose iCalendar feeds directly, just look for the iCalendar link at the bottom of any search results page. Here is an example URL from the Mozilla instance of Bugzilla.

The Ultimate Debian Database consolidates information from the Debian and Ubuntu universe and can already export it as an RSS feed, there is discussion about extrapolating that to an iCalendar feed too.

Further possibilities
  • Prioritizing the issues in Github and mapping these priorities to iCalendar priorities
  • Creating tags in Github that allow issues to be ignored/excluded from the feed (e.g. excluding wishlist items)
  • Creating summary entries instead of listing all the issues, e.g. a single task entry with the title Fix 2 critical bugs for project foo

The screenshots below are based on the issue list of the Lumicall secure SIP phone for Android.

Screenshot - Mozilla Thunderbird/Lightning (Icedove/Iceowl-extension on Debian)

Categories: FLOSS Project Planets

Dirk Eddelbuettel: Rcpp 0.11.4

Sat, 2015-01-24 10:44

A new release 0.11.4 of Rcpp is now on the CRAN network for GNU R, and an updated Debian package will be uploaded in due course.

Rcpp has become the most popular way of enhancing GNU R with C++ code. As of today, 323 packages on CRAN depend on Rcpp for making analyses go faster and further; BioConductor adds another 41 packages, and casual searches on GitHub suggests dozens mores.

This release once again adds a large number of small bug fixes, polishes and enhancements. And like the last time, these changes were made by a group of seven different contributors (counting code commits) plus three more providing concrete suggestions. This shows that the Rcpp development and maintenance rests a large number of (broad) shoulders.

See below for a detailed list of changes extracted from the NEWS file.

Changes in Rcpp version 0.11.4 (2015-01-20)
  • Changes in Rcpp API:

    • The ListOf<T> class gains the .attr and .names methods common to other Rcpp vectors.

    • The [dpq]nbinom_mu() scalar functions are now available via the R:: namespace when R 3.1.2 or newer is used.

    • Add an additional test for AIX before attempting to include execinfo.h.

    • Rcpp::stop now supports improved printf-like syntax using the small tinyformat header-only library (following a similar implementation in Rcpp11)

    • Pairlist objects are now protected via an additional Shield<> as suggested by Martin Morgan on the rcpp-devel list.

    • Sorting is now prohibited at compile time for objects of type List, RawVector and ExpressionVector.

    • Vectors now have a Vector::const_iterator that is 'const correct' thanks to fix by Romain following a bug report in rcpp-devel by Martyn Plummer.

    • The mean() sugar function now uses a more robust two-pass method, and new unit tests for mean() were added at the same time.

    • The mean() and var() functions now support all core vector types.

    • The setequal() sugar function has been corrected via suggestion by Qiang Kou following a bug report by Søren Højsgaard.

    • The macros major, minor, and makedev no longer leak in from the (Linux) system header sys/sysmacros.h.

    • The push_front() string function was corrected.

  • Changes in Rcpp Attributes:

    • Only look for plugins in the package's namespace (rather than entire search path).

    • Also scan header files for definitions of functions to be considerd by Attributes.

    • Correct the regular expression for source files which are scanned.

  • Changes in Rcpp unit tests

    • Added a new binary test which will load a pre-built package to ensure that the Application Binary Interface (ABI) did not change; this test will (mostly or) only run at Travis where we have reasonable control over the platform running the test and can provide a binary.

    • New unit tests for sugar functions mean, setequal and var were added as noted above.

  • Changes in Rcpp Examples:

    • For the (old) examples ConvolveBenchmarks and OpenMP, the respective Makefile was renamed to GNUmakefile to please R CMD check as well as the CRAN Maintainers.

Thanks to CRANberries, you can also look at a diff to the previous release As always, even fuller details are on the Rcpp Changelog page and the Rcpp page which also leads to the downloads page, the browseable doxygen docs and zip files of doxygen output for the standard formats. A local directory has source and documentation too. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets

Dirk Eddelbuettel: RcppGSL 0.2.4

Sat, 2015-01-24 10:20

A new version of RcppGSL is now on CRAN. This package provides an interface from R to the GNU GSL using our Rcpp package.

This follows on the heels on the recent RcppGSL 0.2.3 release and extends the excellent point made by Qiang Kou in a contributed section of the vignette: We now not only allow to turn the GSL error handler off (to not abort() on error) but do so on package initialisation.

No other user-facing changes were made.

The NEWS file entries follows below:

Changes in version 0.2.4 (2015-01-24)
  • Two new helper function to turn the default GSL error handler off (and to restore it) were added. The default handler is now turned off when the package is attached so that GSL will no longer abort an R session on error. Users will have to check the error code.

  • The RcppGSL-intro.Rnw vignette was expanded with a short section on the GSL error handler (thanks to Qiang Kou).

Courtesy of CRANberries, a summary of changes to the most recent release is available.

More information is on the RcppGSL page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

This post by Dirk Eddelbuettel originated on his Thinking inside the box blog. Please report excessive re-aggregation in third-party for-profit settings.

Categories: FLOSS Project Planets