FOSDEM 2012 was very nice, as expected.

This year, I decided to stop consuming only and getting more involved. I helped staff the token sale and held two Lightning Talks. This turned out to be an awesome experience.

During the beer event, we sold 4600 beer tokens (FOSDEM does not earn anything from this; all proceeds go to Cafe Delirium). The sale was staffed with one to three people at all times which made for a constant but mostly bearable workload. While the others made good-natured fun of me and my German efficiency, I like to think that I helped make things run a tad more smoothly. Along those lines, we will have a sign asking people for exact change and/or to buy stacks that can be paid in bills if possible, next year; that should help streamline the whole thing even more. All in all, it turned out to be quite stressful, but tons of fun. I am looking forward to chuck in again next year.

The two Lighting Talks (vcsh on Saturday and git-annex on Sunday) I held very rather well received. Feedback after the talks and online was very positive which is always nice. There's been a slight increase in help requests and generic questions, so I assume people are trying vcsh and git-annex as a consequence of those talks, which is even nicer.

Also, I met a lot of people. Most of the meetings were pre-arranged, some by chance. Carrying a name tag at all times, I was approached by several people who knew me online but didn't know my face. I will try to always carry a name tag on conferences now and I suggest others do the same.

Finally, a huge thank you to everyone who helped make FOSDEM happen; you rock!

If you ever get this cryptic error when loading an Android native app:

FATAL EXCEPTION: main java.lang.RuntimeException: Unable to start activity ComponentInfo{org.wikibooks.OpenGL/android.app.NativeActivity}: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1768) at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:1784) at android.app.ActivityThread.access$1500(ActivityThread.java:123) at android.app.ActivityThread$H.handleMessage(ActivityThread.java:939) at android.os.Handler.dispatchMessage(Handler.java:99) at android.os.Looper.loop(Looper.java:130) at android.app.ActivityThread.main(ActivityThread.java:3835) at java.lang.reflect.Method.invokeNative(Native Method) at java.lang.reflect.Method.invoke(Method.java:507) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:847) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:605) at dalvik.system.NativeStart.main(Native Method) Caused by: java.lang.IllegalArgumentException: Unable to load native library: /data/data/org.wikibooks.OpenGL/lib/libnative-activity.so at android.app.NativeActivity.onCreate(NativeActivity.java:199) at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1047) at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:1722) ... 11 more

This may mean that Java couldn't find the ANativeActivity_onCreate function in your code, because it was stripped by the compiler.

If you use the native_app_glue NDK module, you may have noticed this strange code:

// Make sure glue isn't stripped. app_dummy();

Let's experiment what happens with and without this line:

Calling app_dummy:

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so | grep ANativeActivity_onCreate 000067fc g DF .text 000000f8 ANativeActivity_onCreate

Not calling app_dummy :

$ arm-linux-androideabi-objdump -T libs/armeabi/libnative-activity.so | grep ANativeActivity_onCreate $ # nothing

native_app_glue mainly defines Android callbacks. Since none of them are called directly by your code, the compiler strips the android_native_app_glue.o module entirely. If you use app_dummy however, it embeds it. Fortunately the compiler cannot strip the module on a per-function basis

That's why you need to call app_dummy when using the native_app_glue NDK module.

This looks like a ugly work-around though - isn't there a cleaner way?

Part of our efforts to reduce power consumption is to identify processes which keep waking up the disk even when the computer is idle. This already resulted in a few bug reports (and some fixes, too), but we only really just began with this.

Unfortunately there is no really good tool to trace file access events system-wide. powertop claims to, but its output is both very incomplete, and also wrong (e. g. it claims that read accesses are writes). strace gives you everything you do and don’t want to know about what’s going on, but is per-process, and attaching strace to all running and new processes is cumbersome. blktrace is system-wide, but operates at a way too low level for this task: its output has nothing to do any more with files or even inodes, just raw block numbers which are impossible to convert back to an inode and file path.

So I created a little tool called fatrace (“file access trace”, not “fat race” ) which uses fanotify, a couple of /proc lookups and some glue to provide this. By default it monitors the whole system, i. e. all mounts (except the virtual ones like /proc, tmpfs, etc.), but you can also tell it to just consider the mount of the current directory. You can write the log into a file (stdout by default), and run it for a specified number of seconds. Optional time stamps and PID filters are also provided.

$ sudo fatrace rsyslogd(967): W /var/log/auth.log notify-osd(2264): O /usr/share/pixmaps/weechat.xpm compiz(2001): R device 8:2 inode 658203 [...]

It shows the process name and pid, the event type (Rread, Write, Open, or Close), and the path. Sometimes its’ not possible to determine a path (usually because it’s a temporary file which already got deleted, and I suspect mmaps as well), in that case it shows the device and inode number; such programs then need closer inspection with strace.

If you run this in gnome-terminal, there is an annoying feedback loop, as gnome-terminal causes a disk access with each output line, which then causes another output line, ad infinitum. To fix this, you can either redirect output to a file (-o /tmp/trace) or ignore the PID of gnome-terminal (-p `pidof gnome-terminal`).

So to investigate which programs are keeping your disk spinning, run something like

$ sudo fatrace -o /tmp/trace -s 60

and then do nothing until it finishes.

My next task will be to write an integration program which calls fatrace and powertop, and creates a nice little report out of that raw data, sorted by number of accesses and process name, and all that. But it might already help some folks as it is right now.

The code lives in bzr branch lp:fatrace (web view), you can just run make and sudo ./fatrace. I also uploaded a package to Ubuntu Precise, but it still needs to go through the NEW queue. I also made a 0.1 release, so you can just grab the release tarball if you prefer. Have a look at the manpage and --help, it should be pretty self-explanatory.

I've just updated phpMyAdmin repos on github and enabled notifications hooks there, so the earlier announced switch to github is done.

All you need to do is point your repositories to pull/push to github, for main repository it can be done using:

git remote set-url origin git@github.com:phpmyadmin/phpmyadmin.git

If you are using just read only access then use following:

git remote set-url origin git://github.com/phpmyadmin/phpmyadmin.git

For other repositories just replace last part of the URL with repository name (they have not changed).

Everything should work as before, pushes should be now faster, because all notifications are done in background on github and they don't block pushing.

Filed under: English Phpmyadmin | 1 comments | Flattr this!

A year ago, during FOSDEM 2011, I walked up to the NamurLUG folks who'd been doing video coverage of FOSDEM for years, and suggested that this year, maybe they should consider using dvswitch for video coverage. While they seemed agreeable to this idea, they simply had not had the time to look at it in detail, and were therefore not using it. Since I've used dvswitch in the past, both as part of the DebConf video team and for my own concert recordings, I reasoned this a problem that could be solved by joining the video team.

However, when it came to be time to start preparing for FOSDEM 2012, we found that many of the NamurLUG people were not going to have as much time to prepare for video coverage as they had during past years, and therefore responsibility of FOSDEM video coverage fell entirely to me. This was fairly unexpected, though not too daunting.

For those of you who don't know dvswitch very well: a typical dvswitch installation for coverage of a talk requires:

• Two cameras; one to capture the speaker, one to capture audience members asking questions.
• A twinpact 100, used to convert a VGA output into a DV stream that we can load into dvswitch
• Three laptops: one for the audience camera, one for the twinpact, and one for the speaker camera and the dvswitch installation
• A GigE network between the three laptops, with an uplink to do streaming
• A large amount of diskspace to store the recordings on. Since dvswitch records in raw DV, you require approximately 13GB per hour of recorded video.
• Some powerful systems to do transcoding from DV to a streaming format such as Ogg Theora or WebM
• A fairly extensive audio setup: one wireless headset microphone for the speaker, one or two wireless handheld microphones for people in the audience, and a number of microphones at select positions in the room to pick up ambient noise (which for obvious reasons you don't want to overdo on, but which if done well makes the audio on your recording sound much more natural). This also requires a mixing console.
• And last but not least a (network of) streaming server(s) to do the streaming to people watching from home.

This is a rather large amount of work, and therefore it is not too unexpected that for DebConf, the DebConf video team takes a week to properly set up two rooms so that they can be used for recording. However, for FOSDEM we only have less than a day on-site to set up things, and we were going to cover five rooms. So, I tried to cut corners as much as possible:

• We were going to work with one camera per room, rather than two. Not only would this reduce the number of required volunteers (of which I wasn't sure yet at that point how I'd find them), it would also reduce the amount of preparation needing to be done, and it would reduce the number of things that could go wrong.
• We were going to have the network team do our network, rather than try to do it ourselves. The result was that we were not going to have to lay out cables, cut them, crimp connectors on them, test them, etc, and would only need to worry about using the network, instead of building it.
• Since I'd read Thomas' announcement about an ''offer I cannot refuse'', I contacted him to see about that offer and get it all set up for FOSDEM. This way, I wouldn't have to worry about streaming servers etc, instead just about getting a stream on the network, into a flumotion server running somewhere; and after that, it would no longer be my problem.
• I decided to simplify the audio setup somewhat, and not use ambient mics. These are nice, but not essential, so we could do without them, and that would be (at least) two microphones per room that the team wouldn't have to rent, place, wire up, and tune.

So in the end, I would need to take care of ten firewire-capable laptops, five cameras, five mixing consoles, several microphones, a number of transcoding machines, and shitloads of cabling (power, firewire, and audio). And about a week or two before the conference, as the massiveness of what we were about to do started to sink in, I was starting to have bad dreams of what would happen, wondering what I'd gotten myself into this time.

Now, one day after FOSDEM, I have to say it all turned out okay, but there are some things where there's still room for improvement. So that I don't forget, I thought I'd make a list of things that went well, and one of things that didn't go well. And since that might be interesting for other people, I thought I'd do it here, rather than in a private file.

First, things that did not go so well:

1. Set-up for some rooms took more time than we had, and as such some rooms did not get properly streamed or recorded for their first few talks. This was due to the fact that there was some confused communication between some members of the team, which meant that we lost a day that we'd planned for preparation, and as such we couldn't test as much as we'd hoped. We need to improve on that next year. Ideally, we should run the full set-up somewhere, with all cameras and all laptops running, and making sure that everything is ready to be set up and that we have every cable we need, before dividing everything over a number of boxes (one per room) and bringing it to the conference.
2. While some of our cameras were semi-professional Panasonic cameras that had a balanced XLR audio input, others were much lower-level camcorder-style cameras that did not have such a thing. When a camera has a good audio input, it's fairly easy to set up a link from the mixing console to the camera and get the audio into the stream that way. When a camera does not have such a thing, the set-up gets much more complex. Since we had so much to worry about, we did not notice that in one room, on saturday, something went wrong with audio. To combat this, we should improve on our audio set-up for next year, and also verify the streams every once in a while (I did the latter on sunday, so everything should be fine there).
3. Inherent to an ad-hoc network, there were some network-related issues. For instance, on sunday evening, during the last session in that room, Ben informed me that the link to room H.1301 had gone down. As we were investigating, we easily found the source of the problem:
   
   The cable had been put under a door, and had been 'protected' with loads and loads of duct tape, but apparently that wasn't enough. This wasn't something we wanted to fix anymore, at that point; instead, we just let the cable be and focused on other things.
4. Storage. None of the systems used for the recordings were my own; instead, some were owned by IRILL and some were rented. If I want to do anything with my recordings, that means I have to copy the data off. The system I chose was to bring a USB3 hard disk and copy everything over; but for saturday, this took an hour and thirty minutes. On sunday, my USB3 disk was fairly full, so I had to revert to a USB2 one, which increased the time by much. In the end, we had to abort copying files, and we'll now have to be a bit creative to get them.
5. Volunteer management. I originally set up a wiki page to allow people to sign up for talks that they wanted to help out with. Unfortunately, that didn't work so well as it did for DebConf; there were several talks that did not have enough people, resulting in the volunteers from the previous talk remaining on post and finishing their talk. Also, there were some people that signed up without contacting me, or without telling me what name they used on the wiki page, which made it somewhat harder to know who to talk to. I lost count of who had signed up to help out. On Sunday, I decided to set up an IRC channel instead, through which volunteers could communicate more directly and just ask for people to take over, instead of registering for a talk first and then hoping nobody would forget. Also, since I'm terribly bad with faces and video volunteers did not have anything that made them stand out from the rest, I couldn't just walk around until I would see someone involved with video work and ask them to take over in a room. Having a separate 'video team' t-shirt could help there.

After the bad news, here's the list of things that did go well:

1. Streaming. Even though choosing flumotion rather than icecast (as we'd done for streaming in the past) involved extra work for me before the conference (I had to write some code so that I could stream from dvswitch into flumotion, which wasn't possible out of the box), the guys at Flumotion decided to send over one of their support engineers. As such, I simply did not have to worry about streaming servers crashing or failing to do what they were supposed to do. While I have no reason to think any of the Flumotion servers were having issues, even if they were I wouldn't have known about it, since Francesc took care of it all and never ever required my involvement about something on the flumotion side. This was extremely valuable.
2. Training right before the welcome talk. Explaining to people in a few words how dvswitch works, and then immediately following that up with a live demonstration, isn't a bad way for them to understand.
3. Despite all the problems we did have, once everything had been set up and all the gear was more or less manned, actually recording and streaming did go fairly well. The feedback I received so far was mostly positive, which makes me very happy.

In the end, while there certainly is still room for improvement and things did not go as smoothly as I'd hoped, they have gone much better than my worst fears. There's still some work to do, which I'll be doing with the NamurLUG people this week, but all in all, it's gone pretty good.

As to the usage of the streams, I asked Flumotion for some statistics. Since I was fairly late in asking, they could only provide me with statistics about sunday:

• Number of clients connecting:
  
• Traffic:
  

The following announce is lazily copied from Paul Wise's announce. There is only one thing I like to add: the screenshots that are submitted and collected on screenshots.debian.net are visible on the packages websites (both Debian and Ubuntu) and are also used by the software-center package, so they help people to get a first impression of the package they might want to install.

Have you ever wondered how to start getting involved in Debian/Ubuntu? Do you enjoy discovering new games and playing them? You might want to come to the games screenshot party! We hope that the party will be a fun, easy, low-commitment way to get involved.

The Debian/Ubuntu Games Team is organizing a half-day screenshots party on the weekend of 25th-26th February for creating screenshots for all the games that are available in Debian/Ubuntu.

If you are interested in attending, please add your availability to the poll linked from the announcement so that we can get some idea of attendance and when is a good time for the people who are interested.

Look forward to lots of game playing, screenshots and merry time, hope to see you all there!

/debian | permanent link | Comments: 0 | Flattr this

I hope by now you have heard of ACTA. In any case, here is a nice 6:30 minute video giving a good overview.

Please help stop ACTA. Our freedom is at risk. Whether you tell people about it, write about it, use services like Twitter to tell the world about #StopACTA, or whether you take the time to march against what corporate entities are lobbying politicians to do against their people — please help protect the Internet as we know it.

NP: God is an Astronaut: Moment of Stillness

As I announced on debian-devel, Guillem Jover uploaded a snapshot of dpkg’s multiarch branch to experimental (version 1.16.2~wipmultiarch). Beware: There will
likely be some small “interface” changes between this version and the version that will be released later in unstable (possibly in the output of dpkg --get-selections, dpkg --list, maybe other commands).

multiarch allows you to install packages from different architectures on the same machine. This can be useful if your computer can run programs from 2 architectures (eg. x86 CPU supporting i386 and amd64), or if you often need to cross-compile software and thus need the libraries of your target architecture.

Test dpkg with multiarch support

If you want to test multiarch support in dpkg, install the package from experimental (apt-get install dpkg/experimental assuming you have experimental in your sources.list).

Then you can add a supplementary architecture to your system by doing sudo dpkg --add-architecture <arch> (e.g. i386 if you are on amd64, and vice-versa). APT will automatically pick up the new architecture and start downloading the Packages file for the new architecture (it uses dpkg --print-foreign-architectures to know about them).

From there on you can install packages from the “foreign” architectures with “apt-get install foo:<arch>“. Many packages will not be installable because some of their dependencies have not yet been updated to work with in a multiarch world (libraries must be installed in a multiarch-compliant path so as to be co-installable, and then marked “Multi-Arch: same“). Other dependencies might need to be marked “Multi-Arch: foreign“. See wiki.debian.org/Multiarch/Implementation for more HOWTO-like explanations.

Now is a good time to see if you can install the foreign packages that you could need in such a setup and to help to convert the required libraries.

You can also read Cyril Brulebois’ article which quickly shows how to hunt for the problematic packages which have not been converted to multiarch (in his sample, “ucf” is not ready. Since it’s an “Architecture: all” package which can run on any architecture, it means that it’s lacking a “Multi-Arch: foreign” field).

Report bugs

If you discover any bug in dpkg’s multiarch implementation, please report it to the Bug Tracking System (against “dpkg” with the version “1.16.2~wipmultiarch”).

If you notice important libraries or packages which are not yet multiarch ready, please open wishlist bug reports requesting the conversion and point the maintainers towards the wiki page linked above. Even better, prepare patches and submit those with your bug reports.

Again, you can follow the lead of Cyril Brulebois who filed 6 bugs!

Review the multiarch implementation

If you’re a C programmer and have some good knowledge of dpkg (or are willing to learn more of it), we would certainly benefit from more eyes reviewing the multiarch branch. If you want to discuss some design issues of the multiarch implementation in dpkg (or have questions related to your review), please get in touch via debian-dpkg@lists.debian.org.

The latest version of the branch is pu/multiarch/master in Guillem’s personal repository. I have my own version of the branch (pu/multiarch/full) which is usually a snapshot of Guillem’s branch with my own submitted fixes.

$ git clone git://git.debian.org/dpkg/dpkg.git $ cd dpkg $ git remote add guillem git://git.hadrons.org/git/debian/dpkg/dpkg.git $ git remote add buxy git://git.debian.org/~hertzog/dpkg.git $ git fetch guillem && git fetch buxy

If you followed the instructions above, the relevant branches are thus guillem/pu/multiarch/master and buxy/pu/multiarch/full. Both branches are regularly rebased on top of master where Guillem merges progressively the commits from the multi-arch branch as his review progresses.

Thank you in advance for your help bringing multiarch in shape for Debian Wheezy,

No comment | Liked this article? Click here. | My blog is Flattr-enabled.

Everyone agrees that backups are generally a good thing. But it seems that there is a lot less agreement about how backups should work. Here is a list of 5 principles of backup software that seem to get ignored most of the time:

(1/5) Backups should not be Application Specific

It’s quite reasonable for people to want to extract data from a backup on a different platform. Maybe someone will want to extract data a few decades after the platform becomes obsolete. I believe that vendors of backup software have an ethical obligation to make it possible for customers to get their data out with minimal effort regardless of the circumstances.

Often when writing a backup application there will be good reasons for not using the existing formats for data storage (tar, cpio, zip, etc). But ideally any data store which involves something conceptually similar to a collection of files in one larger file will use one of those formats. There have been backward compatible extensions to tar and zip for SE Linux contexts and for OS/2 EAs – the possibility of extending archive file formats with no consequence other than warnings on extraction with an unpatched utility has been demonstrated.

For a backup which doesn’t involve source files (EG the contents of some sort of database) then it should be in a format that can be easily understood and parsed. Well designed XML is generally a reasonable option. Generally the format should involve plain text that is readable and easy to understand which is optionally compressed with a common compression utility (pkzip is a reasonable choice).

(2/5) Data Store Formats should be Published

For every data store there should be public documentation about it’s format to allow future developers to write support for it. It really isn’t difficult to release some commented header files so that people can easily determine the data structures. This includes all data stores including databases and filesystems. If I suddenly find myself with a 15yo image of a NTFS filesystem containing a proprietary database I should be able to find official header files for the version of NTFS and the database server in question so I can decode the data if it’s important enough.

When an application vendor hides the data formats it gives the risk of substantial data loss at some future time. Imposing such risk on customers to try and prevent them from migrating to a rival product is unethical.

(3/5) Backups should be forward and backward compatible

It is entirely unreasonable for a vendor to demand that all their users install the latest versions of their software. There are lots of good reasons for not upgrading which includes hardware not supporting new versions of the OS, lack of Internet access to perform the upgrade, application compatibility, and just liking the way the old version works. Even for the case of a critical security fix it should be possible to restore data without applying the fix.

For any pair of versions of software that are only separated by a few versions it should be possible to backup data from one and restore to the other. Even if the data can’t be used directly (EG a backup of AMD64 programs that is restored on an i386 system) it should still be accessible. If a new version of the software doesn’t support the ancient file formats then it should be possible for the users to get a slightly older version which talks to both the old and new versions.

Backups made on 64bit systems running the latest development version of Linux and on 10yo 32bit proprietary Unix systems are interchangeable. Admittedly Unix is really good at preserving file format compatibility, but there is no technical reason why other systems can’t do the same. Source code to cpio, tar, and gnuzip, is freely available!

Apple TimeMachine fails badly in this regard, even a slightly older version of Mac OS can’t do a restore. It is however nice that most of the TimeMachine data is a tree of files which could be just copied to another system.

(4/5) Backup Software should not be Dropped

Sony Ericsson has made me hate them even more by putting the following message on their update web site:

The Backup and Restore app will be overwritten and cannot be used to restore data. Check out Android Market for alternative apps to back up and restore your data, such as MyBackup.

So if you own a Sony Ericsson phone and it is lost, stolen, or completely destroyed and all you have is a backup made by the Sony Ericsson tool then the one thing you absolutely can’t do is to buy a new Sony Ericsson phone to restore the data.

I believe that anyone who releases backup software has an ethical obligation to support restoring to all equivalent systems. How difficult would it be to put a new free app in the Google Market that has as it’s sole purpose recovering old Sony Ericsson backups onto newer phones? It really can’t be that difficult, so even if they don’t want to waste critical ROM space by putting the feature in all new phones they can make it available to everyone who needs it. When compared to the cost of developing a new Android release for a series of phones the cost of writing such a restore program would be almost nothing.

It is simply mind-boggling that Sony Ericsson go against their own commercial interests in this regard. Surely it would make good business sense to be able to sell replacements for all the lost and broken Sony Ericsson phones, but instead customers who get burned by broken backups are given an incentive to buy a product from any other vendor.

(5/5) The greater the control over data the greater the obligation for protecting it

If you have data stored in a simple and standard manner (EG the /DCIM directory containing MP4 and JPEG files that is on the USB accessible storage in every modern phone) then IMHO it’s quite OK to leave customers to their own devices in terms of backups. Typical users can work out that if they don’t backup their pictures then they risk losing them, and they can work out how to do it.

My Sony Ericsson phones have data stored under /data (settings for Android applications) which is apparently only accessible as root. Sony Ericsson have denied me root access which prevents me running backup programs such as Titanium Backup, therefore I believe that they have a great obligation to provide a way of making a backup of this data and restoring it on a new phone or a phone that has been updated. To just provide phone upgrade instructions which tell me that my phone will be entirely wiped and that I should search the App Market for backup programs is unacceptable.

I believe that there are two ethical options available to Sony Ericsson at this time, one is to make it easy to root phones so that Titanium Backup and similar programs can be used, and the other option is to release a suitable backup program for older phones. Based on experience I don’t expect Sony Ericsson to choose either option.

Now it is also a bad thing for the Android application developers to make it difficult or impossible to backup their data. For example the Wiki for one Android game gives instructions for moving the saved game files to a new phone which starts with “root your phone”. The developers of that game should have read the Wiki, realised that rooting a phone for the mundane task of transferring saved game files is totally unreasonable, and developed a better alternative.

The best thing for developers to do is to allow the users to access their own data in the most convenient manner. Then it becomes the user’s responsibility to manage it and they can concentrate on improving their application.

Why Freedom is Important

Installing CyanogenMod on my Galaxy S was painful, but having root access so I can do anything I want is a great benefit. If phone vendors would do the right thing then I could recommend that other people use the vendor release, but it seems that vendors can be expected to act unethically. So I can’t recommend that anyone use an un-modded Android phone at any time. I also can’t recommend ever buying a Sony Ericsson product, not even when it’s really cheap.

Google have done a great thing with their Data Liberation Front [1]. Not only are they providing access to the data they store on our behalf (which is a good thing) but they have a mission statement that demands the same behavior from other companies – they make it an issue of competitive advantage! So while Sony Ericsson and other companies might not see a benefit in making people like me stop hating them, failing to be as effective in marketing as Google is a real issue. Data Liberation is something that should be discussed at board elections of IT companies.

Keep in mind the fact that ethics are not just about doing nice things, they are about establishing expectations of conduct that will be used by people who deal with you in future. Sony Ericsson has shown that I should expect that they will treat the integrity of my data with contempt and I will keep this in mind every time I decline an opportunity to purchase their products. Google has shown that they consider the protection of my data as an important issue and therefore I can be confident when using and recommending their services that I won’t get stuck with data that is locked away.

While Google has demonstrated that corporations can do the right thing, the vast majority of evidence suggests that we should never trust a corporation with anything that we might want to retrieve when it’s not immediately profitable for the corporation. Therefore avoiding commercial services for storing important data is the sensible thing to do.

• [1] http://www.dataliberation.org/

Related posts:

1. Galaxy S vs Xperia X10 and Android Network Access Galaxy S Review I’ve just been given an indefinite loan...
2. Standardising Android Don Marti wrote an amusing post about the lack of...
3. On Burning Platforms Nokia is in the news for it’s CEO announcing that...

qemu package is uninstallable in sid, and I think ipxe wants to replace it, not break it. Hmm.. I don't understand.

Further to last week’s blog post that mentioned this Saturday’s (11 Feb) London Stop ACTA march, there’s a map of anti-ACTA marches on Google’s website (thanks to Martin Houston for the link).

There’s also been a new Anti-Counterfeiting Trade Agreement factsheet from European Digital RIghts (EDRI), as apparently there are a lot of misconceptions about ACTA. I don’t feel that has been helped by some spectacular misdirection from the European Commission in its latest “10 Myths” paper (linked from the EDRI factsheet) which is almost as interesting for what it doesn’t mention (like sneaking ACTA through the parliament fisheries committee), what it misunderstands (like the near-uselessness of a non-commercial exemption to Free and Open Source Software or Creative Commons users), and the way it fails to rebut the final point that ACTA was done this way to avoid the oversight of the World Trade Organisation! I mean, if they can’t even get it past the usually very pro-enforcement WTO, surely that should tell you something?

If you can, would you please go along and join your nearest march? Recent marchers seem to have been wearing stylised Guy Fawkes masks, but how would that be viewed in London?

I've just — finally — finished reading Simon Reynold's dense "Rip it Up and Start Again": a history of post-punk music from 1978-1984.

I often think of music journalism in the terms that Zappa famously phrased as (roughly) "people who can't write, writing for people who can't read". This book sometimes suffers from that syndrome, of attempting to describe audio in terms of other senses, which sometimes wanders deep into synesthesia territory, but for the most part the book is coherent and enjoyable.

It's also very "dense": you leave each page with another half-dozen songs or bands of music genres to go and investigate. I'm surprised that there isn't a commercially-released anthology counterpart to the book. There are, however, several playlists, many of which are locked away on Spotify, but via this review I found these two which are web-based (powered by opentape, an interesting piece of AGPL software).

The book has helped me to discover a lot of music that I hadn't heard of before and now love, which I will probably write about in different contexts over the next few months.

FOSDEM is over. Was fun. Nobody booed too much during my keynote. I am not putting up my slides, since they're useless without the actual talk. However, I've put up the text I prepared for the talk, which is approximately what I actually talked, in case anyone wants to read rather than watch the video whenever it gets released.

I Shall Wear Midnight is the 38th Discworld novel and the fourth following the story of Tiffany Aching, young witch of the Chalk.

Reading modern Pratchett, I'm sadly, acutely (and morbidly) aware that, as he succumbs to Alzheimer's disease, these are the likely to be some of the last Discworld novels written. With the last few, this observation has been entirely external to the novel. With Midnight, however, I got the impression that the awful disease has finally started to affect the prose itself.

I could just be jumping at shadows, but I got the impression that he repeated himself a tad more often than usual. With such a long series of books, an author has the difficult job of balancing new and established readers in the same text: as such, as an established reader, you get used to having some back-story with which you must be patient. This necessary background could have contributed to the impression of repetition, but I'm fairly sure doesn't explain all of it.

An enjoyable story, and possibly the one to bring the Aching arc to a conclusion. She stands on her own two feet, proving herself a a capable Witch. There weren't any earth-shattering new concepts or new characters or unexplored areas of the Disc for fans to map out with their minds; non-the-less it was a satisfying read.

What with two kids, a new house, and LiberWriter getting some good traction, I've been looking around for things to give to a good home so as to have less stuff to deal with.

So, on the auction block goes BikeChatter.com : https://flippa.com/2696023-professional-cyclists-on-twitter-plus-2-years-of-history

BikeChatter.com is the place to go on the web to follow professional cyclists on twitter.  With 500+ racers, and nearly half a million status updates from racers like Lance Armstrong, Alberto Contador, Mark Cavendish, and many, many more, this site is the best place to find out what's going on in the world of professional cycling, directly from the participants.

Since I like following the site myself, I really want to see it go to people who will take it and make it even better.

Yet another FOSDEM is behind us and I'd like to thank all people organizing it. It was a great event as usual.

This year there were some changes - the conference grew and there was an extra building. This is great, but on the other side, there were more tracks to follow and occasionally I wanted to be in four places at once, what is of course not manageable.

Combined with quite freezing weather (well it was still much warmer than it is now in Prague), moving from one side of campus to another was not that comfortable as in last years, but there is not much man can do with that.

And the biggest change for me - I did not manage beer event this year. We enjoyed great team dinner on Friday evening and while it ended, I was too lazy to move to crowded beer event and rather enjoyed bed in my hotel.

Filed under: Debian English Phpmyadmin Suse | 0 comments | Flattr this!

dpkg was uploaded to experimental a few times lately, let’s sum it up quickly:

1. It started with my NMU on 2012-01-31, since I hadn’t received any reasons not to upload (except “NACK”, which doesn’t count as such in my book).
2. It got reverted a few minutes after that.
3. Since this was getting nowhere, the Debian Technical Committee finally got involved, and voted in a few days only (wow, thanks!) for an “it’s time to experiment” decision.
4. A new dpkg upload to experimental finally happened. One shall note the prominent “This is a WIP release, command line interfaces will change.” notice.

I’ve updated my previous dpkg with multiarch page accordingly, only keeping the interesting bits.

Playing around with grep-aptavail -F Multi-Arch $i -sPackage (with $i being same or foreign), I found some bugs:

• #658792: libstdc++6-4.6-dbg: libstdc++.so.6.0.16-gdb.py is not an ELF file
• #658793: libisl-dbg: libisl.so.8.0.0-gdb.py is not an ELF file

Also, some wishlist bugs (keeping the prominent notice in mind):

• #658794: apt-file: Please implement multiarch support
• #658795: reportbug: Please report foreign architectures

Finally, one dpkg improvement and one possible dpkg bug:

• #658812: dpkg: Please add a hint in arch_remove about installed foreign packages
• #658814: dpkg: Inconsistent installation state with buggy multiarchified packages?

A very useful little book from various developers and others: things they wish they'd known before starting out. A very sensible contribution from Debian's own Evan Prodromou amongst other names I knew and recognised and some interesting folk I'd not heard of.

http://open-advice.org/Open-Advice.pdf is the downloadable PDF although source and so on is also available. There are likely to be forums for comments and improvements.

[UPDATE - I've just had a go at the training exercises listed at openhatch.org which are linked from the site. Not a bad revision on patching/SVN/git and generally using the tools needed initially to contribute to FLOSS. A good job well done]

ZDNet has an insightful article by Robin Harris predicting the demise of RAID-6 due to the probability of read errors [1]. Basically as drives get larger the probability of hitting a read error during reconstruction increases and therefore you need to have more redundancy to deal with this. He suggests that as of 2009 drives were too big for a reasonable person to rely on correct reads from all remaining drives after one drive failed (in the case of RAID-5) and that in 2019 there will be a similar issue with RAID-6.

Of course most systems in the field aren’t using even RAID-6. All the most economical hosting options involve just RAID-1 and RAID-5 is still fairly popular with small servers. With RAID-1 and RAID-5 you have a serious problem when (not if) a disk returns random or outdated data and says that it is correct, you have no way of knowing which of the disks in the set has good data and which has bad data. For RAID-5 it will be theoretically possible to reconstruct the data in some situations by determining which disk should have it’s data discarded to give a result that passes higher level checks (EG fsck or application data consistency), but this is probably only viable in extreme cases (EG one disk returns only corrupt data for all reads).

For the common case of a RAID-1 array if one disk returns a few bad sectors then probably most people will just hope that it doesn’t hit something important. The case of Linux software RAID-1 is of interest to me because that is used by many of my servers.

Robin has also written about some NetApp research into the incidence of read errors which indicates that 8.5% of “consumer” disks had such errors during the 32 month study period [2]. This is a concern as I run enough RAID-1 systems with “consumer” disks that it is very improbable that I’m not getting such errors. So the question is, how can I discover such errors and fix them?

In Debian the mdadm package does a monthly scan of all software RAID devices to try and find such inconsistencies, but it doesn’t send an email to alert the sysadmin! I have filed Debian bug #658701 with a patch to make mdadm send email about this. But this really isn’t going to help a lot as the email will be sent AFTER the kernel has synchronised the data with a 50% chance of overwriting the last copy of good data with the bad data! Also the kernel code doesn’t seem to tell userspace which disk had the wrong data in a 3-disk mirror (and presumably a RAID-6 works in the same way) so even if the data can be corrected I won’t know which disk is failing.

Another problem with RAID checking is the fact that it will inherently take a long time and in practice can take a lot longer than necessary. For example I run some systems with LVM on RAID-1 on which only a fraction of the VG capacity is used, in one case the kernel will check 2.7TB of RAID even when there’s only 470G in use!

The BTRFS Filesystem

The btrfs Wiki is currently at btrfs.ipv5.de as the kernel.org wikis are apparently still read-only since the compromise [3]. BTRFS is noteworthy for doing checksums on data and metadata and for having internal support for RAID. So if two disks in a BTRFS RAID-1 disagree then the one with valid checksums will be taken as correct!

I’ve just done a quick test of this. I created a filesystem with the command “mkfs.btrfs -m raid1 -d raid1 /dev/vg0/raid?” and copied /dev/urandom to it until it was full. I then used dd to copy /dev/urandom to some parts of /dev/vg0/raidb while reading files from the mounted filesystem – that worked correctly although I was disappointed that it didn’t report any errors, I had hoped that it would read half the data from each device and fix some errors on the fly. Then I ran the command “btrfs scrub start .” and it gave lots of verbose errors in the kernel message log telling me which device had errors and where the errors are. I was a little disappointed that the command “btrfs scrub status .” just gave me a count of the corrected errors and didn’t mention which device had the errors.

It seems to me that BTRFS is going to be a much better option than Linux software RAID once it is stable enough to use in production. I am considering upgrading one of my less important servers to Debian/Unstable to test out BTRFS in this configuration.

BTRFS is rumored to have performance problems, I will test this but don’t have time to do so right now. Anyway I’m not always particularly concerned about performance, I have some systems where reliability is important enough to justify a performance loss.

BTRFS and Xen

The system with the 2.7TB RAID-1 is a Xen server and LVM volumes on that RAID are used for the block devices of the Xen DomUs. It seems obvious that I could create a single BTRFS filesystem for such a machine that uses both disks in a RAID-1 configuration and then use files on the BTRFS filesystem for Xen block devices. But that would give a lot of overhead of having a filesystem within a filesystem. So I am considering using two LVM volume groups, one for each disk. Then for each DomU which does anything disk intensive I can export two LVs, one from each physical disk and then run BTRFS inside the DomU. The down-side of this is that each DomU will need to scrub the devices and monitor the kernel log for checksum errors. Among other things I will have to back-port the BTRFS tools to CentOS 4.

This will be more difficult to manage than just having an LVM VG running on a RAID-1 array and giving each DomU a couple of LVs for storage.

BTRFS and DRBD

The combination of BTRFS RAID-1 and DRBD is going to be a difficult one. The obvious way of doing it would be to run DRBD over loopback devices that use large files on a BTRFS filesystem. That gives the overhead of a filesystem in a filesystem as well as the DRBD overhead.

It would be nice if BTRFS supported more than two copies of mirrored data. Then instead of DRBD over RAID-1 I could have two servers that each have two devices exported via NBD and BTRFS could store the data on all four devices. With that configuration I could lose an entire server and get a read error without losing any data!

Comparing Risks

I don’t want to use BTRFS in production now because of the risk of bugs. While it’s unlikely to have really serious bugs it’s theoretically possible that as bug could deny access to data until kernel code is fixed and it’s also possible (although less likely) that a bug could result in data being overwritten such that it can never be recovered. But for the current configuration (Ext4 on Linux software RAID-1) it’s almost certain that I will lose small amounts of data and it’s most probable that I have silently lost data on many occasions without realising.

• [1] http://www.zdnet.com/blog/storage/why-raid-6-stops-working-in-2019/805
• [2] http://storagemojo.com/2008/02/18/latent-sector-errors-in-disk-drives/
• [3] http://btrfs.ipv5.de/

Related posts:

1. Some RAID Issues I just read an interesting paper titled An Analysis of...
2. ECC RAM is more useful than RAID A common myth in the computer industry seems to be...
3. Software vs Hardware RAID Should you use software or hardware RAID? Many people claim...

For the past few years the anti-spam system I run has been based on a simplified version of something I previously ran commercially.

Although the code is similar in intent there were both explicit feature removals, and simplifications made.

Last month I re-implimented domain-blacklisting - because a single company keeps ignoring requests to remove me.

So LinkedIn.com if you're reading this:

• I've never had an account on your servers.
• I find your junk mail annoying.
• I suspect I'll join your site/service when hell freezes over.

I've also implemented TLD-blacklisting which has been useful.

TLD-blacklisting in my world is not about blocking mail from foo@bar.ph (whether in the envelope sender, or the from: header), instead it is about matching the reverse DNS of the connecting client.

If I recieve a connection from 1.2.3.4 and the reverse DNS of that IP address matches, say, /\.sa$/i then I default to denying it.

My real list is longer, and handled via files:

steve@steve:~$ ls /srv/_global_/blacklisted/tld/ -C ar br cn eg hr in kr lv mn np ph ro sg tg ua ve zw aw cc cy gm hu is kz ma my nu pk rs sk th ug vn be ch cz gr id it lk md mz nz pl ru su tr uy ws bg cl ec hk il ke lt mk no om pt sa sy tw uz za

On average I'm rejecting about 2500 messagse a day at SMTP-time, and 30 messages, or so, hit my SPAM folder after being filtered with CRM114 after being accepted for delivery. (They are largely from @hotmail and @yahoo, along with random compromised machines. The amount of times I see a single mail from a host with RDNS mysql.example.org is staggering.).

(Still looking forward to the development of Haraka, a node.js version of qpsmtpd.)

ObQuote: "Mr. Mystery Guest? Are you still there? " - Die Hard