Planet Debian

Third post about ganeti.

Ganet-debootstrap-instance contains a nice set of scripts to create a debian (or derivatives) image using debootstrap. Images can be configured and customized by writing simple hooks script to modify various aspects of the default installation. However writing these script is not really fun and pushing it too far can lead to long messy scripts, loosing the overall benefit of automatic configuration.

Puppet is my configuration management tool of choice, but installing puppet on a new machine requires few magic incantations that the user should perform manually, or in a semi automatic mode (autosign=true) to make it work. My goal is to install puppet automatically on the newly created instance so it will run and configure the new instance at the first boot. From that moment on I'll forget about ganeti and configure all remaining services of my new VM using puppet.

In order to do so, we need to install puppet (and apt-get update/upgrade...), create the ssl certificates for the client and enabling the puppet daemon on the client. We add another hook in /etc/ganeti/instance-debootstrap/hooks/ :

if [ -z "$TARGET" -o ! -d "$TARGET" ]; then
  echo "Missing target directory"
  exit 1
fi

LANG=C
chroot "$TARGET" apt-get -y --force-yes update
chroot "$TARGET" apt-get -y --force-yes upgrade

# install puppet on the client
chroot "$TARGET" apt-get -y --force-yes install puppet

DOMAIN=localnet.org
instance=$INSTANCE_NAME.$DOMAIN

echo "Installing puppet certificates for $instance"
puppetca clean $instance
puppetca -g $instance

mkdir -p $TARGET/etc/puppet
mkdir -p $TARGET/var/lib/puppet/ssl/private_keys/
mkdir -p $TARGET/var/lib/puppet/ssl/certs/

cp /var/lib/puppet/ssl/private_keys/$instance.pem $TARGET/var/lib/puppet/ssl/private_keys/
rm -f $TARGET/var/lib/puppet/ssl/public_keys/$instance.pem

cp /var/lib/puppet/ssl/certs/$instance.pem $TARGET/var/lib/puppet/ssl/certs/
cp /var/lib/puppet/ssl/certs/ca.pem $TARGET/var/lib/puppet/ssl/certs/

chown root. $TARGET/var/lib/puppet/ssl/private_keys/$instance.pem
chmod 0400 $TARGET/var/lib/puppet/ssl/private_keys/$instance.pem

chown root. $TARGET/var/lib/puppet/ssl/certs/$instance.pem
chmod 0640 $TARGET/var/lib/puppet/ssl/certs/$instance.pem

chown root. $TARGET/var/lib/puppet/ssl/certs/ca.pem
chmod 0641 $TARGET/var/lib/puppet/ssl/certs/ca.pem

#echo "server=puppet" >> /etc/puppet/puppet.conf

echo "START=yes" > $TARGET/etc/default/puppet
echo "DAEMON_OPTS=\"\"" >> $TARGET/etc/default/puppet

This script uses puppetca to create on the puppet (and ganeti) server the client key, sign it, and then copy it to the target machine. Notice that we create the certificate for a fqnd name $INSTANCE_NAME.$DOMAIN or otherwise puppet will complain loudly. This is not strictly needed, but if you want to do otherwise, you'll need to fiddle with the puppet configuration a bit more. The procedure to create a puppet certificate server-side is well documented on the puppet website, so if you are curious about the details duck-duck-it .

John Scalzi recently wrote a piece on straight white male privilege. If you haven't read it already, go and do so. No rush. I'll wait.

So. Some facts:

• Women are underrepresented in free software development
  
• Those women who are involved in free software development are overwhelmingly more likely to have been subject to sexual harassment, belittling commentary or just plain ignored because of their gender
  
• When asked, women tend to believe that these two facts are fairly strongly related

(If you disagree with any of these then that's absolutely your right. You're wrong, but that's ok. But please do me a favour and stop reading here. Otherwise you'll just get angry and then you'll write something ill-tempered and still wrong in the comments and then I'll have to delete it and why not just save everybody the time and effort and go and eat ice cream or something instead)

I know I've said this before, but inappropriate and marginalising behaviour is rife in our community, and at all levels of our community. There's the time an open source evangelist just flat out told a woman that her experiences didn't match his so she must be an outlier. There's the time a leading kernel developer said that most rape statistics were basically made up. There's the time that I said the most useful thing Debian could do with its money would be to buy prostitutes for its developers, simultaneously sexualising the discussion, implying that Debian developers were all straight men and casting sex workers as property. These aren't the exceptions. It's endemic. Almost all of us have been part of the problem, and in doing so we've contributed to an environment that has at best driven away capable contributors. You probably don't want to know what it's done at worst.

But what people have done in the past isn't important. What's important is how we behave in the future. If you're not angry about social injustice like this then you're doing it wrong. If you're reading this then there's a pretty high probability that you're a white male. So, it's great that you're angry. You should be! As a straight white male born into a fairly well-off family, a native English speaker in an English speaking country, I have plenty of time to be angry before going back to my nice apartment and living my almost entirely discrimination-free life. So if it makes me angry, I have absolutely no way of comprehending how angry it must make the people who actually have to live with this shit on a daily basis.


(Were tampon mouse able to form and express coherent thoughts, tampon mouse would not put up with this shit)
The point isn't to be smugly self aware of our own shortcomings and the shortcomings of others. The point is to actually do something about it. If you're not already devoting some amount of your resources to improving fairness in the world, then why not? It doesn't have to be about women in technology - if you're already donating to charity or helping out at schools or engaging in local politics or any of the countless other ways an individual can help make the world a better place, large or small, then keep on doing that. But do consider that many of us have done things in the past that contributed to the alienation of an astounding number of potential community members, and if you can then please do do something to make up for it. It might be donating to groups like The Ada Initiative. It might be mentoring students for projects like the GNOME Outreach Program for Women, or working to create similar programs. Even just making our communities less toxic by pointing out unacceptable behaviour when you see it makes a huge difference.

But most importantly, be aware that it was people like me who were responsible for this problem in the first place and people like me who need to take responsibility for solving it. We can't demand the victims do that for us.

comments

I have just uploaded exim 4.80 rc4 to experimental. As the GnuTLS code has been overhauled testing would be appreciated.

PHPUnit, according to its PTS, has been in Debian since 2009. But it was orphaned, and nobody took care of it for a while. That is until few weeks ago, Luis Uribe started to work on it. Since he isn’t a DD, and that I take care of, I believe, about half of the all the PHP PEAR packages in Debian, I started working with him on re-doing the work of packaging PHPUnit for Debian. Previously, the old version was quite wrong, with missing dependencies, and not really working, what a shame…

PHPUnit 3.6 has been in Debian SID for 3 days now. And it’s working well. I’m now adding runs of unit testings from upstream packages at build time (of course, only if DEB_BUILD_OPTIONS doesn’t contains “nocheck”, as per the policy). All together, it’s been quite some fun to hack this, and I’m quite happy of the results, even though there’s still a lot of work remaining.

PHP itself is running unit tests at build time. And not just a few: more than 11000 of them. The only “small” issue, is that they are totally outdated. Over the time, the vardump() function has evolved, and doesn’t print things the same way. One may say that it prints things in a nicer way, but as a result, many of the tests that were suppose to work, actually fails because of these differences in vardump() over time.

So I started working on fixing some of the tests. It’s most of the time very easy to fix, but it takes a long time to fix all these small unit tests. So far, I’ve been able to fix most of what’s in the tests/, Zend/tests (more than 161 tests fixed), and the beginning of ext/*/tests: for the moment: bcmath, bz2, calendar, curl, date, dba, dom, ereg, and a part of exif, which for the moment represent 214 fixed tests, and I’ll try to do more fixes when I have time. I hope to send the patches upstream soon. The final goal is of course to have the build of PHP to fail if unit tests are failing as well. For the moment, unit tests do run at build time, but the build don’t care of the results, which I think is stupid (it’s wasting CPU cycles for no reasons, IMO).

 

If you maintain some PEAR packages, I would welcome you to first join the PKG-PEAR team on Alioth, and team maintain your packages, switch over to pkg-php-tools and dh 8 auto-sequencer, and to follow the PKG-PHP-PEAR packaging guide lines so that we have consistency in the archive. And of course, run unit testings, by doing a build-depends on phpunit (>= 3.6). Note that unit testings in PEAR packages are tracked on the Debian wiki. This post is also a call for help, because I feel quite alone doing this work of packaging PEAR packages, which many PHP applications depend on (think about roundcube, horde, extplorer, and many more). Other teams are doing very well, like the Perl team, and there’s no reason why Debian wouldn’t maintain PHP libraries as well as the ones for Perl.

A new stable release 3.2.0 of Armadillo is now available. As usual, we have wrapped this into a new RcppArmadillo package, now at 0.3.0.2; and this version is now available via CRAN.

The short NEWS entry follows below. For those interested in following RcppArmadillo eevn more closely, we generally track Conrad's Armadillo release candidates as well in SVN on R-Forge but do no longer submit these to CRAN (as the CRAN maintainers have enough incoming packages).

0.3.2.0 2012-05-21 o Upgraded to Armadillo release 3.2.0 "Creamfields" * faster eigen decomposition via "divide and conquer" algorithm * faster transpose of vectors and compound expressions * faster handling of diagonal views * faster handling of tiny fixed size vectors (≤ 4 elements) * added unique(), for finding unique elements of a matrix Courtesy of CRANberries, there is also a diffstat report for 0.3.2.0 relative to 0.3.0.3 As always, more detailed information is on the RcppArmadillo page. Questions, comments etc should go to the rcpp-devel mailing list off the R-Forge page.

Second post about ganeti. This time I'll talk about adding a swap partition to an image added with ganeti-deboostrap-instance. Browsing the web, it seems that an old version of the ganeti debostrap script allowed for the creation of a swap partition from the command line. The actual version in sid does not, so, if you want to add a swap partition, you need to write a small hook in /etc/ganeti/instance-debootstrap/hooks/.

Part of the code below is taken from the instance-debootstrap script.

if [ $DISK_COUNT -lt 2 -o -z "$DISK_1_PATH" ]; then
    log_error "Skip swap creation"
    exit 0
fi

swapdev=$DISK_1_PATH

# Make sure we're not working on the root directory
if [ -z "$TARGET" -o "$TARGET" = "/" ]; then
    echo "Invalid target directory '$TARGET', aborting." 1>&2
    exit 1
fi

if [ "$(mountpoint -d /)" = "$(mountpoint -d "$TARGET")" ]; then
    echo "The target directory seems to be the root dir, aborting."  1>&2
    exit 1
fi

if [ -f /sbin/blkid -a -x /sbin/blkid ]; then
  VOL_ID="/sbin/blkid -o value -s UUID"
  VOL_TYPE="/sbin/blkid -o value -s TYPE"
else
  for dir in /lib/udev /sbin; do
    if [ -f $dir/vol_id -a -x $dir/vol_id ]; then
      VOL_ID="$dir/vol_id -u"
      VOL_TYPE="$dir/vol_id -t"
    fi
  done
fi

if [ -z "$VOL_ID" ]; then
  log_error "vol_id or blkid not found, please install udev or util-linux"
  exit 1
fi

if [ -n "$swapdev" ]; then
  mkswap $swapdev
  swap_uuid=$($VOL_ID $swapdev || true )
fi

[ -n "$swapdev" -a -n "$swap_uuid" ] && cat >> $TARGET/etc/fstab <<EOF
UUID=$swap_uuid   swap            swap    defaults        0       0
EOF

This script does two things: first it checks if the user passed a second disk argument to the gnt-instance add call. I decided arbitrarily that the second disk is going to be used a swap disk. Second it figures out the vol-id of this disk, create the swap partition and write an entry in the fstab. All in all it's a straightforward procedure, but I love when I can cut and paste easy scripts :)

The call to create the instance is as follows, using a disk of 5G for the system and a disk of 1G for the swap.

gnt-instance add -t plain --disk 0:size=5G --disk 1:size=1G -B memory=1024 -o debootstrap+unstable --no-ip-check --no-name-check node1

While investigating the case of some packages responsible for uninstallability on the s390x architecture, I stumbled upon one FTBFS on a single architecture, reported as #673590 (<insert some nice words about software shipping with a decent testsuite>).

Given the int versus size_t question was asked, I grabbed this old (it’s UNIX!) reference: 64-bit and Data Size Neutrality.

Among other things, it describes the “everything is 32-bit” to “64-bit is becoming the new standard” slow conversion process, where keeping compatibility with existing applications was a high priority. It has a nice description of so-called C data models, making it possible to refer to them quickly: LP32 and ILP32 in the 32-bit world; ILP64, LLP64, and LP64 in the 64-bit world. I won’t go into detail here, this page has a nice table and lots of explanations about pros and cons for those.

(On a personal note, I discovered those on xorg-devel@ when I saw patches floating around, which were about optimizing data sizes for this or that data model, by picking the right types.)

While standards may be boring, this page makes it really easy to understand which data types to use, to ensure the best 32/64-bit compatibility possible. It’s even full of dos and don’ts. See the second half (Porting Issues) for details.

Some more progress since my last entry....

Slovenian, Punjabi fully completed.

Romanian completed for first two sublevels, hence rescued.

Got signs of life from Galician, Croatian, Georgian, Malayalam, Nepali translators. But nothing happened yet

Got offers to help for Welsh, Lithuanian. But nothing happened yet.

Progress for Amharic in level 1

At this very moment, it means that I would deactivate 14 languages:

• Amharic (2u, 14f17u)
• Welsh (39f82u, 32f47u)
• Galician (7f10u, 6f9u)
• Croatian (16f19u, 6f9u)
• Georgian (17f47u, 10f21u)
• Kurdish (12f25u, 6f22u)
• Lithuanian (15f22u, 12f20u)
• Latvian (22f9u, 7f9u)
• Malayalam (5u in sublevel 2 only!)
• Nepali (16f54u, 10f21u)
• Norwegian Nynorsk (13f34u, 13f23u)
• Albanian (19f50u, 13f23u)
• Tamil (2f1u, 1f)
• Tagalog (18f54u, 24f39u)

I was on vacations for a few days last week, therefor my list of RC bugs is a bit shorter this time. thanks again to everyone who sent patches to the BTS that I could just use.

• #625230 – libfile-path-perl: "libfile-path-perl: uninstallable, obsoleted by perl"
  removal bug filed (#672905; pkg-perl)
• #642745 – src:libnetserver-generic-perl: "libnetserver-generic-perl: FTBFS: E: Caught signal 'Terminated': terminating immediately"
  removal bug filed (#672903; pkg-perl)
• #658577 – libevocosm-dev: "-dev library is broken"
  add patch from Vincent Legout, upload to DELAYED/2
• #661500 – src:libevocosm: "FTBFS: dpkg-source: error: aborting due to unexpected upstream changes"
  fix config.{guess,sub} handling, upload to DELAYED/2
• #667244 – libevocosm: "libevocosm: ftbfs with GCC-4.7"
  add patch from Matthias Klose, upload to DELAYED/2
• #672000 – src:structure-synth: "structure-synth: FTBFS: StructureSynth/JavaScriptSupport/../../SyntopiaCore/GLEngine/Sphere.h:25:4: error: 'GLUquadric' does not name a type"
  sponsor NMU from Sebastian Ramacher, upload to DELAYED/2
• #672005 – src:l2tp-ipsec-vpn: "l2tp-ipsec-vpn: FTBFS: src/main.cpp:210:15: error: '::getuid' has not been declared"
  add patch from Paul Tagliamonte, upload to DELAYED/3
• #672010 – src:cryptkeeper: "cryptkeeper: FTBFS: lsof.cpp:40:2: error: 'pipe' was not declared in this scope"
  add patch from Paul Tagliamonte, upload to DELAYED/3
• #672014 – src:megaglest: "megaglest: FTBFS: util.cpp:360:30: error: 'close' was not declared in this scope"
  sponsor NMU from Sebastian Ramacher, upload to DELAYED/2
• #672071 – src:pxe-kexec: "pxe-kexec: FTBFS: networkhelper.cc:211:21: error: 'close' was not declared in this scope"
  add patch from Paul Tagliamonte, upload to DELAYED/3, then closed by maintainer upload

A couple of weeks ago was the first anniversary of orphaning Qt3 in Debian, see bug 625502.

In this year, Qt3 has got a few QA uploads with the most relevant change being support to multiarch. And, more importantly, nobody seemed to care enough to step into maintaining it.

In the last days, I have taken a look into how much needed to be done to remove Qt3 and there were slightly more than 50 packages depending directly or indirectly from Qt3. A removal from Wheezy seemed doable given that removing packages is never a problem during the Debian freeze ;-)
All the packages affected have a bug opened since more than one year and half ago and I have pinged all the bugs with some maintainers responding quick (thanks!). I also filed some removals for packages that were clearly unmaintained and didn’t seem worth keeping, with ftp-masters responding quick too (Thanks!). And finally, a couple of QA upload for orphaned software that were still useful without Qt3.

There is a wiki page tracking the status of the removal if you are curious:
http://wiki.debian.org/qt3-x11-freeRemoval

If in the future, you are reading this and you need Qt3 in Wheezy, you can fetch it from Debian snapshots.

With the sound of the freeze approaching, we have been receiving a few more transition requests lately. Thankfully, a lot of them can apparently be processed at the same time, that's why we have currently the following, crazy tracker summary:

Of course there are some transitions that started because maintainers “forgot” how transitions are supposed to work, but we’re trying to get things done anyway.

So that grad student has just submitted his thesis and you are invited to review it. With these quick tips, you can test the limits of the student’s willpower, and make his degree a little more deserved by putting him up for some constructive mind games.

The oft-promised update of the cnlmisc package for R is now posted. New in this release is a convenience method, sepplot, that produces separation plots using the separationplot package; this method works directly on model fit objects as a post-estimation call, and works with both binary and ordinal models at present. In addition, epcp now works with clm2 objects from the ordinal package.

Most of this was motivated by continued work on the economic voting paper, which has also been updated. cnlmisc still has a long way to go before I submit it to CRAN, but at least it’s progress, right?

The Asus UX21e (and maybe the UX31e?) has the irritating misfeature that it reloads the CPU thermal tables when you unplug the power. One consequence of this is that it'll automatically throttle itself much more aggressively on battery (reducing performance) but the more serious one is that the new critical power off temperature may then be lower than the temperature the CPU is currently operating at, resulting in the machine turning itself off. As far as I can tell from debugging, this is completely OS-independent - it still happens even if I stub out all the ACPI code for power supply events and there are reports of the same thing occurring on Windows. The good news is that it seems to be fixed in newer firmware versions. The even better news is that you can flash it without Windows. Just download the BIOS image from the Asus website, copy it onto a FAT formatted USB stick, insert that, go into the firmware (hit F2 on the splash screen) and start the flash program from there.

comments

While wtf(1) always has been a bit central to MirBSD, and the acronym database has been accessible by CVSweb, what we never had was a DAU compatible (and shellsnippets compatible) lookup. This has now changed: the above link to the acronyms file is a persistent link to its latest version (well, latest when the website was last recompiled), tooltips may very well follow soon, and we’ve got an online WTF lookup service.
Contributions to the acronym database are welcome, of course; just eMail them to tg＠mirbsd．org.

Not to stop there, our online HTML manpage search is also new, shiny, and should replace the “!mbsdman” DuckDuckGo hash-bang shortly. (Both of these services offer a DDG search as fallback. Note that DDG is an external service included herein by linking, under their request to spread it, and not affiliated with The MirOS Project. They do, however, donate some advertising money to Debian.)
For all those who didn’t know: only manpages for software in the MirOS BSD base system and for the MirPorts Framework package tools are listed, not for third-party applications installable using ports or, recently, pkgsrc®. Still, if you want to have a peek at a modern classic BSD’s documentation, you’re welcome. (Not to mention content like re_format(7) and style(9) and that some of our documentation is much more legible than others.)

And because writing all that perl(1) made me ill, not to mention I don’t even know that language, I’ve hacked a bit more in the mirmake(1) and mksh(1) parts of the MirWebsite, finally implementing pointing out where in the navigation sidebar the visitor currently is.

We also have exciting mksh porting news involving RT trying a larger number of ancient platforms than I dare count, me fixing bugs in Linux klibc and diving into other things, learning more about why I consider me lucky for hacking a BSD operating system… sorry, I want to keep this short as it’s mostly an announcement.

The MirWebsite source code is, of course, also available. Improvements welcome. Except for these three CGIs, our website is fully statically precompiled, and that’s a good thing. Please help in making the CGIs secure.

The Maker Faire is one of those awesome Bay Area things that always fills me with excitement and gets my imagination going.

Zoe and I went again this year to check it out, as best we could within the time constraints we had to work within (opening time and her nap time, minus travel time). She definitely enjoyed herself.

We took the Caltrain, because historically driving and parking has been a bit of a nightmare. The optimal train to get to get there before it opened (at 10am) was the 9:19 train from Mountain View, which was scheduled to get in at Hayward Park a little before 10am. It just so happened that there was a Giants game on in San Francisco today as well, and the train was absolutely packed. We only got a seat because one kind gentleman was getting off and explicitly gave his seat to us. One lesson learned: don't try and take the BOB stroller on the train. Even when collapsed, it's way too bulky. For future Caltrain outings, I'll take our City Mini stroller instead, as it folds much flatter.

I also took our macpac Possum child carrier backpack, and Zoe was pretty happy to just sit in it for the bulk of the time. I think it had novelty value for her, as we haven't used it for a while. I probably could have gotten away without taking a stroller at all. I was very glad I took the backpack, as it gave her a much better vantage point for everything that was going on than she would have gotten from sitting in the stroller.

There was supposed to be a free shuttle from the Hayward Park station to the Maker Faire, but there was a huge crowd waiting for it, so I decided to just walk. It didn't take too long. For the return trip, I think I exited from the wrong side of the fairgrounds, and couldn't figure out the shuttles, so I just walked to Hillsdale station. At least the return train wasn't crowded. Overall, using Caltrain to get in and out was successful. Zoe was very well behaved for the ~30 minute train ride each way.

The Faire was quite a bit bigger this year, and has spilled out into the parking lot on one side. I'd heard stories that O'Reilly had quadrupled booth prices as well.

Trying to abide by the program was too difficult, so we mostly just wandered through the main Expo hall and looked at various booths. I just did a full read through the website of all the exhibitors to see what I missed out on.

Here's some of the stuff I saw in person, or discovered via the website:

• there was a really excellent looking Dalek running around (way better than the photo on the page linked to here). I also learned that there's a whole Dalek-making website. Awesome.
• RAFT (Resource Area For Teaching) had lots of really simple, low cost projects for demonstrating various concepts in physics and science.
• Linux for makers was represented
• This Arduino-controlled automatic fish feeder looked cool. I didn't get to see it in person, I discovered it while I was trawling through the list of exhibitors.
• Shop-in-a-box. I'd have liked to have checked this out.
• Build a bug habitat. I'd have liked to check this out out as well.
• Solar bike trailer. My Dad would have liked this, as he has an electric bike. I imagine this wouldn't be all that hard to make. The trailer looked pretty long from the photo though, but hey, no pedaling.
• The water causeway. Now this looks interesting. I would have loved to have seen this one in person. I love clean tech. There's a whole bunch of videos linked off the page for this.
• Wave energy capture model. Another clean tech thing I'd have liked to have checked out.
• Roominate looks really cool. Something for Zoe when she's a bit bigger.
• There was a table extolling the virtues of growing your own algae for consumption (as spirulina) and bio-fuel. I'm interested in finding out more about the latter.
• I saw some HEXBUG-related stuff near the Geekdad table. This looked like a dressed up version of the "take a toothbrush head and glue an electric toothbrush motor on the back" type project. I'm curious to see how expensive the kits are, as they looked like a lot of fun.
• Ratduino sounds intriguing, but I can't find out much about it.
• Urban scale wind turbines. One that I needed to have seen in person. Unfortunately I missed it.
• GlueMotor looks cool.
• Low-cost push-button clicker. I'd have liked to have found out more about this. If it's what I imagine it is, this could be quite revolutionary in the classroom.
• Hardware Startup Showcase. I have ideas. I'd like to see them get out of my head and into existence. Turns out there's even a MeetUp group.
• Kits by Kids. I'll have to check this out to see what sort of stuff I can do with Zoe when she's a bit older.

Kickstarter is really becoming huge in the maker community. There were heaps of exhibitors there with (mostly robotics) projects that were past the initial prototyping phase and were seeking funding on Kickstarter to go into mass production.

Some of the talks I'd have liked to have seen:

• Every Child a Maker
• "Kickstarters," Equity and the New Maker Pro
• Making Professional Makers - Career Tech Ed Instructors Talk Shop
• Manufacturing Open Source Hardware - Get Your Stuff Made!

Zoe was really well behaved for the entire expedition. I don't think she really gave me any grief at all. There was a brief period where she wanted me to carry her, but I managed to negotiate her back into the stroller after not long.

I think her favourite was ArcBotics, which had a robot insect that would dance and wave at her. She kept asking for it to do more dancing.

Back in November I ranted about the migration of Gnome Shell to Debian/Testing. Plenty of other people did the same thing (or have done the same thing about Unity).

I'd just like to say sorry to any of the GNOME people who felt unappreciated; I know you work hard to try and produce a useful user experience out of the box. I ended up doing the dist-upgrade on my work laptop only a week or so after my home machine, and in the process discovered that the nouveau Mesa driver now supports my machine pretty well. It's taken me a while to get used to it, but my frustrations with the change have diminished and I haven't felt the need to move to something different. So, a belated thanks for all your hard work.

I will start linking to places we stayed at etc, from now on. On the one hand, we already left those places so my paranoid self can rest assured that no one will come hunt us down, on the other hand, I decided to only to name and link to those places which really deserve a mention. If they are linked here, I can whole-heartedly recommend them.

Mongolia

After leaving through the Russian border fortifications, Mongolia was an instant and welcome change. Trees growing right next to the tracks, branches forming a bit of a natural tunnel; the trains passing by keeping branches at bay, but the trees free to grow wherever else they pleased. A few hundred meters later below a rocky ledge, a cow had apparently fallen down and died. Partially decomposed flesh still clinging halfway to its rib-cage, it proved to be a sign of the (mostly) more laid-back attitude Mongolians tend to take.

After finally getting our passports back and being allowed to leave the train, we first headed to the station to get some money. At ~1800:1, the exchange rate for Tugruk against Euro speaks volumes about inflation; imagine my surprise that the MNT 20,000 note is the largest one. It's also the one and only amount any and all ATMs will default to.

We were quite surprised by how cheap everything was, even in the one single store within the station directly behind the border. In Russia, travellers expect to pay hefty markups when shopkeepers know they can charge them, not so in Mongolia.

The woman in the trash

Upon returning from the station, our waggon was just merrily being driving away on its own by a lone engine, something we had not anticipated or appreciated a whole lot. A woman (we hadn't noticed her before) who was busy scavenging a trash container for food for herself and her scrawny-looking son happily stopped what she was doing and, with a big smile, moved her hands around and smacked them back together, signalling us that our waggon would be attached to a different train for the subsequent journey.

To put it mildly, I was stunned; we had met people eating from the trash in Russia as well (yes, I always make a point of giving them enough for a few decent meals; anyone eating out of the trash actually needs and definitely deserves something good happening to them), but they were glum and disheartened. Here, there was someone who not only had to look out for herself, but for a three or four year old boy as well, and still was just so... positive... I don't know if I would have it in me to act the same way...

As an aside, even though we were just a few kilometers across the border from Russia, she did not actually take the money from me, she cupped her hands to receive it by having me place it into her hands; this is a very Asian thing in my experience and something we would see both in Mongolia and China consistently.

Later, when we were back in our waggon, she passed by and happened to see the Spot Messgenger 2 dangling in front of our window, its lights blinking every five seconds. For a solid twenty minutes, until the train started moving again, she squatted down with her son, oggling the Spot. I have no idea what she was thinking it was and I had no chance to explain it to her, but she was really fascinated and just kept staring and staring.

When we finally left the station, we waved to and smiled at each other, she ran after us, and then she was gone. All in all, this was one the more memorable things which happened to us.

Mongolian Infrastructure

When I say that Mongolians as a whole tend to be a lot more laid back than Russians, this extends to infrastructure as well, but not in the good way.

• Where Russia uses concrete sleepers, Mongolia uses roughly-hewn wooden ones.
• Where Russia has two rail tracks, Mongolia has a single one which is extended by parking spots to let other trains pass then and now.
• Where Russia maintains a perimeter along their rail tracks, Mongolia has a mainly gapless barbed-wire fence against animals on both sides; said fence may be near or far, covered with bushes or standing on its own. Its there so serve a purpose, but that's it. There's nothing else in ways of track protection.
• Where Russia has well-built, if dirty, crossings with one design spanning across all of Russia, Mongolia has random crossings that happen to look like they look.
• While Russia's streets are covered with potholes, Mongolia's are covered with potcaves.
• Some smaller stations didn't warrant a stop, but they couldn't be passed at full speed, either. Basically, if someone was waiting at the station, the train would come to a complete stop. At night, attendants would lean out of the train with flashlights, scanning the station and its surroundings for potential customers. If anyone was found, they signalled the driver and we would stop. Strange, but it works.

Ulan Bator Little Dhingis Khan

We arrived in Ulan Bator at six in the morning and started haggling with taxi drivers. Our hostel had (stale, it turned out) information about acceptable fare from station to their place on their website, 5,000 MNT was supposedly OK. Our driver tried to get 20,000, then 15,000, then 12,000, then 10,000, went down to 7,000 and finally settled for 5,000, all this interspersed with frequent "no" by us and walking away. Once on the road, he wanted to convince me that we had agreed upon 5,000 per person, something that did not quite work out for him. When we arrived, he started to tell me that he was good friends with the hostel manager and that said manager would surely want us to pay more; again, that did not work too well. He made the mistake of getting our luggage out of his trunk immediately (we always kept all our stuff in the passenger area afterwards...) before I paid and even though I only had a 10,000 or 20,000 note, I insisted on correct change. The same old dance began anew, with him edging closer to the correct change in steps of 1,000 MNT. He tried to get away several times, but I stood in his door and held it open; if he had just driven off, I couldn't have stopped him, but that thankfully didn't occur to him. After he shoved all small change (MNT is bill only, no coins) he had with him at me and wanted to make off yet again, I was close to throwing all the small crap at him, but after he, a man of maybe 1.65 meter and sitting down, tried to signal his willingness for a fist fight and me, a man of 1.94 meter and standing started laughing, he gave me 2,000 more and drove off cursing.

In case you forgot, at that exchange rate, we argued about two to three Euro, but as he tried to cheat, it became a matter of principle.

The Hostel, part one

All parking lots and other property in Ulan Bator have a small watchtower with private 24/7 guards; said guard ambled down, let us in and woke the poor woman who was forced to let us check in. As she was Mongolian as well, we were a bit surprised when she switched from broken English to perfect German, but the owners of the OASIS are a German-Austrian couple so I guess that makes sense.

Fighting through the war

Traffic in Ulan Bator is war. Lonely Planet talks about the city being too newly motorized and that it did not have to develop a culture of driving, yet. This is a euphemism for "no one has any qualms maiming or killing you". Everyone is driving like mad, nudging and racing their way in front of each other. If a driver has an oppurtunity to get ahead by a few centimeters with his small car, thereby blocking the way for several buses, making a whole parking lot grind to a halt and thereby deadlocking themselves, they will take said opportunity, and gladly.

Red lights are gentle suggestions, there to be mainly ignored; police will not drive over red lights, but they will not stop anybody else from doing so, either. A hearse will have a dozen or more cars following it, all driving in a tight pack, aggressively attacking anyone who dares to wedge in between them and expecting all other traffic to make way even if they are driving over a red light.

Crossing a main street in Ulan Bator as a pedestrian is hell. There are a few traffic lights and they help to some extent, but you still need to be extremely careful when crossing a street. Thailand, India, Russia, China, no matter where it's supposedly hard to cross a street by foot, I never had any issues weaseling through. In UB, it takes planning, determination, and a lot of attention, especially as there are stretches where there's not a traffic light in sight. While maintaining eye contact with a driver usually ensures that they will try not to hit you, this does not work in the least in UB. In the evening, I took to shining a high-powered flashlight at the ground to the oncoming traffic's side to mark ourselves and decrease the likelihood of being hit. This worked well in combination with sprinting through gaps in the traffic, except for one guy who actively aimed for us and did not slow down. He even changed lanes, just to keep us in front of his car... Shining the flashlight directly at him in highest mode made him break quite violently though; a good thing, as he may have hit us had he not slowed down.

UB itself

Ulan Bator is weird.

It has clear signs of westernization, but it's still very much an old city. There is one large store which would fit into any random shopping center in Germany (which are tiny when compared to most other countries) and which carries most goods, if with a limited selection. Its supermarket is extremely well stocked with food and drink from all other the world and laughably cheap (for us) prices.

While its infrastructure is definitely crumbling, the city is clean, especially when compared to Russia. Two girls from Stuttgart who we met during our Trans-Siberian travels commented on how clean Russian cities were when compared to home, but that may be more of a statement about Stuttgart than about Russia...

Anyway, in Mongolia in general and in UB in particular, there's a concise effort to make what they have look nice and clean. There is no trash in the streets, loose gravel and earth is removed from the curb by dedicated workers, all areas of packed earth are sieved and all large stones removed. All in all, it's a lot cleaner (if more dusty and sandy) than Russian cities if somewhat older and more shabby-looking. That being said, Mongolia's countryside is littered

There were several street stalls consisting of nothing more than a cardboard box on a stool selling single chewing gums and cigarettes from the original packaging; it's normal for people to buy one cigarette or one piece of chewing gum and walk on.

The sights of UB were nice, but nothing too special.

Outlook

In the next part, I'll cover "The Hostel, part disaster" and generally talk more about excrement than I thought one could write before moving on to China (were I am located at the time of this writing).

I read a translated poem about Russia being "the Motherland" and its vast bosom years ago. Having driven through a significant part of it, I can agree on the "vast" part...

Also, as I am on a train and without access to the Internet, I will refrain from linking to a lot of pages; sorry. (Turns out I am posting this a week later, but I will still not link to stuff now; no time).

Russia in general

• All receipts you receive are torn before you get them; this is most likely due to the old Soviet voucher system, more on that below.
• Russia was hot with temperatures ranging from 27 to 32 degrees Celsius between Moscow and Ulan Ude.
• There aren't a lot of pedestrians bridges, but a lot of pedestrian tunnels. The sides of those tunnels are packed with tiny shops, often only two meters wide and 50-70 cm deep. Everything from stockings to candy over glasses to flowers and watches is being sold through a tiny window by some poor woman who somehow managed to get in there.
• Toilet brushes stand in water. In Germany, that's a sure sign of a really dirty toilet; in Russia, it's the thing to do. If you are lucky, there's blue cleaning stuff added to the water. If not, it will still have color. You are free to guess which.
• Queuing is war.

Moscow Sights Kreml

Our remaining time in Moscow was spent with touring the usual suspects; the Kreml is a lot less impressive in real life, the Red Square is tiny when compared to the stories I heard about it and the Chapel ofi St. Basil is even more colorful and impressive in real life. Lenin's body was inaccessible because workers built seats for the May 9th parade to the left and the right of it and they apparently thought it would be a good idea to block access to one of the main tourist attractions while doing so. A river tour of Moscow was a nice cool-off and we got to see quite a few things.

We managed to see the weekly military parade within the Kreml grounds, but it was mostly pomp and little substance. The National Treasure which you can access with an extra ticket within the Kreml grounds is nice, but less impressive than the tourist guides would make you believe. That being said... There's another museum within the museum and.... Whoah... Tourists pay extra, visitors go through the only non-security-theater check I encountered in Russia, guards are armed, people can only enter and leave in batches, and the stuff which is presented is mind-boggling. Disregarding the fist-to-calf-sized chunks of gold and platinum which are still in their original form directly from the mine, there is real, actual treasure galore. Little heaps of uncut and cut diamonds, an outline of Russia filled with cut diamonds and other random "we have this stuff" displays can be found as well. Then, you have various tiaras and other jewellery made from various gems. Not incorporating, but largely made of. All that pales in comparison to the crown, royal apple, scepter, etc. It's hard to put the amount of tiny multi-colored light points that shine at you into words. I was just standing there, swaying back and forth to catch the moving pattern of pinpoints. It's said that this collection is equalled only by the ones in the Tower of London and the one Shaw of Iran had and boy do I believe it.

TV Tower

Getting up there was funny.

The old-style Soviet queuing system was used:

• Go to a counter to tell an attendant what you want; receive stub
• Go to another counter, hand over stub to another attendant, pay for what you want; receive voucher
• Go to third counter, hand over voucher; receive ticket for tower The whole thing was made even more absurd by the fact that counter one was in the middle, counter two to the right and counter three to the left. As Russians do not believe in queuing and everybody just tries to get in first, this made for a nice little exercise.

"Security" for approaching the tower was multi-level, the guards see you approach along a long walkway way in advance and the main guard shed had several small cabins separated by thick glass. So good so menacing. But in a twist that would make Bizarro and Garry Larson proud, I was required, by means of metal detector gate, metal detector wand and even an x-ray machine to remove every shred of metal and other hard objects from myself and the camera bag and put them onto a table. Once I was without anything except my clothes and the bag was completely empty, I could pass. Everything I had had to remove was just laying there, not inspected in the least, for me to stuff back into pockets and bag and to take with me. This "everything" included a Spot Messenger 2 with lots of green and red blinky lights. The guard did not even glance and it. Security theater? Security theater.

The view from 364 meters down on Moscow was nice, but there was a lot of Smog so I couldn't see very far. Jumping on the glass floor while looking down was a lot of fun, though.

Subway to Thiefing

I bet Christopher Nolan rode the subway in Moscow at least once. That unnerving sound you hear during several key scenes in "The Dark Knight"? Two thirds of all subways make the same sound while moving.

Also, I had an encounter with a pickpocket down there; very classical, too. Guy approaches quickly, talks loudly and sounds as if it's really important (in Russian... duh... that's sure to keep me interested). His approach made me turn and protect my left leg pocket automatically, most likely marking the target for the tiny woman standing behind me. Now, I have to tell you something about my usual travel layout. As my normal pockets are very deep, it looks as if their content was in the leg pocket. Plus, there's an extra, hidden leg pocket where I keep the passports and train tickets. The outermost leg pocket is protected by a velcro flap, but it contains nothing of value; usually the appropriate phrasebook, local map, maybe a tissue or chewing gum. Due to this layering, the outermost pocket looks as if it's full to the brim with stuff. Also, I took pains to make it a habit to protect said leg pocket with my hand, nothing else. This looks as if that's the target, but what I am actually doing is protect my normal pocket with my forearm. The right side is different, but the most easily accessibly pocket always holds some small change. I pay from that stash but my actual wallet is well out of reach. Anyway, once the guy ran off, talking to several others, most likely marking all them for the actual pickpockets, I wanted to enter the subway. While the Russian-style queuing took place, I felt an unusual tug at the velcro flap. I looked down and saw a tiny woman to the left of me with a jacket held over her right side with the left arm; I look up to check no one is trying to steal from my permanently assigned female, feel another tug, look the woman into the eyes, look up again and around me, look down again and she is gone. All that took maybe three seconds and I had boarded the subway after an additional two.

In hindsight, it makes sense to choose the time of entry for attack. It's crowded, you are being pushed around, and once you are in the subway, it will start moving more or less immediately while the thief remains in the station.

In this case, she would only have gotten a grubby map of Moscow's subway and an English-Russian phrasebook, but she got nothing at all.

Moscow-Novosibirsk

Where to begin...

If you think a few hours on a train are a long time, try over fifty hours. Things get so bad, you start getting land-sick while not in a moving train. You even start missing the familiar tunk-cachunk, tunk-cachunk, tunk-cachunk... of driving over rails with gaps in them when you are not moving.

The defining element of the Trans-Siberian Railway are birch trees. And birch trees. And then more birch trees. You would not believe how many birch trees there are. This is made "worse" by the way the Russian Railway protects their rails. Left and right of the track, there's a cleared area of maybe ten to twenty meters, sometimes as little as three. Outside of that, they plant ten to twenty meters of birch trees, presumably to catch snow during winter. Beyond that protective perimeter, there's the normal landscape.As a result, on top of the near endless stretches of birch woods, you see most if not all scenery through a layer of birch trees. You get sick sick of birch trees after a few hours and you see them for days on end.

Bullet points to save myself some typing and you some reading...

• More than a thousand kilometers without a single hill. Flat as a pan.
• The whole route is powered by electricity. No diesel engines in sight.
• Many stations are little more than a heap of smoothed gravel, bordered by some wooden planks. Some stations have obviously been built by locals and are even less well-defined.
• You can see people in the middle of nowhere, walking along the railway tracks. At first this seemed counter-intuitive, but most if not all roads out there are dirt tracks. As there seems to be standing water across a third of Russia, this dirt is turned into mud. After walking maybe twenty meters across a parking lot, I had to scrape a heavy, thick cake of black earth from my soles. The railway is the only functioning footpath those people have. Many people even build shoddy bridges towards the tracks from their homes, obviously preferring to walk along the tracks over walking through the village.
• Railway crossings along the Trans-Siberian route, no matter how tiny, have a small cabin beside them. While the train passes, there's one guy or gal standing in said cabin, holding a yellow stick vertically out towards the train. Sometimes, you have not seen any living thing, other than birch trees, for twenty minutes and there, in the middle of nowhere right beside a dirt track, there's someone holding a stick out towards the train. Weird.
• Railway crossings of paved roads will always have two steel plates coming out of the ground, angled towards oncoming traffic on each side. This may not stop a heavy truck at full speed, but a car will disintegrate on these barriers without touching a passing train.
• The railways is important for Russia. Two parallel tracks cut across the whole country, transporting everything back and forth. Where "everything" means mostly coal and birch wood, I guess.
• All freight trains are usually 70 tanker waggons or 100 box waggons long, but you see the odd 100 tanker waggons, as well. You have more than enough time and opportunity to count them and then some.
• There are supposedly women at every station, selling what they cook at home. Unfortunately, this was only true for two stations. The things we did manage to get were very nice; I do wonder why anyone would offer (or buy) cooked and peeled potatoes, though.
• Every waggon has its own hot-water stove. They are powered by coal. Yep, you have a coal fire burning in every single waggon on the Trans-Siberian.

Novosibirsk The non-existent hostel

We arrived at ~0200 local and made our way to the hostel we had booked a room with. Walking to the correct address, we saw several signs but they all turned out to be for a police station and some other state agency. We walked back, forth, double-checked, triple-checked: no hostel. We then walked around the building through some not-quite-nice back alleys, but other than a few entries to private flats, there was nothing. Thankfully, the booking slip included a number which we called and after at least twenty rings (no kidding), when I had given up and wanted to hang up, it stopped ringing. Dead silence. After maybe ten seconds, someone started talking in Russian. I asked him if he spoke English and told him that we could not find the hostel. He mumbled something about being sorry and that we should wait, he would come down. Fast forward a minute or two and someone walked towards us.

Again, he mumbled about being sorry, that the hostel "did not work" at the moment and that we would need to sleep in his private apartment. He ushered us into some back alley entrance, into his flat, and proceeded to remove the sheets from the couch on which he had slept; after putting on new sheets, we had our "hostel" bed, ready to sleep on. We briefly considered if he would murder us in our sleep, but him and me even got to talking a bit. Over cheese, sausage and rum (at 0300), he admitted that the hostel did not exist and he merely planned to turn his flat into a hostel for the summer while he and his family moved into their summer house (the Russian term of which escapes me, at the moment) in the countryside. He had accepted our reservation as he thought he would be finished by that time. He did not even get started, though. While he sent us an overbooking notice through booking.com two days before, we were on the train at that time, so... booking.com even called him to check what happenend to us as we did not book another place through them. Good customer service/protection, that.

Next morning, he didn't even want to take our money (we paid anyway) and, as a means of compensation, drove us into the city in the morning and to a train museum well outside the city limits, one of the fabled scientist cities, and a large lake which everyone in Novosibirsk claims is an ocean, in the afternoon.

Foreigners, foreigners!

All in all, Novosibirsk was relatively uneventful, safe for one bizarre episode. We took our lunch in a local fast food joint (why do all the good stories happen there, and not at the various truly local places?) and threw the cashier our well-rehearsed "Niet Russkie; anglisky?" with phrasebook in hand and he actually understood a few words of English (beef, chicken, fries). We told him, in our worst Russian, that we are from Germany wished him a nice day and went to sit down. A few minutes later, a girl approached us, literally hopping from one foot to the other and wringing her hands. She told us that the cashier had told her that we spoke English and if it would be OK if she talked to us. We suspected some sort of elaborate ruse, but went with it. Turns out, she had English at school and really wanted someone to practice English on. Two young men passed our table and exchanged a few words with her, sitting down out of sight. When she told us that she had to leave now but if it would be OK if the two boys joined us we suspected a ruse yet again. But those two were law students, one with a minor in English and one with a minor in German; both of them also extremely nervous, asking us if we would talk to them. When they had to leave, they told us that the three of them worked at the burger joint and that their shift was just about to start when the news that foreigners were here spread amongst staff like wildfire. The girl stopped by several times in between cleaning tables, getting in a sentence or two before being cussed at by her supervisor. All in all, this took about twenty minutes and seeing three people so nervous and grateful to talk with us felt beyond absurd.

On the other hand, not a single traveller we met even considered stopping in Novosibirsk during their transit so there really does seem to be a shortage of non-Russians there.

Weird, and memorable.

Novosibirsk-Irkutsk

• Birch trees.
• Lots of burnt underwood, presumably to prevent larger fires.
• Birch trees.
• Sticky, stuffy, 30+ degree waggon with windows that could be opened but which were locked (this is why I always carry a Swisstool with me).
• Birch trees.

Irkutsk / Listvianka / Lake Baikal Listvianka

Aah, lake Baikal... the oldest and deepest lake on Earth which holds a fifth of the global non-salt water reserves; a must-see in my book.

Quad tours at break-neck speeds, dry-suit diving with Russian regulators, walking barefoot in between and across drift ice that made its way onto the shorei, and extended hiking around the lake's coast...

All of which I could not do because I was ill and had to spend two solid days in bed.

The draft from the open window in between Novosibirsk and Irkutsk was enough to give me a rather bad cold which peaked at Lake Baikal.

Still, the area was lovely and we were glad to be out of a train and able to unpack our stuff without having to repack immediately for once.

I am not sure where my current losing streak with regards to diving is coming from (Grimsey, diving north of the Arctic circle with birds that plummet into the water and hunt fish: Only guy who does this is on the Icelandic mainland that day; Svalbard, diving north of the Arctic circle in permanent darkness: The few people who do this privately did not reply while I was there; Baikal, oldest, deepest, largest lake on Earth: ill), but I will most likely return to Russia for a week of ice diving in Lake Baikal next winter or the one after that.

As an aside, I saw several people walking to Lake Baikal with buckets to get their water. Other people got it from a well which was still half frozen. If you have running water consider yourself lucky...

Irkutsk

Nice city, largely uneventful. The farther east you get within Russia, the more normal women look. In Moscow, just as in Paris, they are way over-dressed and even service personnel will walk with high heels. Thankfully, I don't have to wear heels, but for the other males out there: Walking and standing in these things hurts and thus most if not all people who stand and walk for a living have flat shoes.

We happened upon preparations for a military parade, complete with cordon, viewing podests, at least half a dozen TV cameras etc, but were not sure if it would start soon enough for us to catch our train.We asked someone who told us it would start at 2100 local, at 1945 local it seemed about to start, and sure enough at 1955 sharp, the whole thing went under way. About a dozen groups of 50-100 people each, all in their own, respective uniforms stood against one side of a cordoned-off street and several higher-ups on the other side. Two highest-ups shouted into microphones and the throng of people on the other side shouted back answers. Then, the two highest-ups stood in the back of a jeep each and drove past said throng, stopping in front of each group, shouting into microphones mounted in the back of the jeeps and the groups shouted back once again. After that, all groups marched around the make-shift plaza once, saluting the higher ups. Once they were done, and they took ages, two trucks drove by with soldiers jumping out of the moving trucks and moving into crouching positions. They ran around in a circle a few times and engaged in pretend hand-to-hand combat. I am sure they are skilled at whatever style they wanted to show, but they were overdoing things so badly, they were funny, not imposing. When they jumped over some barriers, the barriers fell to pieces and everyone scrambled to make it look as if that was part of the show. While carrying off the gear, it fell into further pieces which was even more funny. An armoured personnel carrier ended the show; several tougher looking guys jumped off of that one and their mock combat involved fully automatic fire (of blanks), several flashbangs, smoke grenades and, to top things off, the machine gun mounted on the APC moving down the opposing team with blanks.

I never witnessed a "real" military parade in person but this one was somewhat disappointing. On the one hand, there was a distinct lack of ballistic missile carriers and tanks like you see in movies, documentaries and games, on the other hand, the whole thing had a make-do feeling to it. The cordoning police had designated spots to stand on, yet walked around. They were standing to attention, yet checking their cell phones. Several people in one uniformed group were wearing track suits and jeans. Another uniformed guy had a grocery bag with him; yet another one was carrying a huge water bottle. Bikers zig-zagged through the cordon and when the whole show was just about to wrap up the police finally started putting up barriers around the unmoving pedestrians, not blocking the bikers. One little girl was standing well within the cordoned area, watching with big eyes and after she did not react to the police talking to her, they just built the barriers in a curve around her.

And to top it all off, some guy with a cane walked all through the parade with his personal camcorder, trying to direct the whole show while being ignored by everyone. Still, I am sure he managed to mess up some otherwise perfectly good TV scenes.

Irkutsk-Russian border

• Diesel-powered trains.
• Single track most of the time with frequent stops to let other trains pass.
• Distinctively less developed cities, stations, streets, and other infrastructure along the road.
• 32+ degrees in our waggon.
• The train attendant was extremely unfriendly and just generally miserable even by Russian standards.
  • No toilet paper or towels at all on toilets.
  • While the other attendants made a point of presenting themselves well, he shuffled around in slacks all the time (not bad per se, but Russia is big on uniforms, so...)
  • He took all our tickets and stubs (including the ones not from this part of the journey) and kept them without comment. After we asked for them several times, he barked at us that we would get them back before Ulan Bator. Why? No idea...
  • He refused to let us exit the train during the very few stops. We were unable to exit through other waggons as the connecting door was locked. Being stuck in a train sucks.
• Border and customs took NINE HOURS!!! Stuck in blistering heat without a breeze, without access to a toilet, just waiting for bureaucracy to go its way. I checked all doors, we were locked into said waggon and there were no 'break glass to leave in emergency' windows. Especially nice as there's a coal fire burning in the hot-water stove and the whole train is plastered with warning signs about fire and what to do. In our case, presumably, burn to death; preferably without disturbing the attendant.
• The Russian stamp for entering Russia (by plane) has a plane on it, the departure one a train.
• The Russian side of the border is built like a fortress. There are several towers and bridges over the rails so trains can be checked from above, and reinforced holes dug into the ground in which soldiers stand and check the train from below.

TL;DR

3000 kilometers of birch trees

I'll start here a small series of posts about ganeti, xen and puppet. For my work I run few servers sitting on xen and it has always been a bit of a pain to create a new instance and keep it up to date. Up to now I've used the excellent xen-create-image tool to create my VMs, but I wanted to try something new and more sexy... Last week I finally found some time to learn (and a spare box to run my experiments) how to use ganeti. Ganeti is the only tool I tried out, but it seems to fit the bill for my use and it seems polished and mature project to me... Moreover I've seen a presentation about it in every FLOSS conference I've attended in the last few years and I thought it was time to give it a try.

Installing and configuring ganeti is fairly easy, there is a lot of documentation available and this post is not going to be about installing it, but rather how to create a new bare instance with ganeti-deboostrap-instance. There is also a way to create a new instance from an image, but I didn't go that way yet.

This first post is about the first problem I've encountered, that is, how to automatically assign a network address and a name to each new instance created by gnt-instance add. Since all my instances should be able to communicate together on the same subnet, I've decided to configure xen to create a NATted private network and add each new instance to this network.

The first step is to create an interface in /etc/network/interfaces .

auto xen-br0
iface xen-br0 inet static
    address 10.0.0.1
    netmask 255.255.255.0
    bridge_stp off
    bridge_fd 0
    bridge_ports none

This is the standard debian way but since xen uses a different naming convention (here I'm using ganeti naming convention xenbr0 vs xen-br0), I need to convince tell xen what I intend to do by adding these lines in /etc/xen/xend.config :

(network-script 'network-virtual bridgeip="10.0.0.1/24" brnet="10.0.0.0/24" bridge="xen-br0"')
(vif-script     vif-bridge)

Next I have to connect my real network interface to the private network using few iptables rules in /etc/rc.local (probably there is a better place to do this...):

echo 1 > /proc/sys/net/ipv4/ip_forward
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -A FORWARD -i eth0 -o xen-br0 -m state --state RELATED,ESTABLISHED -j ACCEPT
/sbin/iptables -A FORWARD -i xen-br0 -o eth0 -j ACCEPT

The xen setup is complete and every new image should have a vif connected to the subbet 10.0.0.0. The xen setup corresponds to the physical wiring of the network. The next step is to configure each instance so to allow them to communicate on this subnet. Since I build my VMs using ganeti-debootstrap-instance, and by default debootstrap does not configure the network, we need to add a new hook in the directory /etc/ganeti/instance-debootstrap/hooks.

#!/bin/bash
if [ -z "$TARGET" -o ! -d "$TARGET" ]; then
  echo "Missing target directory"
  exit 1
fi

if [ ! -d "$TARGET/etc/network" ]; then
  echo "Missing target network directory"
  exit 1
fi

if [ -z "$NIC_COUNT" ]; then
  echo "Missing NIC COUNT"
  exit 1
fi

if [ "$NIC_COUNT" -gt 0 ]; then

  cat > $TARGET/etc/network/interfaces <<EOF
# This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet dhcp

EOF

fi

DAEMON_PID_FILES="/var/run/dnsmasq.pid /var/run/dnsmasq/dnsmasq.pid"

instance=$INSTANCE_NAME
[ -n "$instance" ] || exit 1
nic_count=$((NIC_COUNT - 1))
mac_var="NIC_${nic_count}_MAC"
echo $mac_var
echo $nic_count
mac=${!mac_var}
echo $mac
echo "dhcp-host=$mac,$instance" > /etc/dnsmasq.d/$instance.conf

This hook will do two things. First it will configure the interfaces of the new instance to get configured using dhcp, second, it will add an entry to the dnsmasq configuration file to make this instance known to the world. This basically boils down to add a file in /etc/dnsmasq.d/ with the mac address of the new instance and its designated name. Dnsmasq will then provide an ip address for this instance and add it to the dns.

dhcp-host=aa:00:00:24:6c:8a,node1

Configuring dnsmasq is pretty easy as well. First I want it to answer dhcp queries only on the internal network, second I want to configure my clients passing 10.0.0.1 as nameserver and gataway. You can just add the following lines in /etc/dnsmasq.d/general to get it going.

interface=xen-br0
interface=lo
dhcp-range=10.0.0.128,10.0.0.250
domain=localnet.org,10.0.0.128,10.0.0.250
dhcp-option=3,10.0.0.1
bogus-priv
#expand-hosts
local=/localnet.org/

To create your new instance you can just run the following command :

gnt-instance add -t plain -s 5g -B memory=1024 -o  debootstrap+unstable --no-ip-check --no-name-check node1

I use --no-ip-check and --no-name-check to skip ip and dns check performed by ganeti and to avoid a sort of chicken-egg problem, where the name and address of this new instance is yet unknown to dnsmasq and that node1 is the name that will be used by the hook to add an entry in the dnsmasq configuration. debootstrap+unstable is a variant of the default configuration and you need to add it to the list of variants used by ganeti-deboostrap-instance.

That should be it. The new instance should come up with a dynamically assigned ip address, able to talk to the outside world and automatically known by all the other machine on the subnet via dns.

Next post will be about how to add a swap hook for ganete-debootstrap-istance.