FLOSS Project Planets

Steve Kemp: Switched to using attic for backups

Planet Debian - Fri, 2014-12-19 08:51

Even though seeing the word attic reminds me too much of leaking roofs and CVS, I've switched to using the attic backup tool.

I want a simple system which will take incremental backups, perform duplication-elimination (to avoid taking too much space), support encryption, and be fast.

I stopped using backup2l because the .tar.gz files were too annoying, and it was too slow. I started using obnam because I respect Lars and his exceptionally thorough testing-regime, but had to stop using it when things started getting "too slow".

I'll document the usage/installation in the future. For the moment the only annoyance is that it is contained in the Jessie archive, not the Wheezy one. Right now only 2/19 of my hosts are Jessie.

Categories: FLOSS Project Planets

Python Anywhere: #VATMESS - or, how a taxation change took 4 developers a week to handle

Planet Python - Fri, 2014-12-19 07:36

A lot of people are talking about the problems that are being caused by a recent change to taxation in the EU; this TechCrunch article gives a nice overview of the issues. But we thought it would be fun just to tell about our experience - for general interest, and as an example of what one UK startup had to do to implement these changes. Short version: it hasn't been fun.

If you know all about the EU VAT changes and just want to know what we did at PythonAnywhere, click here to skip the intro. Otherwise, read on...

The background

"We can fight over what the taxation levels should be, but the tax system should be very, very simple and not distortionary." - Adam Davidson

The tax change is, in its most basic form, pretty simple. But some background will probably help. The following is simplified, but hopefully reasonably clear.

VAT is Value Added Tax, a tax that is charged on pretty much all purchases of anything inside the EU (basic needs like food are normally exempt). It's not dissimilar to sales or consumption tax, but the rate is quite high: in most EU countries it's something like 20%. When you buy (say) a computer, VAT is added on to the price, so a PC that the manufacturer wants EUR1,000 for might cost EUR1,200 including VAT. Prices for consumers are normally quoted with VAT included, when the seller is targetting local customers. (Companies with large numbers of international customers, like PythonAnywhere, tend to quote prices without VAT and show a note to the effect that EU customers have to pay VAT too.)

When you pay for the item, the seller takes the VAT, and they have to pay all the VAT they have collected to their local tax authority periodically. There are various controls in place to make sure this happens, that people don't pay more or less VAT than they've collected, and in general it all works out pretty simply. The net effect is that stuff is more expensive in Europe because of tax, but that's a political choice on the part of European voters (or at least their representatives).

When companies buy stuff from each other (rather than sales from companies to consumers) they pay VAT on those purchases if they're buying from a company in their own country, but they can claim that VAT back from their local tax authorities (or offset it against VAT they've collected), so it's essentially VAT-free. And when they buy from companies in other EU countries or internationally, it's VAT-free. (The actual accounting is a little more complicated, but let's not get into that.)

What changed

"Taxation without representation is tyranny." - James Otis

Historically, for "digital services" -- a category that includes hosting services like PythonAnywhere, but also downloaded music, ebooks, and that kind of thing -- the rule was that the rate of VAT that was charged was the rate that prevailed in the country where the company doing the selling was based. This made a lot of sense. Companies would just need to register as VAT-collecting businesses with their local authorities, charge a single VAT rate for EU customers (apart from sales to other VAT-registered businesses in other EU countries), and pay the VAT they collected to their local tax authority. It wasn't trivially simple, but it was doable.

But, at least from the tax authorities' side, there was a problem. Different EU countries have different VAT rates. Luxembourg, for example, charges 15%, while Hungary is 27%. This, of course, meant that Hungarian companies were at a competitive disadvantage to Luxembourgeoise companies.

There's a reasonable point to be made that governments who are unhappy that their local companies are being disadvantaged by their high tax rates might want to consider whether those high tax rates are such a good idea, but (a) we're talking about governments here, so that was obviously a non-starter, and (b) a number of large companies had a strong incentive to officially base themselves in Luxembourg, even if the bulk of their business -- both their customers and their operations -- was in higher-VAT jurisdictions.

So, a decision was made that instead of basing the VAT rate for intra-EU transactions for digital services on the VAT rate for the seller, it should be based on the VAT rate for the buyer. The VAT would then be sent to the customer's country's tax authority -- though to keep things simple, each country's tax authority would set up a process where the company's local tax authority would collect all of the money from the company along with a file saying how much was meant to go to each other country, and they'd handle the distribution. (This latter thing is called, in a wonderful piece of bureaucratese, a "Mini One-Stop Shop" or MOSS, which is why "VATMOSS" has been turned into a term for the whole change.)

As these kind of things go, it wasn't too crazy a decision. But the knock-on effects, both those inherent in the idea of companies having to charge different tax rates for different people, but also those caused by the particular way the details of laws have been worked out, have been huge. What follows is what we had to change for PythonAnywhere. Let's start with the basics.

Different country, different VAT rate

"The wisdom of man never yet contrived a system of taxation that would operate with perfect equality." - Andrew Jackson

Previously we had some logic that worked out if a customer was "vattable". A user in the UK, where we're based, is vattable, as is a non-business anywhere else in the EU. EU-based businesses and anyone from outside the EU were non-vattable. If a user was vattable, we charged them 20%, a number that we'd quite sensibly put into a constant called VAT_RATE in our accounting system.

What needed to change? Obviously, we have a country for each paying customer, from their credit card/PayPal details, so a first approximation of the system was simple. We created a new database table, keyed on the country, with VAT rates for each. Now, all the code that previously used the VAT_RATE constant could do a lookup into that table instead.

So now we're billing people the right amount. We also need to store the VAT rate on every invoice we generate so that we can produce the report to send to the MOSS, but there's nothing too tricky about that.

Simple, right? Not quite. Let's put aside that there's no solid source for VAT rates across the EU (the UK tax authorities recommend that people look at a PDF on an EU website that's updated irregularly and has a table of VAT rates using non-ISO country identifiers, with the countries' names in English but sorted by their name in their own language, so Austria is written "Austria" but sorted under "O" for "Österreich").

No, the first problem is in dealing with evidence.

Where are you from?

"Extraordinary claims require extraordinary evidence." - Carl Sagan

How do you know which country someone is from? You'd think that for a paying customer, it would be pretty simple. Like we said a moment ago, they've provided a credit card number and an address, or a PayPal billing address, so you have a country for them. But for dealing with the tax authorities, that's just not enough. Perhaps they feared that half the population of the EU would be flocking to Luxembourgeoise banks for credit cards based there to save a few euros on their downloads.

What the official UK tax authority guidelines say regarding determining someone's location is this (and it's worth quoting in all its bureaucratic glory):

1.5 Record keeping

If the presumptions referred above don’t apply, you’ll be expected to obtain and to keep in your records 2 pieces of non-contradictory evidence from the following list to support your taxing decisions. Examples include:

  • the billing address of the customer
  • the Internet Protocol (IP) address of the device used by the customer
  • location of the bank
  • the country code of SIM card used by the customer
  • the location of the customer’s fixed land line through which the service is supplied to him
  • other commercially relevant information (for example, product coding information which electronically links the sale to a particular jurisdiction)

Once you have 2 pieces of non-contradictory evidence that is all you need and you don’t need to collect any further supporting evidence. This is the case even if, for example, you obtain a third piece of evidence which happens to contradict the other 2 pieces of information. You must keep VAT MOSS records for a period of 10 years from 31 December of the year during which the transaction was carried out.

For an online service paid with credit cards, like PythonAnywhere, the only "pieces of evidence" we can reasonably collect are your billing address and your IP address.

So, our nice simple checkout process had to grow another wart. When someone signs up, we have to look at their IP address. We then compare this with a GeoIP database to get a country. When we have both that and their billing address, we have to check them:

  • If neither the billing address nor the IP address is in the EU, we're fine -- continue as normal.
  • If either the billing address or the IP address is in the EU, then:
    • If they match, we're OK.
    • If they don't match, we cannot set up the subscription. There is quite literally nothing we can do, because we don't know enough about the customer to work out how much VAT to charge them.

We've set things up so that when someone is blocked due to a location mismatch like that, we show the user an apologetic page, and the PythonAnywhere staff get an email so that we can talk to the customer and try to work something out. But this is a crazy situation:

  • If you're a British developer with a UK credit card, currently on a business trip to Germany, then you can't sign up for PythonAnywhere. This does little good for the free movement of goods and services across the EU.
  • If you're an American visiting the UK and want to sign up for our service, you're equally out of luck. This doesn't help the EU's trade balance much.
  • If you're an Italian developer who's moved to Portugal, you can't sign up for PythonAnywhere until you have a new Portuguese credit card. This makes movement of people within the EU harder.

As far as we can tell (and we've spoken to our [very expensive] accountants about this) there's no other way to implement the rules in a manner consistent with the guidelines.

This sucks.

And it's not all. Now we need to deal with change...


"Time changes everything except something within us which is always surprised by change." - Thomas Hardy

VAT rates change (normally upwards). When you're only dealing with one country's VAT rate, this is pretty rare; maybe once every few years, so as long as you've put it in a constant somewhere and you're willing to update your system when it changes, you're fine. But if you're dealing with 28 countries, it becomes something you need to plan for happening pretty frequently, and it has to be stored in data somewhere.

So, we added an extra valid_from field to our table of VAT rates. Now, when we look up the VAT rate, we plug today's date into the query to make sure that we've got the right VAT rate for this country today. (If you're wondering why we didn't just set things up with a simple country-rate mapping that would be updated at the precise point when the VAT rate changed, read on.)

No big deal, right? Well, perhaps not, but it's all extra work. And of course we now need to check a PDF on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard" to see when it changes. We're hoping a solid API for this will crop up somewhere -- though there are obviously regulatory issues there, because we're responsible for making sure we use the right rate, and we can't push that responsibility off to someone else. The API would need to be provided by someone we were 100% sure would keep it updated at least as well as we would ourselves -- so, for example, a volunteer-led project would be unlikely to be enough.

So what went wrong next? Let's talk about subscription payments.

Reconciling ourselves to PayPal

"Nothing changes like changes, because nothing changes but the changes." - Gary Busey

Like many subscription services, at PythonAnywhere we use external companies to manage our billing. The awesome Stripe handle our credit card payments, and we also support PayPal. This means that we don't need to store credit card details in our own databases (a regulatory nightmare) or do any of the horrible integration work with legacy credit card processing systems.

So, when you sign up for a paid PythonAnywhere account, we set up a subscription on either Stripe or PayPal. The subscription is for a specific amount to be billed each month; on Stripe it's a "gross amount" (that is, including tax) while PayPal split it out into separate "net amount" and a "tax amount". But the common thing between them is that they don't do any tax calculations themselves. As international companies having to deal with billing all over the world, this makes sense, especially given that historically, say, a UK company might have been billing through their Luxembourg subsidiary to get the lower VAT rate, so there's no safe assumptions that the billing companies could make on their customers' behalf.

Now, the per-country date-based VAT rate lookup into the database table that we'd done earlier meant that when a customer signed up, we'd set up a subscription on PayPal/Stripe with the right VAT amount for the time when the subscription was created. But if and when the VAT rate changed in the future, it would be wrong. Our code would make sure that we were sending out billing reminders for the right amount, but it wouldn't fix the subscriptions on PayPal or Stripe.

What all that means is that when a VAT rate is going to change, we need to go through all of our customers in the affected country, identify if they were using PayPal or Stripe, then tell the relevant payment processor that the amount we're charging them needs to change.

This is tricky enough as it is. What makes it even worse is an oddity with PayPal billing. You cannot update the billing amount for a customer in the 72 hours before a payment is due to be made. So, for example, let's imagine that the VAT rate for Malta is going to change from 18% to 20% on 1 May. At least three days before, you need to go through and update the billing amounts on subscriptions for all Maltese customers whose next billing date is after 1 May. And then sometime after you have to go through all of the other Maltese customers and update them too.

Keeping all of this straight is a nightmare, especially when you factor in things like the fact that (again due to PayPal) we actually charge people one day after their billing date, and users can be "delinquent" -- that is, their billing date was several days ago but due to (for example) a credit card problem we've not been able to charge them yet (but we expect to be able to do so soon). And so on.

The solution we came up with was to write a reconciliation script. Instead of having to remember "OK, so Malta's going from 18% to 20% on 1 May so we need to run script A four days before, then update the VAT rate table at midnight on 1 May, then run script B four days after", and then repeat that across all 28 countries forever, we wanted something that would run regularly and would just make everything work, by checking when people are next going to be billed and what the VAT rate is on that date, then checking what PayPal and Stripe plan to bill them, and adjusting them if appropriate.

A code sample is worth a thousand words, so here's the algorithm we use for that. It's run once for every EU user (whose subscription details are in the subscription parameter). The get_next_charge_date_and_gross_amount and update_vat_amount are functions passed in as a dependency injection so that we can use the same algorithm for both PayPal and Stripe.

def reconcile_subscription( subscription, get_next_charge_date_and_gross_amount, update_vat_amount ): next_invoice_date = subscription.user.get_profile().next_invoice_date() next_charge_date, next_charge_gross_amount = get_next_charge_date_and_gross_amount(subscription) if next_invoice_date < datetime.now(): # Cancelled subscription return if next_charge_date < next_invoice_date: # We're between invoicing and billing -- don't try to do anything return expected_next_invoice_vat_rate = subscription.user.get_profile().billing_vat_rate_as_of(next_invoice_date) expected_next_charge_vat_amount = subscription.user.get_profile().billing_net_amount * expected_next_invoice_vat_rate if next_charge_gross_amount == expected_next_charge_vat_amount + subscription.user.get_profile().billing_net_amount: # User has correct billing set up on payment processor return # Needs an update update_vat_amount(subscription, expected_next_charge_vat_amount)

We're pretty sure this works. It's passed every test we've thrown at it. And for the next few days we'll be running it in our live environment in "nerfed" mode, where it will just print out what it's going to do rather than actually updating anything on PayPal or Stripe. Then we'll run it manually for the first few weeks, before finally scheduling it as a cron job to run once a day. And then we'll hopefully be in a situation where when we hear about a VAT rate change we just need to update our database with a new row with the appropriate country, valid from date, and rate, and it will All Just Work.

(An aside: this probably all comes off as a bit of a whine against PayPal. And it is. But they do have positive aspects too. Lots of customers prefer to have their PythonAnywhere accounts billed via PayPal for the extra level of control it gives them -- about 50% of new subscriptions we get use it. And the chargeback model for fraudulent use under PayPal is much better -- even Stripe can't isolate you from the crazy-high chargeback fees that banks impose on companies when a cardholder claims that a charge was fraudulent.)

In conclusion

"You can't have a rigid view that all new taxes are evil." - Bill Gates

The changes in the EU VAT legislation came from the not-completely-unreasonable attempt by the various governments to stop companies from setting up businesses in low-VAT countries for the sole purpose of offering lower prices to their customers, despite having their operations on higher-VAT countries.

But the administrative load placed on small companies (including but not limited to tech startups) is large, it makes billing systems complex and fragile, and it imposes restrictions on sales that are likely to reduce trade. We've seen various government-sourced estimates of the cost of these regulations on businesses floating around, and they all seem incredibly low.

At PythonAnywhere we have a group of talented coders, and it still took over a week out of our development which we could have spent working on stuff our customers wanted. Other startups will be in the same position; it's an irritation but not the end of the world.

For small businesses without deep tech talent, we dread to think what will happen.

Categories: FLOSS Project Planets

Petter Reinholdtsen: Of course USA loses in cyber war - NSA and friends made sure it would happen

Planet Debian - Fri, 2014-12-19 07:10

So, Sony caved in (according to Rob Lowe) and demonstrated that America lost its first cyberwar (according to Newt Gingrich). It should not surprise anyone, after the whistle blower Edward Snowden documented that the government of USA and their allies for many years have done their best to make sure the technology used by its citizens is filled with security holes allowing the secret services to spy on its own population. No one in their right minds could believe that the ability to snoop on the people all over the globe could only be used by the personnel authorized to do so by the president of the United States of America. If the capabilities are there, they will be used by friend and foe alike, and now they are being used to bring Sony on its knees.

I doubt it will a lesson learned, and expect USA to lose its next cyber war too, given how eager the western intelligence communities (and probably the non-western too, but it is less in the news) seem to be to continue its current dragnet surveillance practice.

There is a reason why China and others are trying to move away from Windows to Linux and other alternatives, and it is not to avoid sending its hard earned dollars to Cayman Islands (or whatever tax haven Microsoft is using these days to collect the majority of its income. :)

Categories: FLOSS Project Planets

Petter Reinholdtsen: Of course USA looses in cyber war - NSA and friends made sure it would happen

Planet Debian - Fri, 2014-12-19 07:10

So, Sony caved in (according to Rob Lowe) and demonstrated that America lost its first cyberwar (according to Newt Gingrich). It should not surprise anyone, after the whistle blower Edward Snowden documented that the government of USA and their allies for many years have done their best to make sure the technology used by its citizens is filled with security holes allowing the secret services to spy on its own population. No one in their right minds could believe that the ability to snoop on the people all over the globe could only be used by the personnel authorized to do so by the president of the United States of America. If the capabilities are there, they will be used by friend and foe alike, and now they are being used to bring Sony on its knees.

I doubt it will a lesson learned, and expect USA to loose its next cyber war too, given how eager the western intelligence communities (and probably the non-western too, but it is less in the news) seem to be to continue its current dragnet surveillance practice.

There is a reason why China and others are trying to move away from Windows to Linux and other alternatives, and it is not to avoid sending its hard earned dollars to Cayman Islands (or whatever tax haven Microsoft is using these days to collect the majority of its income. :)

Categories: FLOSS Project Planets

Kenshi Muto: smart "apt" command

Planet Debian - Fri, 2014-12-19 05:12

During evaluating Jessie, I found 'apt' command and noticed it was pretty good for novice-usual users.

Usage: apt [options] command CLI for apt. Basic commands: list - list packages based on package names search - search in package descriptions show - show package details update - update list of available packages install - install packages remove - remove packages upgrade - upgrade the system by installing/upgrading packages full-upgrade - upgrade the system by removing/installing/upgrading packages edit-sources - edit the source information file

'apt list' is like a combination of 'dpkg -l' + 'apt-cache pkgnames'. 'apt search' is a bit slower than 'apt-cache search' but provides with useful information. 'apt show' formats bytesizes and hides some (for experts) fields. install/remove/upgrade/full-upgrade are mostly same as apt-get. 'apt edit-sources' opens a editor and checks the integrity.

So, I'd like to recommend 'apt' command to Debian users.

Well, why did I write this entry...? ;) I found a mistranslation I had made in ja.po of apt. Because it is critical mistranslation (Japanese users will confuse by it), I want to fix it strongly.

Dear apt deity maintainers, could you consider to update apt for Jessie? (#772678. FYI there are other translation updates also: #772913, #771982, and #771967)

Categories: FLOSS Project Planets

Django Weblog: DjangoCon Europe 2015

Planet Python - Fri, 2014-12-19 04:34

2015's DjangoCon Europe will take place in Cardiff, Wales, from the 2nd to the 7th June, for six days of talks, tutorials and code. Here’s a snapshot of what's in store.

Six whole days

For the first time ever, we’re holding a six-day DjangoCon.

The event will begin with an open day. All sessions on the open day - talks and tutorials of various different kinds - will be free, and open to the public. (This follows the example of Django Weekend Cardiff, where the open day proved hugely successful.)

The open day will:

  • help introduce Django to a wider audience
  • give newer members of the community a headstart, to help them get the most from the following five days

There’ll be a DjangoGirls workshop on our open day, and lots more besides.

Two days of code

Following the three days of talks, we won’t just have two days of sprints: they will be two days of code: code sprints, code clinics and workshops. We want everyone to have a reason to stay on after the talks, and participate, whatever their level of experience.

All these sessions will be free of charge to conference attendees. Some of them will be worth the ticket price of the entire conference on their own.


We aim to put on a first-class technical conference. We also want the event to embody three very important values:


We aim to make DjangoCon Europe 2015 a milestone in the Django community's effort to improve diversity.


We want this DjangoCon to set the highest standards for accessibility, and to ensure that we do not inadvertently exclude anyone from participating fully in the event.

Social responsibility

DjangoCon Europe 2015 expresses the Django community's values of fairness, respect and consideration as undertakings of social responsibility.


We’re sure you’ll enjoy the city and our venues, and we're looking forward to welcoming you in June. Here's some comprehensive information on how to get to Cardiff.

Social events

We have a number of social events planned - more details will be published soon.

Registration, call for papers and other key milestones

Here's a list of key dates. Ticket prices will be published when registration opens.

And finally

We're seeking sponsorship, of course, and would love to hear from any organisations that can contribute financial support to the event.

There's contact information on the website. If there's anything you want to know, you need just ask - the organising committee are at your disposal.

Categories: FLOSS Project Planets

On Domain Models and Layers in kdepim

Planet KDE - Fri, 2014-12-19 04:28

In our current kdepim code we use some classes throughout the codebase. I’m going to line out the problems with that and propose how we can do better.

The Application Domain

Each application has a “domain” it was created for. KOrganizer has for instance the calendar domain, and kmail the email domain, and each of those domains can be described with domain objects, which make up the domain model. The domain model of an application is essential, because it is what defines how we can represent the problems of that domain. If Korganizer didn’t have a domain model with attendees to events, we wouldn’t have any way to represent attendees internally, and thus couldn’t develop a feature based on that.

The logic implementing the functionality on top of those domain objects is the domain logic. It implements for instance what has to happen if we remove an event from a calendar, or how we can calculate all occurrences of a recurring event.

In the calendaring domain we use KCalCore to provide many of those domain objects and a large part of the domain logic. KCalCore::Event for instance, represents an event, can hold all necessary data of that event, and has the domain logic directly built-in to calculate recurrences.
Since it is a public library, it provides domain-objects and the domain-logic for all calendaring applications, which is awesome, right? Only if you use it right.


KCalCore provides additionally to the containers and the calendaring logic also serialization to the iCalendar format, which is also why it more or less tries to adhere to the iCalendar RFC, for both representation an interpretation of calendaring data. This is of course very useful for applications that deal with that, and there’s nothing particularly wrong with it. One could argue that serialization and interpretation of calendaring data should be split up, but since both is described by the same RFC I think it makes a lot of sense to keep the implementations together.


A problem arises when classes like KCalCore::Event are used as domain objects, and interface for the storage layer, and as actual storage format, which is precisely what we do in kdepim.

The problem is that we introduce very high coupling between those components/layers and by choosing a library that adheres to an RFC the whole system is even locked down by a fully grown specification. I suppose that would be fine if only one application is using the storage layer,
and that application’s sole purpose is to implement exactly that RFC and nothing else, ever. In all other cases I think it is a mistake.

Domain Logic

The domain logic of an application has to evolve with the application by definition. The domain objects used for that are supposed to model the problem at hand precisely, in a way that a domain logic can be built that is easy to understand and evolve as requirements change. Properties that are not used by an application only hide the important bits of a domain objects, and if a new feature is added it must be possible to adjust the domain object to reflect that. By using a class like KCalCore::Event for the domain object, these adjustments become largely impossible.

The consequence is that we employ workarounds everywhere. KCalCore doesn’t provide what you need? Simply store it as “custom property”. We don’t have a class for calendars? Let’s use Akonadi::Collection with some custom attributes. Mechanisms have been designed to extend these rigid structures so we can at least work with it, but that only lead to more complex code that is ever harder to understand.

Instead we could write domain logic that expresses precisely what we need, and is easier to understand and maintain.

Zanshin for instance took the calendaring domain, and applied the GettingThingsDone (GTD) methodology to it. It takes a rather simple approach to todo’s and initially only required a description, a due date and a state. However, it introduced the notion that only “projects” can have subtodo’s. This restriction needs to be reflected in the domain model, and implemented in the domain logic.
Because there are no projects in KCalCore, it was simply defined that todo’s with a magic property “X-Project” are defined as project. There’s nothing wrong with that itself, but you don’t want to litter your code with “if (todo->hasProperty(X-Project))”. So what do you do? You create a wrapper. And that wrapper is now already your new domain object with a nice interface. Kevin fortunately realized that we can do better, and rewrote zanshin with its own custom domain objects, that simply interface with the KCalCore containers in a thin translation layer to akonadi. This made the code much clearer, and keeps those “x-property”-workarounds in one place only.


A useful approach to think about application architecture are IMO layers. It’s not a silver bullet, and shouldn’t be done too excessively I think, but in some cases layer do make a lot of sense. I suggest to think about the following layers:

  • The presentation layer: Displays stuff and nothing else. This is where you expose your domain model to the UI, and where your QML sits.
  • The domain layer: The core of the application. This is where all the useful magic happens.
  • The data access layer: A thin translation layer between domain and storage. It makes it possible to use the same storage layer from multiple domains and to replace the storage layer without replacing all the rest.
  • The storage layer: The layer that persists the domain model. Akonadi.

By keeping these layer’s in mind we can do a better job at keeping the coupling at a reasonable level, allowing individual components to  change as required.

The presentation layer is required in any case if we want to move to QML. With QML we can no longer have half of the domain logic in the UI code, and most of the domain model should probably be exposed as a model that is directly accessible by QML.

The data access layer is where akonadi provides a standardized interface for all data, so multiple applications can shared the same storage layer. This is currently made up by the i.e. KCalCore for calendars, the akonadi client API, and a couple of akonadi objects, such as Akonadi::Item and Akonadi::Collection. As this layer defines what data can be accessed by all applications, it needs to be flexible and likely has to be evolved frequently.

The way forward

For akonadi’s client API, aka the data access layer, I plan on defining a set of interfaces for things like calendars, events, mailfolders, emails, etc. This should eventually replace KCalCore, KContacts and friends from being the canonical interface to the data.

Applications should eventually move to their own domain logic implementation. For reading and structuring data, models are IMO a suitable tool, and if we design them right this will also pave the way for QML interfaces. Of course i.e. KCalCore still has its uses for its calendaring routines, or as a serialization library to create iTip messages, but we should IMO stop using it for everything. The same of course applies to KContacts.

What we still could do IMO, is share some domain logic between applications, including some domain objects. A KDEPIM::Domain::Contact could be used across applications, just like KContact::Adressee was. This keeps different applications from implementing the same logic, but of course also introduces coupling between those again.

What IMO has to stay separate is the data access layer, which implements an interface to the storage layer, and that doesn’t necessarily conform to the domain layer (you could i.e. store “Blog posts” as notes in storage). This separation is IMO useful, as I expect the application domain to evolve separately from what actual storage backends provide (see zanshin).

This is of course quite a chunk of work, that won’t happen at once. But need to know now where we want to end up in a couple of years, if we intend to ever get there.

Categories: FLOSS Project Planets

Python Anywhere: New PythonAnywhere update: Mobile, UI, packages, reliability, and the dreaded EU VAT change

Planet Python - Fri, 2014-12-19 04:23

We released a bunch of updates to PythonAnywhere today :-) Short version: we've made some improvements to the iPad and Android experience, applied fixes to our in-browser console, added a bunch of new pre-installed packages, done a big database upgrade that should make unplanned outages rarer and shorter, and made changes required by EU VAT legislation (EU customers will soon be charged their local VAT rate instead of UK VAT).

Here are the details:

iPad and Android
  • The syntax-highlighting in-browser editor that we use, Ace, now supports the iPad and Android devices, so we've upgraded it and changed the mobile version of our site to use it instead of the rather ugly textarea we used to use.
  • We've also re-introduced the "Save and Run" button on the iPad.
  • Combined with the console upgrade we did earlier on this month, our mobile support should now be pretty solid on iPads, iPhones, and new Android (Lollipop) devices. Let us know if you encounter any problems!
User interface
  • Some fixes to our in-browser consoles: fixed problems with zooming in (the bottom line could be cut off if your browser zoom wasn't set to 100%) and with the control key being stuck down if you switched tabs while it was pressed.
  • A tiny change, but one that (we hope) might nudge people in a good direction: we now list Python 3 before Python 2 in our list of links for starting consoles and for starting web apps :-)
New packages

We've added loads of new packages to our "batteries included" list:

  • A quantitative finance library, Quandl (Python 2.7, 3.3 and 3.4)
  • A backtester for financial algorithms, zipline (Python 2.7 only)
  • A Power Spectral Densities estimation package, spectrum (2.7 only)
  • The very cool remix module from Echo Nest: make amazing things from music! (2.7 only)
  • More musical stuff: pyspotify. (2.7 only)
  • Support for the netCDF data format (2.7, 3.3 and 3.4)
  • Image tools: imagemagick and aggdraw (2.7, 3.3 and 3.4)
  • Charting: pychart (2.7 only)
  • For Django devs: django-bootstrap-form (2.7, 3.3 and 3.4)
  • For Flask devs: flask-admin (2.7, 3.3 and 3.4)
  • For web devs who prefer offbeat frameworks: falcon and wheezy.web (2.7, 3.3 and 3.4)
  • A little ORM: peewee (2.7, 3.3 and 3.4)
  • For biologists: simplehmmer (2.7 only)
  • For statistics: Augustus. (2.7 only)
  • For thermodynamicists (?): CoolProp (2.7, 3.3 and 3.4)
  • Read barcodes from images or video: zbar (2.7 only)
  • Sending texts: we've upgraded the Python 2.7 twilio package so that it works from free accounts, and also added it for Python 3.3 and 3.4.
  • Locating people by IP address: pygeoip (2.7, 3.3 and 3.4)
  • We previously had the rpy2 package installed but there was a bug that stopped you from importing rpy2.robjects. That's fixed.

Additionally, for people who like alternative shells to the ubiquitous bash, we've added fish.

Reliability improvement

We've upgraded one of our underlying infrastructural databases to SSD storage. We've had a couple of outages recently caused by problems with this database, which were made much worse by the fact that it took a long time to start up after a failover. Moving it to SSD moved it to new hardware (which we think will make it less likely to fail) and will also mean that if it does fail, it should recover much faster.

EU VAT changes

For customers outside the EU, this won't change anything. But for non-business customers inside the EU, starting 1 January 2015, we'll be charging you VAT at the rate for your country, instead of using the UK VAT rate of 20%. This is the result of some (we think rather badly-thought-through) new EU legislation. We'll write an extended post about this sometime soon.

Categories: FLOSS Project Planets

Enrico Zini: upgrade-encrypted-cyanogenmod

Planet Debian - Fri, 2014-12-19 04:21
Upgrade Cyanogenmod with an encrypted phone

Cyanogenmod found an update, it downloaded it, then it rebooted to install it and nothing happened. It turns out that the update procedure cannot work if the zip file to install is in encrypted media, so a workaround is to move the zip into unencrypted external storage.

As far as I know, my Nexus 4 has no unencrypted external storage.

This is how I managed to upgrade it, I write it here so I can find it next time:

  1. enable USB debugging
  2. adb pull /cmupdater/cm-11-20141115-SNAPSHOT-M12-mako.zip
  3. adb reboot recovery
  4. choose "install zip from sideload"
  5. adb sideload cm-11-20141115-SNAPSHOT-M12-mako.zip
Categories: FLOSS Project Planets

kdepim 4.14 + unittest

Planet KDE - Fri, 2014-12-19 03:52

As you saw I continue to work on 4.14 branch, in parallel I work on KF5 porting.

Why do I continue to work on 4.14 ?

Because KF5 kdepim version will not arrive before that AkonadiNext will be stable, so it will take time. AkonadiNext is writing from scratch.

I don’t know if you remember but during porting to akonadi, kdepim 4.4 was not improving during 2 years, no bug fixing, nothing. It was a very bad idea.

So I decided to continue to fix 4.14, until we release KF5.

What do I do in 4.14 ?

I continue to fix bug. And there is still a lot of bugs

I try to make it stable.

During this period I try to write a lot of unit-test on this branch.

Why do I write unit test ?

Because there is not unit test on kdepim It’s a good reason.

But there is a second reason, as I port kdepim to KF5 and it’s not stable to use it, I can’t know if all works fine. So I create unit test, I verify that it works on 4.14 and after that I just need to run unit test on KF5 to be sure that it works fine on KF5 too.

What do I do when I add a new feature in KF5 ?

Now each new feature that I add, or feature will add in kdepim will have a unit test.

So in KMail or in each lib/apps that I maintain in kdepim, I will not accept new feature if there is not unit-test, because it’s very useful and I don’t want to write them after

It’s a pain when we need to write unit test on a code that we didn’t write.

And sometime code is not adapted to create unit test so it’s necessary to rewrite it. (I rewrote some code in 4.14 to allow to create some unit test).

It’s more easy to create unit test when we write feature that write them after.

Categories: FLOSS Project Planets

Music Production and Experimentation – Part 3

LinuxPlanet - Fri, 2014-12-19 02:29
A follow on from:

Have created 'Classical', 'Foreign Hip Hop and R&B', and 'Soundtrack' playlists on my YouTube profile. Not much there at the moment. I'll add more as time goes on.

I've been looking at doing a music course of some sort for a while now (short course or even a degree). Fees can range from several hundred to several thousand dollars.

There may be some government help but you must fit specific criteria.

There are, of course, some online options which will also provide certification of skills if you aren't keen on spending too much time on campus and/or don't have the time/dedication to go the other way. In most cases, you'll have to pass an audition of some sort though which involves a demonstration of proficiency, a portfolio, as well as possibly an academic pedigree (high school or private tuition).

There will be some websites which will often place there reference materials behind walls of some sort but with intelligent searching you can often find a way around these limitations without having to register/signup for further marketing material.


Some material on programming synthesisers.

A place where you can purchase parts to experiment with .

There are a lot of tablet based music making applications now .

Sometimes you don't have a vocalist nearby. An option is to try computerised vocals.

Sometimes, I have difficulties with getting the type of sound that I want and/or need. Here are some itneresting manuals.

Having being having some frustrations with sound libraries being built with later versions of Kontakt/Reaktor. Has been frustrating me to the point where I thought is there a way to bypass the checks (easily possible with many simple system checks. I only investigated as I'm on a mobile prepaid connection at the moment which means that I am trying limit my downloads.).

Some interesting tips with regards to 'House Music'.

Setup a new Tumblr account. Basically, a mirror of my Twitter account.
Categories: FLOSS Project Planets

Password hygiene – every man’s responsibility

LinuxPlanet - Fri, 2014-12-19 02:18

Everybody knows what hand hygiene is: hands are disinfected every time we enter a hospital. In the same way as germ-free hands stops diseases from spreading, password hygiene helps to prevent the misuse of credentials. Password hygiene may feel useless when the effect is not immediately visible, but it is never the less very important in the age of information society.

Password hygiene is no harder that normal hygiene. Everybody can learn it with the following simple principles:

1. Store your passwords securely

A trick to avoid storing passwords in plain-text on paper

The best would of course be to store all passwords inside one’s head, but since remembering all credentials by heart isn’t simply possible, another good option is keeping the in your wallet. People keep money in their wallets, so most of us are used to store it securely.

Passwords can be “encrypted” on paper with a simple trick: use a common first part in all passwords, for example ‘Ma55i’, which you remember by heart. Then write the last part of the password on paper, for example ‘hU8kkP’. Only you will know that the complete password is then ‘Ma55ihU8kkkP’.

Remember also to make backups of it for example by xeroxing the paper slip. Use the money analogue to choose a good place to store the backup: put the important paper in a safe.

Using paper has some drawbacks, like the need to type passwords all the time. Using a piece of software on the computer will make things easier, because then it is possible to simply copy-paste the usernames and passwords from the program. The recommended program is KeePass. It is free and open source software, its internal functioning is transparent and it is unlikely to have backdoors or other weaknesses, and the same program is available for multiple platforms, including Windows, Linux, OS X, Jolla, Android.

2. Use different passwords in different systems

It is dangerous to use the same password in multiple accounts. For example if your Twitter-password leaks and somebody can log in to Twitter using your account, they will quickly find out what your e-mail address is, and then for sure try to log into your e-mail account using the same password you had on Twitter. Can you afford to loose control of multiple accounts at the same time?

Here too, programs like KeePass help, because they facilitate the automatic generation of new passwords for each new account.

3. Use passwords which are difficult to guess and complex

KeePass password strength meter

Password crackers always start out by trying to guess typical passwords, like horse99 or 123hound. Don’t have a weak password that is likely to be cracked by brute-force password cracking tools. A good password is complex and consists of at least 9 characters, and includes both capitals and small letters, numbers and special characters like +-.,_:;.

Again, KeePass helps with this too, because it has a built-in password strength meter, which will tell you it the password is strong enough, or it can even generate the passwords automatically for you.

4. Change your passwords regularly – or immediately if you think it has leaked!

If the password is strong enough, then a suitable interval to change passwords is a few years. If a password is too weak, then it does not matter much if you change if often or not. Rule number 3 is therefore more important.

In KeePass the dates when a new password was saved is automatically stored, so it can also help you see how old your passwords are.

If somebody suspect somebody peaked over your shoulder and spied on your password, you need to change it immediately. Well designed computer systems show you on login when your previous login was. For example if you return to work after a holiday, and when you log in the system tells your last visit was a few days ago – while you were still on holiday – you will notice that something is wrong and know that the password needs to be changed.

5. Never tell you password to anybody

Never ever tell your password to anybody. No administrator on any system needs to know your password – administrators can anyway always reset your account and get the password that way. If somebody asks for your passwords, it is almost for sure some kind of fraud.

The most common reason for password leaks is that users have themselves told the password to somebody who had a credible enough sounding reason to ask for it!

6. Send you passwords to the correct system and only using a secure connection

When you are about to sign in to a system, try to make sure that you are really connected to the correct system and that the connection is securely protected. A common way to steal passwords is to do a so called man-in-the-middle attack, where the user is tricked in a way or another to enter their credentials to a false system, yet the connection is passed on to the real system so the user does not notice anything unusual. For websites make sure the URL in the address bar is correct before you sign in and make sure it contains the s in the https part so that the connection is secure.

This is however not a guarantee – sophisticated attacks can make the remote system appear perfectly normal with correct address and everything, because the attack might target the underlying network infrastructure. But none the less, every user must make sure they do their own part in keeping the systems secure, and what happens after that is up to the system engineers to take care of.

Original presentation in Finnish

..is available on Slideshare:

Categories: FLOSS Project Planets

Ga&#235;l Varoquaux: PRNI 2016: call for organization

Planet Python - Thu, 2014-12-18 20:38

PRNI (Pattern Recognition for NeuroImaging) is an IEEE conference about applying pattern recognition and machine learning to brain imaging. It is a mid-sized conference (about 150 attendee), and is a satellite of OHBM (the annual “Human Brain Mapping” meeting).

The steering committee is calling for bids to organize the conference in June 2016, in Europe, as a satellite the OHBM meeting in Geneva.

Categories: FLOSS Project Planets

Invent with Python: Cover for &quot;Automate the Boring Stuff with Python&quot; is Released

Planet Python - Thu, 2014-12-18 18:59

The final cover design for my next book, Automate the Boring Stuff with Python, is available on the No Starch Press website.

I'm super excited. :D

Categories: FLOSS Project Planets

Justin Mason: Links for 2014-12-18

Planet Apache - Thu, 2014-12-18 18:58
Categories: FLOSS Project Planets

Omaha Python Users Group: December Meeting Notes

Planet Python - Thu, 2014-12-18 18:51

We had a great meeting which included a raffle of the awesome Python book, Learning Python 4th Edition.  Thanks to O’Reilly Publishing.

Further topics included, in no particular order:

  • Tox – a generic virtualenv management and test command line tool
  • Bitbucket – for private remote repositories
  • Virtualenvwrapper - a set of extensions to Ian Bicking’s virtualenv tool.
  • Django Data Migrations in 1.7
  • Moving from Django 1.2 to 1.7.
  • python-social-auth – an easy to setup social authentication/registration mechanism with support for several frameworks and auth providers.
  • yamjam – a multi-project, shared, yaml based configuration system.
  • dhp – A collection of handy modules, almost guaranteed to get you into trouble.
  • dennis python package – a set of utilities for working with PO files to ease development and improve quality.

Merry Christmas to all!

Categories: FLOSS Project Planets

Wouter Verhelst: Introducing libjoy

Planet Debian - Thu, 2014-12-18 18:29

I've owned a Logitech Wingman Gamepad Extreme since pretty much forever, and although it's been battered over the years, it's still mostly functional. As a gamepad, it has 10 buttons. What's special about it, though, is that the device also has a mode in which a gravity sensor kicks in and produces two extra axes, allowing me to pretend I'm really talking to a joystick. It looks a bit weird though, since you end up playing your games by wobbling the gamepad around a bit.

About 10 years ago, I first learned how to write GObjects by writing a GObject-based joystick API. Unfortunately, I lost the code at some point due to an overzealous rm -rf call. I had planned to rewrite it, but that never really happened.

About a year back, I needed to write a user interface for a customer where a joystick would be a major part of the interaction. The code there was written in Qt, so I write an event-based joystick API in Qt. As it happened, I also noticed that jstest would output names for the actual buttons and axes; I had never noticed this, because due to my 10 buttons and 4 axes, which by default produce a lot of output, the jstest program would just scroll the names off my screen whenever I plugged it in. But the names are there, and it's not too difficult.

Refreshing my memory on the joystick API made me remember how much fun it is, and I wrote the beginnings of what I (at the time) called "libgjs", for "Gobject JoyStick". I didn't really finish it though, until today. I did notice in the mean time that someone else released GObject bindings for javascript and also called that gjs, so in the interest of avoiding confusion I decided to rename my library to libjoy. Not only will this allow me all kinds of interesting puns like "today I am releasing more joy", it also makes for a more compact API (compare joy_stick_open() against gjs_joystick_open()).

The library also comes with a libjoy-gtk that creates a GtkListStore* which is automatically updated as joysticks are added and removed to the system; and a joytest program, a graphical joystick test program which also serves as an example of how to use the API.

still TODO:

  • Clean up the API a bit. There's a bit too much use of GError in there.
  • Improve the UI. I suck at interface design. Patches are welcome.
  • Differentiate between JS_EVENT_INIT kernel-level events, and normal events.
  • Improve the documentation to the extent that gtk-doc (and, thus, GObject-Introspection) will work.

What's there is functional, though.

Update: if you're going to talk about code, it's usually a good idea to link to said code. Thanks, Emanuele, for pointing that out

Categories: FLOSS Project Planets

Graham Dumpleton: Apache/mod_wsgi for Windows on life support.

Planet Python - Thu, 2014-12-18 17:41
The critics will say that Apache/mod_wsgi as a whole is on life support and not worth saving. I have a history of creating Open Source projects that no one much wants to use, or get bypassed over time, but I am again having lots of fun with Apache/mod_wsgi so I don't particularly care what they may have to say right now. I do have to say though, that right now it looks like continued support for
Categories: FLOSS Project Planets

Kubuntu Vivid Alpha 1

Planet KDE - Thu, 2014-12-18 17:18
"How much wood could a woodchuck chuck if a woodchuck could chuck wood?" - Guybrush Threepwood, Monkey Island

The first Alpha of Vivid (to become 15.04) has now been released!

The Alpha-1 images can be downloaded at: http://cdimage.ubuntu.com/kubuntu/releases/vivid/alpha-1/

More information on Kubuntu Alpha-1 can be found here: https://wiki.ubuntu.com/VividVervet/Alpha1/Kubuntu

Categories: FLOSS Project Planets

Gregor Herrmann: GDAC 2014/18

Planet Debian - Thu, 2014-12-18 17:01

what constantly fascinates me in debian is that people sit at home, have an idea, work on it, & then suddenly present it to an unexpecting public; all without prior announcements or discussions, & totally apart from any hot discussion-de-jour. the last example I encountered & tried out just now is the option to edit source packages online & submit patches. - I hope we as a project can keep up with this creativity!

this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.

Categories: FLOSS Project Planets
Syndicate content