FLOSS Project Planets

Nick Coghlan: DTCA Public Consultation - Brisbane

Planet Python - Tue, 2015-01-20 07:41

Over the weekend, Asher Wolf alerted me (and many others in the open source and cryptographic communities) to the Australian Defence Trade Controls Act 2012, and the current public consultation taking place around a bill proposing amendments to that act.

Being heavily involved in improving the security of open source infrastructure like the Python Package Index and the Python 2 reference interpreter, working at a multinational open source vendor, and having an extensive background in working under the constraints of the US International Traffic in Arms regulations, Asher's concern caught my attention, since bad legislation in this area can have significant chilling effects on legitimate research and development activities.

As a result, I've escalated this legislation for review by the legal teams at various open source companies and organisations, with a view to making formal submissions to the public consultation process that is open until January 30th (ready for bills to be submitted for consideration to federal parliament on February 23rd).

However, I was also able to attend the first public consultation session held at the University of Queensland on January 19, so these are my impressions based primarily on that sessions and my own experiences dealing with ITAR. I'm not a lawyer and I haven't actually read the legislation, so I'm not going to pick up on any drafting errors, but I can at least speak to the intent of the folks involved in moving this process forward.

What not to worry about

To folks encountering this kind of legislation for the first time, the sheer scope of the Defence and Strategic Goods List can seem absolutely terrifying. This was very clear to me from some of the questions various academics in the room were asking.

On this particular point, I can only say: "Don't panic". This isn't a unique-to-Australia list, it's backed by a treaty called the Wassenaar Arrangement - the DSGL represents part of the implementation of that arrangement into Australian law.

When the laws implementing that arrangement are well drafted, everyone outside the military industrial complex (and certain easily weaponised areas of scientific research) can pretty much ignore them, while everyone inside the military industrial complex (and the affected areas of research) pays very close attention to them because we like not being in jail (and because gunrunning is bad, and bioterrorism is worse, mmm'kay?).

A heavily regulated military supply chain is already scary enough, we really don't want to see the likely consequences of an unregulated one. (And if you're tempted to make a snarky comment about the latter already being the case, no, it really isn't. While folks can sometimes use overclassification to avoid regulations they're supposed to be following, that still introduces significant friction and inefficiencies into whatever they're doing. It's not as good as people actually respecting the laws of the countries they're supposedly defending, including genuinely meeting the requirement for civilian authority over the military, but it's still a hell of a lot better than nothing).

Getting back on topic, the US ITAR and crypto export control laws are currently considered the most strict implementation of the Wassenaar Arrangement amongst the participating nations (going beyond the requirements of the treaty in several areas), so if you see plenty of US nationals participating in an activity without being fined and going to jail, you can be fairly confident that it isn't actually a controlled activity under the DSGL (or, even if it is, permits for that specific activity will be fairly easy to get, and the most likely consequence of not realising you need a permit for something you're doing will be someone from your government getting in touch to point out that you should apply for one).

There are certainly some very questionable aspects of this list (with the perennial "favourite" being the fact the Wassenaar Arrangement does, in fact, attempt to regulate the global trade in mathematics, which is just as stupid and problematic as it sounds), but it's a known quantity, and one we're pretty sure we can continue to live with (at least for the time being).

What to worry about

The real problem here is that the regulations included in the 2012 Act are not well drafted, and the legislated 2 year transition period from May 2013 through to May 2015 prior to the enforcement provisions kicking in is about to run out.

The biggest problem with the 2012 act is that in trying to keep things simple (essentially, "if its on the DSGL, you need a permit"), it ended up becoming extraordinarily draconian, requiring a permit for things that don't require an export license even under ITAR.

For the general public, the most significant shift in the 2015 amendment bill is the fact that several cases around open publication of information related to dual-use technologies shift to being allowed by default, and only in exceptional cases would a permit be required (and in those cases, the onus would be on the government to inform the covered individuals of that requirement).

The amendments also include a variety of additional exemptions for little things like making it legal for Australian's own police and security agencies to collaborate with their international counterparts. (Snarky comment opportunity #2: in certain areas, making such collaboration illegal seems like a potentially attractive idea...)

That 2 year pilot was included in the original legislation as a safety mechanism, the feedback from the associated steering group has been extensive, and if things had gone according to plan, the relevant amendments to the bill would have been passed last year in the spring sitting of federal parliament, leaving DECO with at least 6 months to educate affected organisations and individuals, and start issuing the now necessary permits before the enforcement provisions became active in May. Unfortunately, we currently have a federal government that views pushing a particular ideological agenda as being more important than actually doing their job, so we're now faced with the prospect of regulations that industry doesn't want, academia doesn't want, the Australian public service don't want, and the Australian military don't want, coming into effect anyway.

Isn't politics fun?

What DECO are (trying) to do about it

The group tasked with untangling this particular legislative Charlie Foxtrot is the Australian Defence Export Control Office (DECO). Their proposal for addressing the situation hinges on two bills that they plan to put before the next sitting of federal parliament:

  • an amendment bill for the Act itself, which fixes it to be a conventional implementation of the Wassenaar Arrangement, in line with existing implementations in other Wassenaar nations (why we didn't just do that in the first place is beyond me, but at least DECO are trying to fix the mistake now)
  • a second bill to delay the enactment of the enforcement provisions for a further six months to provide sufficient time for DECO to properly educate affected parties and start issuing permits

As far as I am aware, the second bill is needed primarily due to the consideration of the first bill slipping by six months, since we're now looking at the prospect of only having 4 weeks for DECO to start issuing permits before the enforcement provisions come into effect. Nobody involved thinks that's a good idea.

If both of those bills pass promptly, then the only cause for concern is whether or not there are any remaining devils in the details of the legislation itself. Member of the general public aren't going to be able to pick those up - despite the surface similarities, legalese isn't English, and reading it without interpreting it in the context of relevant case law can be a good way to get yourself into trouble. Summary translations from legalese to English by a competent lawyer are a much safer bet, although still not perfect. (For the programmers reading this: I personally find it useful to think of legalese as source code that runs on the language interpreter of a given nation's legal system, while the English translations are the code comments and documentation that anyone should be able to read if they understand the general concepts involved).

If at least the second bill passes, then we have another 6 months to work on a better resolution to the problem.

If neither bill passes, then DECO end up in a bad situation where they'll be required by law to implement and enforce regulations that they're convinced are a bad idea. They actually have everything in place to do that if they have to, but they don't want this outcome, and neither does anyone else.

What industry and academia can do about it

While it's very short notice, the main thing industry and academia can do is to file formal submissions with DECO as described in their overview of the public consultation process.

There are three main things to be addressed on that front:

  • ensuring federal parliament are aware of the importance of amending the Defence Trade Controls Act 2012 to eliminate the more draconian provisions
  • ensuring federal parliament are aware of the infeasibility of putting this into effect on the original timeline and the need for a significant delay in the introduction of the enforcement provisions
  • ensuring DECO are alerted to any remaining areas of concern in the specific drafting of the amended legislation (although I'd advise skipping this one if you're not a lawyer yourself - it's the functional equivalent of a lawyer with no training as a programmer proposing patches to the Linux kernel)

We were apparently asleep at the wheel when DTCA went through in 2012, so we owe a lot of thanks to whoever it was that advocated for and achieved the inclusion of the two year transition and consultation period in the original bill. Now we need to help ensure that our currently somewhat dysfunctional federal parliament doesn't keep us from receiving the benefit of that foresight.

What's definitely not going to happen

This consultation process is not the place to rail against the details of the Wassenaar Arrangement or Australia's participation in it. You won't achieve anything except to waste the time of folks that currently have a really serious problem to fix, and a very limited window in which to fix it.

Yes, Wassenaar has some serious problems, especially around its handling of cryptography and cryptographic research, but we have a fairly settled approach to handling that at this point in history. The critical concern in this current case is to help DECO ensure that the associated Australian regulations can be readily handled through the mechanisms that have already been put in place to handle existing Wassenaar enforcement regimes in other countries. With the way the 2012 Act was drafted, that's almost certainly currently not the case, but the proposed 2015 amendments should fix it (assuming the amendments actually have the effects that DECO has indicated they're intended to).

Categories: FLOSS Project Planets

FSF Events: Richard Stallman - "A Free Digital Society" (Sidi Bel Abbes, Algeria)

GNU Planet! - Tue, 2015-01-20 07:34
There are many threats to freedom in the digital society. They include massive surveillance, censorship, digital handcuffs, nonfree software that controls users, and the War on Sharing. Other threats come from use of web services. Finally, we have no positive right to do anything in the Internet; every activity is precarious, and can continue only as long as companies are willing to cooperate with it.

Richard Stallman's speech will be nontechnical, admission is free of charge, and the public is encouraged to attend.

Exact location to be determined.

Please fill out our contact form, so that we can contact you about future events in and around Siddi Bel Abbès.

Categories: FLOSS Project Planets

FSF Events: Richard Stallman - "The Free Software Movement" (Algiers, Algeria)

GNU Planet! - Tue, 2015-01-20 05:58
Richard Stallman will speak about the goals and philosophy of the Free Software Movement, and the status and history of the GNU operating system, which in combination with the kernel Linux is now used by tens of millions of users world-wide.

Richard Stallman's speech will be nontechnical, admission is gratis, and the public is encouraged to attend.

Please fill out our contact form, so that we can contact you about future events in and around Algiers.

Categories: FLOSS Project Planets

FSF Events: Richard Stallman - "Free Software and Your Freedom" (Djelfa, Algeria)

GNU Planet! - Tue, 2015-01-20 05:52
The Free Software Movement campaigns for computer users' freedom to cooperate and control their own computing. The Free Software Movement developed the GNU operating system, typically used together with the kernel Linux, specifically to make these freedoms possible.

Richard Stallman's speech will be nontechnical, admission is gratis, and the public is encouraged to attend.

Please fill out our contact form, so that we can contact you about future events in and around Djelfa.

Categories: FLOSS Project Planets

Claus Ibsen: Webinar - Integrating microservices with Apache Camel

Planet Apache - Tue, 2015-01-20 02:28
Tomorrow on wednesday 21st of january, our fellow Camel rider, Christian Posta is giving a webinar about microservices and Apache Camel.
Microservices architectures are all the rage these days, but we have to balance hype with reality. Microservices make it harder to manage your deployments and makes for complex inter-service communications patterns. How do you balance the tradeoffs and focus on getting the most out of your investment in highly scalable, decoupled systems?
Join this webinar to learn how to use patterns from SOA to build out intelligent routing systems with Apache Camel, and centralized management, service discovery, versioning, and tooling support from Red Hat JBoss Fuse for managing complex integrations using a microservice approach.Date: Wednesday, January 21, 2015Time: 16:00 UTC | 11:00 am (New York) | 5:00 pm (Paris) | 9:30 pm (Mumbai)Duration: 60 minutes
You can find more details and register for the webinar using the following link.
Categories: FLOSS Project Planets

Raphael Geissert: Edit Debian, with iceweasel

Planet Debian - Tue, 2015-01-20 02:00
Soon after publishing the chromium/chrome extension that allows you to edit Debian online, Moez Bouhlel sent a pull request to the extension's git repository: all the changes needed to make a firefox extension!

After another session of browser extensions discovery, I merged the commits and generated the xpi. So now you can go download the Debian online editing firefox extension and hack the world, the Debian world.

Install it and start contributing to Debian from your browser. There's no excuse now.

Categories: FLOSS Project Planets

Amit Saha: LCA 2015 talk: Beaker’s Hardware Inventory system

Planet Python - Tue, 2015-01-20 01:13

The video is up on YouTube: http://t.co/WorOwbv37w

Slides: https://amitksaha.fedorapeople.org/lca2015/slides.html

Since I could not make it to LCA, Nick Coghlan presented the talk on my behalf. Thanks Nick!

Categories: FLOSS Project Planets

Bryan Pendleton: Stuff I'm reading, mid-January edition

Planet Apache - Mon, 2015-01-19 22:51

Everybody came back from their holiday break with new things for me to read.

  • One year into the Ebola epidemic: a deadly, tenacious and unforgiving virusOne year after the first Ebola cases started to surface in Guinea, WHO is publishing this series of 14 papers that take an in-depth look at West Africa’s first epidemic of Ebola virus disease.
  • HC26 Full ProgramHot Chips: A Symposium on High Performance Chips; Sponsored by the IEEE Technical Committee on Microprocessors and Microcomputers in Cooperation with ACM SIGARCH
  • Why does a single integer assignment statement consume all of my CPU?Many processor architectures are less forgiving of misaligned data access and raise an alignment exception if you break the rules. When such an exception occurs, the operating system might choose to terminate the application. Or the operating system may choose to emulate the instruction and fix up the misaligned access. The program runs much slower, but at least it still runs. (In Windows, the decision how to respond to the alignment exception depends on whether the process asked for alignment faults to be forgiven. See SEM_NO­ALIGNMENT­FAULT­EXCEPT.)

    It appears that the original program is in the last case: An alignment exception occurred, and the operating system handled it by manually reading the four bytes from m_data[0] through m_data[4] and assembling them into a 32-bit value, then resuming execution of the original program.

    Dispatching the exception, parsing out the faulting instruction, emulating it, then resuming execution. That is all very slow. Probably several thousand instruction cycles. This can easily dwarf the actual computation performed by Calculate­The­Value.

  • GoGo does not need to run "Man in the Middle Attacks" on YouTube I use GoGo a lot. I’ve discovered that their system architecture suffers from “bufferbloat” (the same problem that caused Comcast to deploy Sandvine DPI gear to discover and attack bittorrent with “forged TCP” packet attacks, and jump-started the political net neutrality movement by outraging the Internet user community). Why does that matter? Well, if GoGo eliminated bufferbloat, streaming to the airplane would not break others’ connections, but would not work at all, with no effort on Gogo’s part other than fixing the bufferbloat problem. [The reason is simple - solutions to bufferbloat eliminate excess queueing delay in the network, thereby creating "fair" distribution of capacity among flows. That means that email and web surfing would get a larger share than streaming or big FTP's, and would not be disrupted by user attempts to stream YouTube or Netflix. At the same time, YouTube and Netflix connections would get their fair share, which is *not enough* to sustain video rates - though lower-quality video might be acceptable, if those services would recode their video to low-bitrate for this limited rate access].
  • Tintri: We have ZERO interest in adding compute to storageCurrently a VMware-focused storage supplier using VMware storage abstractions, such as virtual machines (VMs) and virtual disks instead of LUNs and RAID groups, Tintri is expanding out its virtualised server remit to support KVM comparatively soon and Hyper-V around the end of the year.
  • Five recent results in high-performance data paths.The sum of these two facts is that we are currently in a period of systems design in which I/O performance is in its ascent: it is becoming proportionally faster relative to computation. This environmental change is demanding that systems researchers and designers reconsider the parameters in how they architect systems. As evidence of this trend, here are five spectacularly interesting papers that have been published at top systems and networking conferences over the past 12 months.
  • The little book about OS developmentThis text is a practical guide to writing your own x86 operating system. It is designed to give enough help with the technical details while at the same time not reveal too much with samples and code excerpts. We’ve tried to collect parts of the vast (and often excellent) expanse of material and tutorials available, on the web and otherwise, and add our own insights into the problems we encountered and struggled with.
  • Why Docker, Containers and systemd Drive a Wedge Through the Concept of Linux DistributionsThere’s been an unholy war raging through the Linux world over systemd for some time. Pretty much everything on a system gets touched by what is selected as the first process on a system and how that impacts everything getting started up. People care a lot about this stuff, and the arguments have been passionate. Nevertheless, Mark Shuttleworth conceding defeat on behalf of Ubuntu marked the last major distribution going all in on systemd. Unless forks like Devuan become successful it’s going to be pretty hard to get Linux in a couple of years time without getting systemd as part of it.
  • OVN, Bringing Native Virtual Networking to OVSOVN is a new project from the Open vSwitch team to support virtual network abstraction. OVN will put users in control over cloud network resources, by allowing users to connect groups of VMs or containers into private L2 and L3 networks, quickly, programmatically, and without the need to provision VLANs or other physical network resources. OVN will include logical switches and routers, security groups, and L2/L3/L4 ACLs, implemented on top of a tunnel-based (VXLAN, NVGRE, Geneve, STT, IPsec) overlay network.
  • PCC: Performance-oriented Congestion ControlPerformance-oriented Congestion Control (PCC) is a new architecture that achieves consistent high performance even under challenging conditions. PCC senders continuously observe the connection between their actions and empirically experienced performance, enabling them to consistently adopt actions that result in high performance.
  • What Doesn't Seem Like Work?If something that seems like work to other people doesn't seem like work to you, that's something you're well suited for. For example, a lot of programmers I know, including me, actually like debugging. It's not something people tend to volunteer; one likes it the way one likes popping zits. But you may have to like debugging to like programming, considering the degree to which programming consists of it.
  • A Non-Programmer's Introduction to GitThis is all well and good, but how can I (as a non-programmer) use a tool like Git? Here are a couple examples.
  • Do elite software developers exist?We should consider the possibility that someone could have 10X talent (whatever that means) without necessarily generating 10X output volume. Maybe the 10X shows up in quality instead of quantity. Maybe this is why elite developers are not paid 10X the average developer.
  • University Of Chicago's New Free Speech Policy Actually Protects Free SpeechRather than actually deal with speech issues on a case-by-case basis, universities have instead enacted broadly-written bans on campus speech.

    The University of Chicago, however, isn't jumping on this particular bandwagon. Its new speech policy is more of manifesto than a policy. It's assertive and it's comprehensive -- not in its restrictions, but in its liberties. It's the outgrowth of a study performed by the school and the conclusions it reaches are decidedly contrary to the prevailing collegiate winds.

    The committee behind the report and policy is chaired by Geoffrey Stone, a professor specializing in constitutional law (and member of the administration's intelligence review task force). Stone is a fierce defender of civil liberties, previously having taken Arizona legislators to task for their First Amendment-steamrolling cyberbullying/harassment bill.

  • Linus Torvalds on why he isn’t nice: “I don’t care about you”Following his keynote speech at the Linux.conf.au Conference in Auckland, New Zealand, Torvalds opened a Q&A session by fielding a question from Nebula One developer Matthew Garrett that accused Torvalds of having an abrasive tone in the Linux kernel mailing list. "Some people think I'm nice and are shocked when they find out different," Torvalds said in response (quoted via multiple Twitter accounts of the event). "I'm not a nice person, and I don't care about you. I care about the technology and the kernel—that's what's important to me."
  • But Where Do People Work in This Office?I see a lot of awesome stuff, but where is the quiet area where your big brains go to make world-changing software? Oh, jeez.

It looks like the entire month of January may pass with no rain at all. December's rain is just a distant memory now.

It does mean that I get to continue riding my bike.

But I would rather have the rain.

Categories: FLOSS Project Planets

Bryan Pendleton: The Name of the Wind: a very short review

Planet Apache - Mon, 2015-01-19 22:08

At some point late last fall, I was anticipating having some time to spend with my Kindle, so I bought Patrick Rothfuss's The Name of the Wind

As usual with me, I am about 10 years behind the times, as this book came out some time ago.

But I was looking for a page-turner (is it fair to say that, when you are reading an e-book on an e-reader?)

At any rate, I turned the pages.

And kept turning them (there are a lot of pages...).

And I turned them all the way until the end.

Which is not always the case with me, and a book. I have too little time and too many distractions, and many is the book that I nobly start yet do not finish.

Rothfuss's style appealed to me, because he knows how to take his time with his story. Sometimes books rush along, hurrying to force the tale to be told, cramming adventures and villains and escapades willy-nilly into every page.

But Rothfuss is trying to tell the story of a boy growing up (even though that boy may become a mighty wizard).

And, as every boy knows (and surely, every girl as well), growing up takes its own time, and proceeds on its own schedule.

So, the long and short of it is: I enjoyed The Name of the Wind, and felt it lived up to my expectations.

Rothfuss has written a sequel, and promises that he will complete his story.


When the time comes.

And, down the road, when I find that I again have some time with my Kindle, I expect that I will continue reading Rothfuss, moving on to The Wise Man's Fear.

And see how I like turning those pages.

Categories: FLOSS Project Planets

Nick Kew: Dodgy Data

Planet Apache - Mon, 2015-01-19 20:11

Oxfam grabs a headline with a report telling us the richest 1% will own half the world’s wealth in 2016.

As with many reports coming from lobbying organisations, this one provokes scepticism.  Not outright dismissal, but a “really“, and a need to know what they’re actually measuring before I can treat it as meaningful.  It also provokes mild curiosity: how rich do you have to be to be in that 1% (not least because I have a sneaking suspicion it includes a great many people who our chattering classes don’t consider at all rich).

The Oxfam report itself is a mere twelve pages and disappointingly light on data.  If there’s any attempt to substantiate the headline claim then I missed it.  But googling “World Wealth” finds this report, which tells me total world wealth is projected to be $64.3 trillion in 2016.  OK, that’ll do for a ballpark calculation.  $64.3 trillion between 7 billion people is an average of about $9k per head.  If the top 1% own half of it, that’s $32.15 trillion between 70 million people: an average of $459k per head within that top 1%.

That’s £300k.  There must be a millions in Blighty with that much in housing wealth alone (and others correspondingly locked out).  Not to mention in other high-cost countries around Europe, America, Asia, and I expect even a few in the third world.  All above the average of that fabled top 1%.

But of course housing isn’t our only asset.  In Blighty and around the developed world, a big chunk of our wealth takes the form of Entitlements.  One such in the UK is the Basic State Pension, which is worth £200k, and even the poorest Brit is entitled to it.  It seems you can be in that top 1% without being rich enough to buy a house in Blighty!

Hmmm.  Oh dear.  Maybe Oxfam’s spin isn’t really very meaningful at all.  Except perhaps to highlight how incredibly egalitarian we are within Blighty – and probably all developed countries – once you include the effect of government actions.

Categories: FLOSS Project Planets

Igor Galić: equality

Planet Apache - Mon, 2015-01-19 19:00

i’ve started reading a book on lisp. as Alan Perlis said:

A language that doesn’t affect the way you think about programming, is not worth knowing.

but this chapter title:

"Truth, Falsehood, and Equality" — sounds like a chapter from legend of korra

— The Wrath of PB™ (@hirojin) January 19, 2015

made me think beyond programming. i’ve been contemplating this in terms of political systems & stories, and i’m thinking that there’s no chance to achieve radical equality:

societies change over generations, as do their their stories. and while, as societies, we frown at those (ancient or contemporary) societies that use murder of prisoners and slaves as entertainment, our stories are filled with such things.

the fight for power.
the struggle against corrupt power.
we even have to fight for love.

we have no need for equality, because the stories we are raised with neither prepare us for what such an equal society would look like, nor do they raise a desire to achieve it.

we are inching ourselves towards it. that’s societal change over generations. i’m starting to fear the only way we know how to radically change is to erase the past, and that would be profoundly dangerous.

even more dangerous than forgetting the (often recent) past, and regress into “good old” patterns.

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-01-19

Planet Apache - Mon, 2015-01-19 18:58
  • carbon-c-relay

    A much better carbon-relay, written in C rather than Python. Linking as we’ve been using it in production for quite a while with no problems.

    The main reason to build a replacement is performance and configurability. Carbon is single threaded, and sending metrics to multiple consistent-hash clusters requires chaining of relays. This project provides a multithreaded relay which can address multiple targets and clusters for each and every metric based on pattern matches.

    (tags: graphite carbon c python ops metrics)

  • Surveillance of social media not way to fight terrorism – Minister

    Blanket surveillance of social media is not the solution to combating terrorism and the rights of the individual to privacy must be protected, Data Protection Minister Dara Murphy said on Monday. [He] said Ireland and the European Union must protect the privacy rights of individuals on social media. “Freedom of expression, freedom of movement, and the protection of privacy are core tenets of the European Union, which must be upheld.”

    (tags: dara-murphy data-protection privacy surveillance europe eu ireland social-media)

Categories: FLOSS Project Planets

VM(doh): Be Careful with Large Select Lists on Drupal Commerce Line Item Type Configuration

Planet Drupal - Mon, 2015-01-19 17:26

Recently, we were debugging some performance issues with a client's Drupal Commerce website. After doing the standard optimizations, we hooked up New Relic so we could see exactly what else could be trimmed.

The site is using different line item types to differentiate between products that should be taxed in different ways. Each line item type has a field where administrators can select the tax code to use for that line item type. The options for the select list are populated via an API call to another service provider. The call for the list was using the static cache because it was thought that the list would only be populated when needed on the line item type configuration page. In reality, that's not the case.

When an Add to Cart form is displayed in Drupal Commerce, it also loads the line item type and the line item type's fields. When loading the fields, it loads all of the options even if the "Include this field on Add to Cart forms for line items of this type" option is not enabled for that field. In this case, it resulted in 90 HTTP calls to populate the list of tax codes every time someone viewed a page with an Add to Cart form.

The solution was to actually cache those results using Drupal's Cache API. You can see the improvement:

Categories: FLOSS Project Planets

Jim Jagielski: Telaen 2.0 Status

Planet Apache - Mon, 2015-01-19 17:19

As noted in a previous blog post, I've started working on the 2.0 version of Telaen: a simple but powerful PHP-based Webmail system. Quite a bit has been changed, fixed and added under-the-covers, including baselining PHP 5.4, a more robust installation checker, and some significant performance increases.

However, as I was working to make the backend stuff as up-to-date as possible, it became increasingly obvious that Telaen's UI was extremely dated. It was functional, yes, but made very limited use of CSS, HTML5, Javascript, etc, all of which combine to affect the user experience. Luckily, a very good friend of mine, Mike Hill, has started work on a new UI for Telaen, making it not only more streamlined and attractive, but also much more functional as well.

Now I know, of course, that there are a number of other PHP webmail offerings out there, so some may be questioning the need for yet another. I can think of a few reasons:

  • Telaen is designed to have as few dependencies as possible; the goal is that any typical PHP setup will be able to run Telaen.
  • No external database is required.
  • Extensive support for both IMAP and POP; to be honest, most webmail systems don't support POP at all, or are extremely limited in their support.
  • Consistent functionality, no matter which IMAP/POP server is used; most webmail systems are simple "front ends" for IMAP servers, meaning the capability of the webmail system depends on what IMAP server is being used. Telaen puts that capability within the webmail system for a consistent feature set.
  • Fast caching
  • Designed to serve as both someone's primary Email client, as well as their supplemental client.
  • Lots of what you need/want, and none of what you don't: Telaen is as simple as it can be, but no more so.
  • A fast and secure upgrade path for all those people still using UebiMaiu
  • Open to ALL contributions!

The last point is important: we really want as many people as possible to use, contribute, drive and develop Telaen. It's a great project for someone just starting out as well as for more experienced developers. Or if your passion is documentation, we could definitely use your help! In fact, however you want to be involved, we want to welcome you to the project.

Our goal is to have a beta available sometime within a month's timeframe. Stay tuned!

Categories: FLOSS Project Planets

Przemysław Kołodziejczyk: Google App Engine - prevent of tasks duplication in the queues

Planet Python - Mon, 2015-01-19 17:10

This is a second part of how we solved issues with huge number of queues on Google App Engine. I recommend to start from first part - Google App Engine - Get the least loaded queue

Let's start again with this nice screenshot ;-)

The first value is a creation time of the oldest task and the second value is a number of the tasks in a single queue. Reason of that? At first, a lot of tasks, really, we run big operations, some of them can be repeated because of some errors, and of course.. a lot of legacy code.

Because first fix described here didn't get us satisfying results we needed to think about another solution. Because we had a lot of tasks and as i investigated that those tasks can be repeated a lot of times i wanted to prevent that situation. Example: user can add/edit/delete/copy some item, _post/_put methods for this item runs N tasks, if user edits this item several times the number of the tasks can grow. Combining this with a large number of users, large number of tasks run by actions made by those users we got huge number of the tasks.

What is the solution? I've used two things from queues:

First, the method.

import collections import datetime import hashlib import math import re from google.appengine.api import taskqueue from google.appengine.ext import deferred def unique_task_defer(obj, *args, **kwargs): def normalize_name(obj_name, name_from_args, minutes=None): date = datetime.datetime.now() if minutes: date += datetime.timedelta(minutes=minutes) name = '{}-{}-{}-{}'.format( obj_name, hashlib.sha1(name_from_args.encode('utf-8')).hexdigest(), date.strftime('%Y%m%d%H'), int(math.ceil(float(int(date.strftime('%M')) + 1) / 15)) ) name = name.encode('utf-8', 'replace') name = name.replace('_', '-') name = re.sub('[^\w-]', '-', name) return name kwargs_values = [ str(value) for key, value in collections.OrderedDict( sorted(kwargs.items()) ).iteritems() if not str(key).startswith('_') ] name_from_args = '-'.join(map(unicode, args + tuple(kwargs_values))) name = normalize_name(obj.__name__, name_from_args) kwargs['_name'] = name try: deferred.defer(obj, *args, **kwargs) except taskqueue.TaskAlreadyExistsError: logging.info( 'Skipping task (already exists): {}, {}'.format( obj.__name__, name ) ) except taskqueue.TombstonedTaskError: name = normalize_name(obj.__name__, name_from_args, minutes=15) logging.info( 'TombstonedTask (retry): {}, {}'.format( obj.__name__, name ) ) kwargs['_name'] = name kwargs['_countdown'] = 900 try: deferred.defer(obj, *args, **kwargs) except taskqueue.TaskAlreadyExistsError: logging.info( 'Skipping task (already exists, second try): {}, {}'.format( obj.__name__, name ) )

Some explanation how it works. Look at this.

unique_task_defer( task, parameter1, parameter2, _queue=get_least_loaded_queue(SOME_QUEUES_LIST), )

This is how we use our method. Method get_least_loaded_queue is described in separate post, you can read about it here. If you want you can just remove _queue parameter or put some queue name.

At first we need to create a unique name for given task taking into account his parameters. It is made by normalize_name method. If the method gets the same task with the same parameters then method returns the same string every time. Queue name is composed from four parts:

  • task name
  • combined string from all parameters
  • date in format %Y%m%d%H
  • number of quarter of an hour, it could have four values:
    • 1 if it is a first quarter, example range: 01:00 - 01:14
    • 2 if it is a second quarter, example range: 01:15 - 01:29
    • 3 if it is a third quarter, example range: 01:30 - 01:45
    • 4 if it is a fourth quarter, example range: 01:46 - 01:59

If normalized_name will get minutes argument then those number of minutes will be added to third part of queue name - date. I will explain it later.

Let's run this task. We are using unique_task_defer method with some parameters. This method creates a unique queue name as described above and push it into some queue. This operation could raise two exceptions which we are interested in.

  • TaskAlreadyExistsError - it will be raised when task with given name will exist in a given queue
  • TombstonedTaskError - it will be raised when task with given name was executed in a given queue, it doesn't exist because it was finished

First, our method sets task name and wants to push task to the queue using deferred.defer method. If the task has never existed then it is pushed normally to the queue. Without magic.

If deferred.defer method raises TaskAlreadyExistsError then we don't want to run this task again. It exists in the queue, it is not finished and every new task will do the same operation so it is unnecessary.

If deferred.defer method raises TombstonedTaskError then it means that someone ran this task some time ago with the same parameters. We want to run this task once again because something could change but in this time we will delay time execution for 900 seconds (15 minutes) . Why? Answer is simple, to prevent of huge number of tasks in queue which are executed and to get better task management. We are using this method in cases when some delay is allowed.

Simple example.

  1. We run the task first time. It runs without problems and it is added to the queue.
  2. We run this task once again. It exists in the queue so deffered.defer raises TaskAlreadyExistsError and the task is killed and is not added to the queue once again.
  3. Our task is finished.
  4. We run this task once again. deffered.defer method raises TombstonedTaskError. We want to run this task again but with 15 minutes’ delay. Our method changes task name by changing fourth parameter in queue name - number of quarter of an hour. It gives us possibility to push task to the queue once again because it has different name. Our method also sets countdown parameter to the task which means that this task will wait in queue minimum 15 minutes and then it will be executed. In this time other tasks can be finished.
  5. When deffered.defer raises TombstonedTaskError we are also checking once again if this task exists in the queue. It is the same operation as in point 2.

Maybe it doesn't look like a great solution but it really helped us to resolve all issues with queues quickly. At the moment the oldest task in the queue has maximum 15 - 20 minutes but most of the time our queues look like this which is is the best proof that it works. ;-)

This was achieved by two steps:
1. Described in the post: Google App Engine - Get least loaded queue
2. Described in this post.

Categories: FLOSS Project Planets

Przemysław Kołodziejczyk: Google App Engine - Get the least loaded queue

Planet Python - Mon, 2015-01-19 16:39

Some time ago we had a problem with a huge number of tasks in the queues. Look at this ;-)

The first value is a creation time of the oldest task and the second value is a number of the tasks in a single queue. Reason of that? At first, a lot of tasks, really, we run big operations, some of them can be repeated because of some errors, and of course.. a lot of legacy code.

We wanted to fix that problem as quick as possible. The first try was to divide tasks for several queues. I've created a method which pushed a task to least loaded queue. It checks which queue has the smallest number of tasks and push the task into this queue. It is very simple. First, i've added several queues to queue.yaml, for example:

- name: analytics-0 rate: 200/s bucket_size: 100 - name: analytics-1 rate: 200/s bucket_size: 100 - name: analytics-2 rate: 200/s bucket_size: 100 - name: analytics-3 rate: 200/s bucket_size: 100 - name: analytics-4 rate: 200/s bucket_size: 100 - name: analytics-5 rate: 200/s bucket_size: 100 - name: analytics-6 rate: 200/s bucket_size: 100 - name: analytics-7 rate: 200/s bucket_size: 100 - name: analytics-8 rate: 200/s bucket_size: 100 - name: analytics-9 rate: 200/s

Here is a method.

import random from google.appengine.api.taskqueue import QueueStatistics from google.appengine.runtime import DeadlineExceededError def get_least_loaded_queues_list(queue_names, full_list=False): """ Example of queue_names parameter: ANALYTICS_QUEUES = [ 'analytics-{}'.format(number) for number in range(0, 10) ] full_list=False - return only most empty queue full_list=True - return all queues with number of tasks """ queues_list = {} try: queue_stats = QueueStatistics.fetch(queue_names) except DeadlineExceededError: logging.info('Cannot fetch queue statistics. Random queue.') return random.choice(queue_names) for queue in queue_stats: queues_list[queue.queue.name] = queue.tasks if not queues_list: queues_list = [('default', 1), ] # It is safe to always have a queue else: queues_list = sorted( queues_list.iteritems(), key=operator.itemgetter(1) ) return queues_list if full_list and len(queues_list) else queues_list[0][0]

This method requires one parameter - queue_names, which should be a list of queue names and optional full_list parameter which defines how the queues should be returned. If full_list is set to True then method will return a list with all passed queues from the least to the most loaded. If full_list will be set to False then the method will return only least loaded queue. This method gets statistics for all queues passed to the method and return a list with sorted names queues by their number of the tasks (or string with queue name if full_list is set to False). As you can see we catches one exception - DeadlineExceededError. Sometimes Google API can't get us queue statistic because of this exception, then i don't want to ask again. The method returns random queue - i think that it is faster.

How did this help us?

We had a smaller number of tasks in every queue and all the tasks were executed faster. But it wasn't enough for us because the queues still have too much tasks and sometimes we needed to wait several hours or even one day for some tasks (but not two months! ;-)). We decided to manage queue by checking queues names. I described it in a separate post. You can read about it here

Categories: FLOSS Project Planets

Andre Roberge: Disgusted by LavaSoft AdAware antivirus

Planet Python - Mon, 2015-01-19 16:33
A short while ago, I installed LavaSoft AdAware free anti-virus after reading a review praising it.  Many years (and computers) ago, before it had an anti-virus, I found LavaSoft AdAware very useful in cleaning up my PC and keeping it that way, free of various malware.

To make a long story short, I found that it caused the following problems:

1. I could no longer update Bracket's extensions.

2. I could not run IPython notebooks, getting the error message:

WebSocket connection failedA WebSocket connection could not be established. You will NOT be able to run code. Check your network connection or notebook server configuration.
It took me many hours, deleting my IPython profile, uninstalling and reinstalling IPython, doing the same with Tornado, trying in various browsers, searching for posts about people having had similar problems, etc., etc., etc. until I found this stackoverflow post which had a different problem but made me think that it was perhaps related to an anti-virus program issue.

I stopped MadAware (and restarted the default Windows Defender) ... no change.  On a hunch, I uninstalled it and rebooted (for the Nth time) my computer.

Sure enough, that was the problem.

After deleting MadAware, my problems went away.

I had to get this off my chest before I could resume working on tomorrow's lesson where I wanted to show off the IPython notebook.  

Categories: FLOSS Project Planets

Daniel Pocock: jSMPP project update, 2.1.1 and 2.2.1 releases

Planet Debian - Mon, 2015-01-19 16:29

The jSMPP project on Github stopped processing pull requests over a year ago and appeared to be needing some help.

I've recently started hosting it under https://github.com/opentelecoms-org/jsmpp and tried to merge some of the backlog of pull requests myself.

There have been new releases:

  • 2.1.1 works in any project already using 2.1.0. It introduces bug fixes only.
  • 2.2.1 introduces some new features and API changes and bigger bug fixes

The new versions are easily accessible for Maven users through the central repository service.

Apache Camel has already updated to use 2.1.1.

Thanks to all those people who have contributed to this project throughout its history.

Categories: FLOSS Project Planets

Music video: Dire Straits – Walk of Life

LinuxPlanet - Mon, 2015-01-19 13:06
By Vasudev Ram

An old favorite, listened to it again today. The video is good too.

Dire Straits

Dire Straits - Walk of Life:

- Vasudev Ram - Dancing Bison Enterprises

Share | var addthis_config = {"data_track_clickback":true}; Vasudev Ram

Categories: FLOSS Project Planets

libtool @ Savannah: GNU libtool-2.4.5 released [stable]

GNU Planet! - Mon, 2015-01-19 12:11


The Libtool Team is pleased to announce the release of libtool 2.4.5.

GNU Libtool hides the complexity of using shared libraries behind a
consistent, portable interface. GNU Libtool ships with GNU libltdl, which
hides the complexity of loading dynamic runtime libraries (modules)
behind a consistent, portable interface.

This is a bugfix release and a recommended upgrade for all users. Most
likely, this will be the last release that supports copying the libltdl
sources directly into your project -- libltdl is widely deployed now, and
there is absolutely no reason to give it special treatment compared to
any other library a project depends on.

Here are the compressed sources:
http://ftpmirror.gnu.org/libtool/libtool-2.4.5.tar.gz (1.7MB)
http://ftpmirror.gnu.org/libtool/libtool-2.4.5.tar.xz (952KB)

Here are the GPG detached signatures[*]:

Use a mirror for higher download bandwidth:

[*] Use a .sig file to verify that the corresponding file (without the
.sig suffix) is intact. First, be sure to download both the .sig file
and the corresponding tarball. Then, run a command like this:

gpg --verify libtool-2.4.5.tar.gz.sig

If that command fails because you don't have the required public key,
then run this command to import it:

gpg --keyserver keys.gnupg.net --recv-keys 151308092983D606

and rerun the 'gpg --verify' command.

This release was bootstrapped with the following tools:
Autoconf 2.69
Automake 1.15
Gnulib v0.1-336-g342d9f0


  • Noteworthy changes in release 2.4.5 (2015-01-19) [stable]
    • New features:

- Libtoolize searches for the best available M4 on the user PATH at
runtime, rather than settling for the first one found.

- Support munging sys_lib_dlsearch_path_spec with LT_SYS_LIBRARY_PATH
environment variable.

    • Bug fixes:

- Bail out at configure time if the installed M4 is not sufficient
for the purposes of libtoolize.

- freebsd-elf library versioning was upgraded incorrectly in 2.4.4,
but now works properly again.

- Fix a 2.4.4 regression so that libltdl subprojects do not warn
about missing libltdl/libltdl directory as in prior releases.

- When using Sun C++ on Solaris or GNU/Linux we used to set libtool's
postdeps permanently, based on the contents of $CXX and $CXXFLAGS at
configure time, which was brittle and error-prone. Now, we no
longer check for a SunCC ABI at configure time, but augment the
postdeps at libtool time based on the current invocation flags on
each call.

    • Changes in supported systems or compilers:

- /usr/local prefixed rpaths are now added to the link-line on
ia64-hp-hpux*, because the default system runtime loader path does
not contain them.

- Previously, when using Sun C++ on Solaris or GNU/Linux, `-Cstd -Crun`
flags were added to $postdeps unless CXX or CXXFLAGS contained
`-library=stlport4`. Newer releases have added other compiler flags
that are also incompatible with `-Cstd -Crun`, so now we don't add
them if any of `-std=c++[0-9][0-9]`, `-library=stdcxx4` or
`-compat=g` were found in CXX or CXXFLAGS when the Sun C++ compiler
is detected.


Categories: FLOSS Project Planets
Syndicate content