Matthew Tift: What I Learned about Drupal at a Go/Rust User Group

Planet Drupal - Thu, 2015-04-30 13:11
What I Learned about Drupal at a Go/Rust User Group

Last night I had the pleasure of attending the first joint meeting of Minnesota Go and Rust users, and arrived home inspired -- not just by Go and Rust, but also about Drupal and its community.

Attending a wide variety of local meetups is something I really enjoy. Since I moved back to the Twin Cities area nearly seven years ago, I have attended meetings related to PHP, Linux, Concrete5, NodeJS, .NET, JavaScript, Flash, and more. Each of these groups has a unique community and way of doing things, although just about every one of them involves pizza, beer, and predominantly white, male participants.

Given the fact that Rust and Go are both C-like low-level languages, it might not come as a surprise that this Go/Rust user group meeting was quite technical. There were three presentations and lots of terminals on the big screen. The first speaker introduced Rust by illustrating the process for generating the Fibonacci sequence. The end result looked something like this:

struct Fibonacci { curr: u32, next: u32, } impl Iterator for Fibonacci { type Item = u32; fn next(&mut self) -> Option { let new_next = self.curr + self.next; self.curr = self.next; self.next = new_next; Some(self.curr) } } fn fibonacci() -> Fibonacci { Fibonacci { curr: 1, next: 1 } } fn main() { for i in fibonacci().take(30) { println!("{}", i); } }

At other local meetups I have attended, this could have made for a dry, boring talk. However, at this Go/Rust meetup, each aspect of this Fibonacci program was followed by remarkably sophisticated audience questions and answers about language architecture, Rust's underlying assumptions, and frequent references to how Rust compared to C, Python, Go, and other languages. I learned a lot and found it very entertaining. (Incidentally, while I personally prefer Go, I think this is a great article comparing Go and Rust -- the author likes Rust).

I have attended a lot of DrupalCon and DrupalCamp sessions over the past five years or so, and I don't recall any of them feeling like this Go/Rust meetup. Perhaps there are a lot of more technical sessions and I just avoided them. Perhaps sessions like the (Drupal) Core Conversations are very technical, but I just don't notice it as much. Whatever the case, these Rust and Go talks got me thinking about Drupal and its community.

Drupal meetings, especially in the Twin Cities, tend to be focused on bringing in new users and not leaving anyone out of the conversations. That often means less technical presentations. Furthermore, every year at the Twin Cities DrupalCamp we have made it an explicit goal to welcome new users into our community.

Getting off the Drupal island to go to this Go/Rust group was a nice reminder of just how much Drupal lets me do without ever having to think about these low-level issues. For comparison, in PHP (Drupal is written in PHP), we might do something more simpler looking to create the Fibonacci series:

$fib = [1,0]; for ($i = 0; $i < 30; $i++) { $next = array_sum($fib); array_shift($fib); array_push($fib,$next); echo $next.', '; }

Or perhaps more relevant is the fact that I can just spin up a Drupal blog for one of my friends or one of my kids, easily download themes to radically change the appearance, and quickly add sophisticated integrations without writing code. And at that point get on with my hacking, or not. My code, my data, my content. Sometimes I lose track of that when I'm working on a project with teams of people I've never met, testing my code in ways I never would have anticipated, making spectacularly complex, cool websites.

The Go/Rust meetup had the unexpected effect of reminding me that Drupal can be very laid-back and non-technical, and that the web can be the same. Not all websites will need the sophistication of Drupal 8's fancy new configuration system, and its ability to facilitate development, testing, and production environments. Drupal allows me to come home, emboldened by a meetup, and quickly jot down my unfiltered, half-baked thoughts for all the world to see, without having to think about the complexities of immutable references or garbage collection. This Drupal site lets me spill out these ideas in short order.

As Drupal becomes an increasingly capable and accommodating entrance onto the wonderfully diverse information superhighway, it's nice to be reminded once in a while of the fact that Drupal also can do a darn good job of hiding complexity and letting us get on with creating and managing our content of all shapes, sizes, and qualities.

Tags: drupaldrupal planetgorustcommunity
Categories: FLOSS Project Planets

Lullabot: Beyond Decoupling: The Inherent Virtues of an API

Planet Drupal - Thu, 2015-04-30 13:00

Fellow Lullabot Andrew Berry has written an article on why to decouple. If you do go this route, it’s because you’ve thought a lot about how to separate concerns. Content consumers are separated from content producers. Front-end developers are freed from the dictates of a back-end CMS. This article isn't about the separation of concerns, but rather what lies at the middle of all of these concerns— your HTTP API.

Categories: FLOSS Project Planets

Drupal Watchdog: VIDEO: DrupalCon Amsterdam Interview: Leslie Hawthorn

Planet Drupal - Thu, 2015-04-30 12:54

A lively LESLIE HAWTHORN (Director of Developer Relations at Elasticsearch), finds herself momentarily corralled and questioned in a corner of the RAI Convention Center.

Besides her day-job, Leslie is involved in open source security issues and, as a second calling, mentors young women on the nuances of building careers in technology.

Privately, Leslie is an avid pickler, with a cupboard stash of yummy, lacto-fermented veggies: radishes, kohlrabi, cabbage, and cucumbers.

Categories: FLOSS Project Planets

Plasma 5.4 Kicked Off

Planet KDE - Thu, 2015-04-30 12:20

I just closed the Plasma 5.4 kickoff meeting. It was well attended with lots of Plasma devs and VDG members there. Here’s the minutes.

The To Do board [public link] has been updated with a load of new cards.

If you’re looking for a way to join the team of beautiful desktop developers (you can read that both ways and be correct) there’s plenty to do now.  For example the VDG have written a nice design document on System Settings which they need someone to help implement.

Plasma 5.4 is scheduled for August, it’ll be a great addition to Kubuntu 15.10.


Categories: FLOSS Project Planets

Daily Tech Video (Python): Andrew T. Baker: Demystifying Docker

Planet Python - Thu, 2015-04-30 11:37

Docker is sort of like a virtual machine, but not exactly. It lets you install applications more easily, and is extremely popular — but it’s hard for people to describe what it is, what it does, and why people are going ga-ga over it. In this talk, Andrew T. Baker introduces Docker to a Python audience. (So for example, he describes the Python-related Docker installations, and what extras they provide.) He explains how Docker simplifies application configuration and rollout, how it is different from (and similar to) other virtualization technologies, and where it’s going in the near future.

The post Andrew T. Baker: Demystifying Docker appeared first on Daily Tech Video.

Categories: FLOSS Project Planets

roomify.us: Vacation Rentals, Hotels, B&amp;Bs, Multiple Properties and Booking with Drupal.. oh my!

Planet Drupal - Thu, 2015-04-30 11:35
It’s now just over three months since roomify launched. We’ve been enjoying the process immensely, despite the inevitable frustration of trying to set up everything that a business requires. What has kept us going is the satisfaction of hitting product milestones, and hearing back from users and clients about how our products are helping them. (We also like getting paid - that’s awesome too!). As such, we are particularly pleased to share this update with you.
Categories: FLOSS Project Planets

Alexey Evseev: Python tips &amp; tricks

Planet Python - Thu, 2015-04-30 10:51

Recently i've read the book Learning Python, 5th Edition by Mark Lutz. Here is a list of most interesting insights for me.

  • set generation:

    {x for x in [1,2]} set(x**2 for x in [1,2])
  • dict generation:

    {x:x*2 for x in [1,2]} dict((x, x**2) for x in [1,2])
  • division of integers

    In python 3 division of integers returns float

    >>> 1 / 2 0.5 >>> - 1 / 2 ...
Categories: FLOSS Project Planets

Bad Voltage Season 1 Episode 41: Second Lunch is my Favourite Lunch

LinuxPlanet - Thu, 2015-04-30 10:19

Jono Bacon, Bryan Lunduke, Stuart Langridge and myself present Bad Voltage, in which hell may be slightly chillier than previously. Featuring the uses for abundant graphical power, the nature of what “cross-platform” really means, and:

  • 00:02:15 Google announce Google Fi, a new MVNO-style mobile network joining together wifi, Sprint, and T-Mobile for US customers and allowing international roaming and a pay-what-you-need rate for data. Is this actually a good idea? What about how it only works on the Nexus 6?
  • 00:18:00 We speak to Mashable senior tech correspondent and podcaster Christina “@film_girl” Warren about the Microsoft Build conference announcement that the Visual Studio Code editor is newly available for Linux as well as other platforms, and MS’s apparent increasing friendliness to open source. Is it real? Is it good?
  • 00:37:16 Bryan reviews the NVIDIA Jetson TK-1 development kit, a Raspberry-Pi-style small board but with 192 GPU cores
  • 00:51:12 A blog comment from Glyph suggesting that “Linux is not, practically speaking, more tweakable” than alternative desktop OSes starts a discussion about whether that’s the truth and why Linux desktop automation tools aren’t (or are) as good as AppleScript and Windows COM automation

Listen to 1×41: Second Lunch is my Favourite Lunch

As mentioned here, Bad Voltage is a project I’m proud to be a part of. From the Bad Voltage site: Every two weeks Bad Voltage delivers an amusing take on technology, Open Source, politics, music, and anything else we think is interesting, as well as interviews and reviews. Do note that Bad Voltage is in no way related to LinuxQuestions.org, and unlike LQ it will be decidedly NSFW. That said, head over to the Bad Voltage website, take a listen and let us know what you think.


Categories: FLOSS Project Planets

Kay Hayen: Nuitka Progress in Spring 2015

Planet Python - Thu, 2015-04-30 10:08

It's absolutely time to speak about what's going on with Nuitka, there have been a few releases, and big things are going to happen now. The ones I have always talked of, it's happening now.

I absolutely prefer to talk of things when they are completed, that is why I am shy to make these kinds of postings, but this time, I think it's warranted. The next couple of releases are going to be very different.


SSA (Single State Assignment Form)

For a long, long time already, each release of Nuitka has worked towards increasing "SSA" usage in Nuitka.

The component that works on this, is now called "trace collection", and does the major driving part for optimization. It collects "variable traces" and puts them together into "global" forms as well.

Based on these traces, optimizations can be made. Having SSA or not, is (to me) the difference between Nuitka as a mere compiler, and Nuitka as an optimizing compiler.

The major news is that factory versions of Nuitka now do this in serious ways, propagating values forward, and we also are close to eliminating dead assignments, some of which become dead by being having been forward propagated.

So we can now finally see that big step, jump really, happening, and Nuitka does now do some pretty good static optimization, at least locally.

Still, right now, this trival code assigns to a local variable, then reads from it to return. But not for much longer.

def f(): a = 1 return a

This is going to instantly give performance gains, and more importantly, will enable analysis, that leads to avoiding e.g. the creation of function objects for local functions, becoming able to in-line, etc.

This is major excitement to me. And I cannot wait to have the releases that do this.


The focus has also been lately, to reduce Nuitka's own memory usage. It has gone down by a large factor, often by avoiding cyclic dependencies in the data structures, that the garbage collector of Python failed to deal with properly.

The scalability of Nuitka also depends much on generated code size. With the optimization become more clever, less code needs to be generated, and that will help a lot. On some platforms, MSVC most notably, it can be really slow, but it's noteworthy that Nuitka works not just with 2008 edition, but with the lastest MSVC, which appears to be better.


There was not a whole lot to gain in the compatibility domain anymore. Nothing important certainly. But there are import changes.

Python 3.5

The next release has changes to compile and run the Python3.4 test suite successfully. Passing here means, to pass/fail in the same way as does the uncompiled Python. Failures are of course expected, and a nice way of having coverage for exception codes.

The new @ operator is not supported yet. I will wait with that for things to stabilize. It's currently only an alpha release.

However, Nuitka has probably never been this close to supporting a new Python version at release time. And since 3.4 was such a heavy drain, and still not perfectly handled (super still works like it's 3.3 e.g.), I wanted to know what is coming a bit sooner.

Cells for Closure

We now provide a __closure__ value for compiled functions too. These are not writable in Python, so it's only a view. Having moved storage into the compiled function object, that was easy.

Importing Enhancements

The the past couple of releases, the import logic was basically re-written with compatibility much increased. The handling of file case during import, multiple occurrences in the path, and absolute import future flags for relative imports has been added.

It's mainly the standalone community that will have issues, when just one of these imports doesn't find the correct thing, but picking the wrong one will of course have seriously bad impacts on compile time analysis too. So once we do cross module optimization, this must be rock solid.

I think we have gotten there, tackling these finer details now too.

Performance Graphs and Benchmarks

Nuitka, users don't know what to expect regarding the speed of their code after compilation through Nuitka, neither now nor after type inference (possibly hard to guess). Nuitka does a bunch of optimizations for some constructs pretty heavily, but weak at others. But how much does that affect real code?

There may well be no significant gain at all for many people, while there is a number for PyStone that suggests higher. The current and future versions possibly do speed up but the point is that you cannot tell if it is even worth for someone to try.

Nuitka really has to catch up here. The work on automated performance graphs has some made progress, and they are supposed to show up on Nuitka Speedcenter each time, master, develop or factory git branches change.


There currently is no structure to these graphs. There is no explanations or comments, and there is no trend indicators. All of which makes it basically useless to everybody except me. And even harder for me than necessary.

However, as a glimpse of what will happen when we in-line functions, take a look at the case, where we already eliminate parameter parsing only, and make tremendous speedups:

Lambda call construct case

Right now (the graph gets automatic updates with each change), what you should see, is that develop branch is 20 times faster than CPython for that very specific bit of code. That is where we want to be, except that with actually in-line, this will of course be even better.

It's artifical, but once we can forward propagate local function creations, it will apply there too. The puzzle completes.

But we also need to put real programs and use cases to test. This may need your help. Let me know if you want to.


The standalone mode of Nuitka is pretty good, and as usual it continued to improve only.

Nothing all that important going on there, except the work on a plug-in framework, which is under development, and being used to handle e.g. PyQt plug-ins, or known issues with certain packages.

The importing improvements already mentioned, have now allowed to cover many more libraries successfully than before.

Other Stuff Debian Stable

Nuitka is now part of Debian stable, aka Jessie. Debian and Python are the two things closest to my heart in the tech field. You can imagine that being an upstream worthy of inclusion into Debian stable is an import milestone to Nuitka for me.


Nuitka receives the occasional donation and those make me very happy. As there is no support from organization like the PSF, I am all on my own there.

This year I likely will travel to Europython 2015, and would ask you to support me with that, it's going to be expensive.

EuroPython 2015

I have plans to present Nuitka's function in-lining there, real stuff, on a fully and functional compiler that works as a drop-in replacement.

Not 100% sure if I can make it by the time, but things look good. Actually so far I felt ahead of the plan, but as you know, this can easily change at any point. But Nuitka stands on very stable grounds code wise.


Things are coming along nicely. When I started out, I was fully aware that the project is something that I can do on my own if necessary, and that has not changed. Things are going slower than necessary though, but that's probably very typical.

But you can join and should do so now, just follow this link or become part of the mailing list and help me there with request I make, e.g. review posts of mine, test out things, pick up small jobs, answer questions of newcomers, you know the drill probably.

Nuitka is about to make break through progress. And you can be a part of it. Now.


So, there is multiple things going on:

  • More SSA usage

    The next releases are going to be all about getting this done.

    Once we take it to that next level, Nuitka will be able to speed up some things by much more than the factor it basically has provided for 2 years now, and it's probably going to happen long before EuroPython 2015.

  • Function in-lining

    For locally declared functions, it should become possible to avoid their creation, and make direct calls instead of ones that use function objects and expensive parameter handling.

    The next step there of course is to not only bind the arguments to the function signature, but then also to in-line and potentially specialize the function code. It's my goal to have that at EuroPython 2015 in a form ready to show off.

When these 2 things come to term, Nuitka will have made really huge steps ahead and layed the ground for success.

From then on, a boatload of work remains. The infrastructure in place, still there is going to be plenty of work to optimize more and more things conretely, and to e.g. do type inference, and generate different codes for booleans, ints or float values.

Let me know, if you are willing to help. I really need that help to make things happen faster. Nuitka will become more and more important only.

Categories: FLOSS Project Planets

Bryan Pendleton: Warriors advance in the playoffs

Planet Apache - Thu, 2015-04-30 10:04

Having swept New Orleans, the Warriors now face Memphis, a much stronger team.

Bleacher Report breaks it all down in tremendous detail: Grizzlies Will Present Brand-New Playoff Challenge for Hungry Warriors Squad

But the NBA is about matchups, and not all matchups are equal. Just as the Grizzlies will have to solve, or at least partially impede, Curry and Thompson, the Warriors will have to slow down Memphis' post play.

And while it's easy to game-plan rotations and plan for various scenarios, on-court situations usually lend themselves to chaos. Will the Grizzlies' perimeter players be able to knock down shots and punish Golden State for overhelping? Will Green and Bogut handle their business enough to prevent the need for help in the first place?

Game 1 is Sunday, May 3rd.

Categories: FLOSS Project Planets

Jamie McClelland: So long email, it's been good to know yuh

Planet Debian - Thu, 2015-04-30 10:01

Yesterday I permanently deleted 15 years of email.

It wasn't because I didn't have enough hard disk space to store it. It's because I decided, after 15 years, that the benefits of keeping all this email did not outweigh the risks. Although I have never had my email subpoenaed, I have had many legal interactions due to my involvement with May First/People Link, some of which were about finding the real identities of May First/People Link members. I'd rather not risk compromising anyone or needlessly exposing my networks. Now I have an Inbox, Sent Box, Trash Box and Detected Spam Box. The Inbox I empty manually and the other boxes are automatically purged on a scheduled basis.

In this age of surveillance it's sad to see data evaluated based on risk of exposure.

Categories: FLOSS Project Planets

End Point: Raw Packet Manipulation with Scapy

Planet Python - Thu, 2015-04-30 09:36

Scapy is a Python-based packet manipulation tool which has a number of useful features for those looking to perform raw TCP/IP requests and analysis. To get Scapy installed in your environment the best options are to either build from the distributed zip of the current version, or there are also some pre-built packages for Red Hat and Debian derived linux OS.

Using Scapy

When getting started with Scapy, it's useful to start to understand how all the aspects of the connection get encapsulated into the Python syntax. Here is an example of creating a simple IP request:

Welcome to Scapy (2.2.0) >>> a=IP(ttl=10) >>> a <IP ttl=10 |> >>> a.dst="" >>> a <IP ttl=10 dst= |> >>> a.src '' >>> a.ttl 10

In this case I created a single request which was point from one host on my network to the default gateway on the same network. Scapy will allow the capability to create any TCP/IP request in raw form. There are a huge number of possible options for Scapy that can be applied, as well as huge number of possible packet types defined. The documentation with these options and packet types is available on the main site for Scapy.

Creating custom scripts with Scapy

Using Scapy within Python rather than as a standalone application would allow for creating more complex packets, sending them, and then parsing the response that is given. Here is a simple tester script example in which I will initiate a HTTP 1.1 request: #! /usr/bin/env python import logging logging.getLogger("scapy").setLevel(1) from scapy.all import * def make_test(x,y): request = "GET / HTTP/1.1\r\nHost: " + y + "\r\n" p = IP(dst=x)/TCP()/request out = sr1(p) if out: out.show() if __name__ == "__main__": interact(mydict=globals(), mybanner="Scapy HTTP Tester")

Within this script there is the make_test function which takes as parameters the destination address and host header string respectively. The script will attempt to send the HTTP GET request to that address with the proper Host header set. If the request is successful, it will print out the details of the response packet. It would also be possible to perform more complex analysis of this response packet using the built in psdump and pdfdump functions which will create a human readable analysis of the packet in PostScript and PDF respectively.

Welcome to Scapy (2.2.0) Scapy HTTP Tester >>> make_test("www.google.com","google.com") Begin emission: ...Finished to send 1 packets. .* Received 5 packets, got 1 answers, remaining 0 packets ###[ IP ]### version= 4L ihl= 5L tos= 0x20 len= 56 id= 64670 flags= frag= 0L ttl= 42 proto= tcp chksum= 0x231b src= dst= \options\ ###[ TCP ]### sport= http dport= ftp_data seq= 1130043850 ack= 1 dataofs= 9L reserved= 0L flags= SA window= 42900 chksum= 0x8c7e urgptr= 0 options= [('MSS', 1430), (254, '\xf9\x89\xce\x04bm\x13\xd3)\xc8')] >>> Conclusions

Scapy is a powerful tool, if a bit daunting in syntax initially. Creating raw TCP/IP packets systematically will probably challenge most people's understanding of the TCP/IP stack (it certainly did mine!) but exposing this level of configuration has serious advantages. Full control of the requests and responses as well as ability to add custom Python logic allows Scapy to become a packet foundry which you can use for things like unit testing of web applications, verification of state of an unknown network, etc. I will definitely be using Scapy in the future when performing raw HTTP testing of web applications.

Categories: FLOSS Project Planets

Olivier Berger: A howto record a screencast on Linux and tablet

Planet Debian - Thu, 2015-04-30 09:31

I’ve documented the process of how I’m trying to perform DIY screencast recording, for the needs of a MOOC.

I’m working on my Debian or Ubuntu desktop, using an external graphic tablet with integrated display for annotating slides.

The main software used for the process are xournal for annotating PDFs and vokoscreen for the screen and video recording.

Here is the documentation : http://www-public.telecom-sudparis.eu/~berger_o/screencast-linux.html

And here’s the companion video : https://youtu.be/YxcUNqXPYZE

I hope this is useful to some.

Categories: FLOSS Project Planets

Jim Birch: Integrating Drupal with Cloudflare

Planet Drupal - Thu, 2015-04-30 06:30

The super smart lead developer at Xeno Media first brought the Cloudflare service to my attention as it was integrated with one of the hosts we were using.  We experimented with a site that was experiencing outages due to traffic spikes it received after sending an email newsletter.  After the switch, the server never went down again, saw considerable speed improvements, and we were able to quadruple the number of emails we could send per hour, with no performance hits.

According to Cloudflare, sites that use their service:
  • Load twice as fast
  • Use 60% less bandwidth
  • Have 65% fewer server requests
  • Are way more secure

If a free Content Delivery Network that makes your site seriously super awesome, incredibly blazing fast wasn't enough, the more you look at Cloudflare's features, the more you will be impressed.  Not only does Cloudflare serve your site from a distributed network, closer to where the user actually is, it optimizes, aggregating content from different external services (think Google/Facebook/Marketo) into one, speeding up delivery to the browser

Read more

Categories: FLOSS Project Planets

Intellimath blog: What is recordclass

Planet Python - Thu, 2015-04-30 04:50

Recordclass is a library for creation of record-like classes – "mutable" analog of collections.namedtuple. They support same API as namedtuples. In addition records support assignment operations.

Read more… (3 min remaining to read)

Categories: FLOSS Project Planets

Tennessee Leeuwenburg: The Effect of Randomising the Order of Your Inputs

Planet Python - Thu, 2015-04-30 04:50
This post is going to focus on the impacts on a single example of randomising the order of your inputs (training examples) in one particular problem. This will look at the effect of adding input shuffling to a poorly-performing network, and also the effect of removing it from a well-performing network.

Let's just straight to the punch: Always randomise the order. The network will start effective training much sooner. Failure to do so may jeopardise the final result. There appears to be no downside. (if anyone has an example where it actually makes things worse, please let me know)

Here's a screenshot from my Ipython notebook of the initial, poorly-performing network. Sorry for using an image format, but I think it adds a certain amount of flavour to show the results as they appear during development. This table shows that after 13 iterations (picked because it fits in my screenshot easily), the training loss is 1.44, and the % accuracy is 11.82%. This network eventually trains to hit a loss of about 0.6, but only after a large (800-odd) number of iterations.

It takes 100 iterations or so to feel like the network is really progressing anywhere, and it's slow, steady progress the whole way through. I haven't run this network out to >1k iterations to see where it eventually tops out, but that's just a curiosity for me. Alternative techniques provide more satisfying results much faster.

Here's the improved result. I added a single line to the code to achieve this effect:

train = shuffle(train)Adding the shuffling step wasn't a big deal -- it's fast, and conceptually easy as well. It's so effective, I honestly think it would be a good thing for NN libraries to simply to by default rather than leave to the user. We see here that by iteration 13, the valid loss is 0.72 as opposed to 2.53 in the first example. That's pretty dramatic.

If anyone knows of any examples where is it better not to randomise the input examples, please let me know!!!

For the time being, I think the benefits of this result are so clear, that a deeper investigation is not really called for in fact. I'm just going to add it to my 'standard technique' for building NNs going forward, and consider changing this step only if I am struggling with a particular problem-at-hand. What's more likely is that more sophisticated approaches to input modification will be important, rather than avoiding the step entirely. I'm aware that many high-performing results have been achieved by transforming input variables to provide additional synthetic learning data into the training set. Examples of this include image modifications such as skew, colour filtering and other similar techniques. I would be very interested to learn more about other kinds of image modification preprocessing, like edge-finding algorithms, blurring and other standard algorithms.

Interestingly, the final results for these two networks are not very different. Both of them seem to train up to around the maximum capability of the architecture of the network. Neither result arising from this tweak only approach the performance provided by the best known alternative approach that I have copied from.

I wondered whether shuffling the inputs was a truly necessary step, or just a convenient / beneficial one. If you don't care about letting the machine rip through more iterations, then is this step really relevant? It turns out the answer is a resounding "Yes", at least sometimes.

To address this I took the best-performing code (as discussed in the last post) and removed the shuffle step. The result was a network which trained far more slowly, and moreover did not reach the optimal solution nor even approach it. The well-performing network achieved a valid loss of 0.5054, which looks pretty good compared to random forest.

Here is the well-performing network with input shuffling removed. You can see here that the training starts off badly, and gets worse. Note the "valid loss" is the key characteristic to monitor. The simple "loss" improves. This shows the network is remembering prior examples well, but extrapolating badly.

After 19 rounds (the same number of iterations taking by the most successful network design), avoiding the shuffling step results a network that just wanders further and further off base. We end up with a valid loss of 13+, which is just way off.
As an interesting aside, there is a halting problem here. How do we know, for sure, that after sufficient training this network isn't suddenly going to 'figure it out' and train up to the required standard? After all, we know for a fact that the network nodes are capable of storing the information of a well-performing predictive system. It's clearly not ridiculous to suggest that it's possible. How do we know that the network is permanently stuck? Obviously indications aren't good, and just as obviously we should just use the most promising approach. However, this heuristic is not actually knowledge.

Question for the audience -- does anyone know if a paper has already been publishes analysing the impacts of input randomisation across many examples (image and non-image) and many network architectures? What about alternative input handling techniques and synthetic input approaches?

Also, I haven't bothered uploading the code I used for these examples to the github site, since they are really quite simple and I think not of great community value. Let me know if you'd actually like to see them.
Categories: FLOSS Project Planets

WordCamp Finland 2015 brings WordPress enthusiasts together

LinuxPlanet - Thu, 2015-04-30 04:21

WordCamps are casual, locally-organized conferences covering everything related to WordPress, the free and open source personal publishing software that powers over 75 million sites on the web.

In May of 2015, WordCamp will finally have its debut in Finland. The event is set to take place at the home base of Seravo in Tampere.

WordCamps come in all different flavours, based on the local communities that produce them. In general, WordCamps include sessions on how to use WordPress more effectively, plugin and theme development, advanced techniques and security. Conference talks can also dig into such topics as how WordPress can be used in marketing or media businesses.

WordCamps are attended by people ranging from blogging newbies to professional WordPress developers and consultants, and usually combine scheduled programming with unconference sessions and other activities.

The first WordCamp was organised in San Francisco by Matt Mullenweg in 2006. Since then, local communities around the world have organised over three hundred WordCamps.

This year’s WordCamp Finland is held on May 8–9. The conference will bring together more than two hundred WordPress enthusiasts. Seravo is proud to participate by sponsoring the event and helping with the organising effort. The tickets to the event have already been sold out, but you can always participate in the discussions by following @WordCampFinland and #wcfi on Twitter.

See you there!

More information: WordCamp Finland 2015

Categories: FLOSS Project Planets

Acquia: PHP Reset, PHP Renaissance: Unify everything in PHP with Composer

Planet Drupal - Thu, 2015-04-30 02:20
Language Undefined

It was great to get the chance to sit down and talk with Jordi Boggiano at SymfonyCon Madrid 2014. Jordi is responsible for Composer, one of the most important pieces of technology that is driving PHP interoperability and the PHP "renaissance" of the last couple of years. He's also on the Symfony2 core team, "and bad about telling things about myself."

Categories: FLOSS Project Planets

Adrian Sutton: Making End-to-End Tests Work

Planet Apache - Thu, 2015-04-30 01:47

The Google Testing Blog has an article “Just Say No to More End-to-End Tests” which winds up being a rather depressing evaluation of the testing capabilities, culture and infrastructure at Google. For example:

Let’s assume the team already has some fantastic test infrastructure in place. Every night:

  1. The latest version of the service is built. 
  2. This version is then deployed to the team’s testing environment. 
  3. All end-to-end tests then run against this testing environment. 
  4. An email report summarizing the test results is sent to the team.

If your idea of fantastic test infrastructure starts with the words “every night” and ends with an email being sent you’re doomed. Bryan Pendleton does a good job of analysing and correcting the details so I won’t recover that ground. Instead, let me provide a view of what reasonable test infrastructure looks like.

At LMAX we’ve recently reached the milestone of 10,000 end to end acceptance tests. We’ve obviously invested a lot of time in building up all those tests but they’re invaluable in the way they free us to try daring things and make sweeping changes, confident that if anything is broken it will be caught. We’re happy to radically restructure components in ways that require lots of changes to unit tests because of those end-to-end tests.

We also have huge numbers of unit tests, integration tests, performance tests, static analysis and various other forms of tests, but the end-to-end tests are far more than a sanity-check, they’re a primary form of quality control.

Those end-to-end tests, or acceptance tests as we call them:

  • run constantly through the day
  • complete in around 50 minutes, including deploying and starting the servers, running all the tests and shutting down again at the end
  • are all required to pass before we consider a version releasable
  • are included in our information radiators to ensure the team has constant visibility into the test results
  • are owned by the whole team – testers, developers and business analysts together

That’s pretty much entry-level for doing end-to-end testing (or frankly any testing). We’ve also got a few extra niceties that I’ve written about before:

Plus the test results are displayed in real-time, so we don’t even have to wait for the end of the test run to see any failures. Tests that failed on the previous run are run first to give us quick feedback on whether they’ve been fixed or not.

There’s lots of great stuff in there, but we have more work to do. We have an intermittency problem. When we started out we didn’t believe that intermittency could be avoided and accepted a certain level of breakage on each build – much like the Google post talks about expecting 90% of tests to pass. That attitude is a death-knell for test reliability. If you don’t have all the tests passing consistently, gradually and inevitably more and more tests become intermittent over time.

We’ve been fighting back hard against intermittency and making excellent progress – we’ve recently added the requirement that releases have no failures and green builds are the norm and if there are intermittent failures it’s usually only one or two per run. Currently we’re seeing an intermittent failure rate of around 0.00006% of tests run (which actually sounds pretty good but with 10,000 tests that’s far too many runs with failures that should have been green).

But improvements come in waves with new intermittency creeping in because it can hide in amongst the existing noise. It has taken and will take a lot of dedication and commitment to dig ourselves out of the intermittency hole we’re in but it’s absolutely possible and we will get there.

So next time you hear someone try to tell you that end-to-end tests aren’t worth the effort, point them to LMAX. We do end-to-end testing big time and it is massively, indisputably worth it. And we only expect it to become more worth it as we reduce the intermittency and continue improving our tools over time.

Categories: FLOSS Project Planets

Jason Meyers: Building a Keyboard: Part I - Plates, Switches, and Keycaps

Planet Python - Thu, 2015-04-30 01:00

I’ve been into keyboards for quite a while, and into mechanical keyboards for the past few years. I’ve owned many so called “awesome gaming” keyboards from Razer, Microsoft, and Corsair; however, a few years ago I got my first mechanical keyboard. I wasn’t sure what the hype was all about. The gaming keyboards I had been using where in the $100 “high end” keyboard market, and I was sure nothing could be better. Then I learned about the difference between rubber dome keyboards and those with actual mechanically activated switches. I bought a CODE keyboard with Cherry MX Green switches, and it blew me away. It honestly felt good to type, and type I did for hours on that board at home, and then at work. (Sorry it was so loud former coworkers.)

Then I got the urge to get a second board, and to try some other switches. In the end I tried every kinda of Cherry Mx, Matias, and Topre switch. I also sampled Kailh and Gateron switches. The switches are the foundation components of a keyboard. They define how hard it is to press the keys (force), what it feels like to press them down (linear, tactile, or click), and how far the switches need to be pressed down (actuation point and bottom out). The rest of the article is gonna focus on Cherry Mx switches as they are the mostly commonly used switches, and my personal favorites the Matias quiet clicks. The Kailh and Gateron are Cherry Mx clones. Topre switches are a completely different type of switch that uses electro-capacitive technology under an inverted rubber cup.

Now… Now I wanna build a keyboard from scratch

No prebuilt fancyness except a teensy microcontroller, keyswitches, wires, solder and C… plain old C.

Details of Cherry Mx and Matias Switches

The first thing to know about switches, is that there is no single best switch. Different switches are preferred for different usages and vary by each individual. You can see a good breakdown of the stats about Cherry Mx switches on wikipedia. As a synopsis of that information basically switches are divided by their feel and weight. Keyswitches have two important points in the keystroke, the actuation point and the bottom out point. The actuation point is when the keystroke is registered, and bottoming out is when it hits the bottom of the switch. On a rubber dome keyboard these are often one in the same. So the keystroke is registered when the the PCB underneath the rubber dome is touched. This hurts my hands just thinking back to those days. If you take a look at an animation of the Cherry Mx Clear switch. (my personal favorite)

You can see the way the tactile bump tells the user the switch has been actuated early in the keystroke. Once you learn to feel for that bump, you can swiftly and lightly press the switches to just that point and avoid the abrupt stop of bottoming out. The tactile bump is one type of Cherry MX switch feeling, some switches also make an audible clicking sound; however, some people prefer the smoothness of a linear switch. With a linear switch there is still quite a bit of separation between the actuation point and bottoming out, and you can learn to touch type on them as well. The weight of a switch refers to how hard you have to press to reach the actuation and then bottom out point. Typically a light switch is around 45g of force to actuate with mediums around 55g and heavy switches anything higher than 60g. Bottoming out force is often either the same or slightly lower than the actuation force.

Here is a simple chart with the most common types of Cherry Mx Switches:

Switch Type Weight Type Black Heavy Linear Red Light Linear Brown Light Tactile Clear Medium Tactile Blue Medium Click Green Heavy Click

The Matias Quiet Click switch is roughly the same specs as a Cherry MX Clear but with a higher actuation point and a stronger bump. It also gives in completely after actuation with just a 35g bottoming out force compared to the 45g of the Clear. This makes it easier to feel the switch give and start to pull the finger back up. Due to some old hand injury, it’s this bottoming out that causes pain in my fingers after a few hours of typing on a rubber dome or laptop keyboard.

The switches snap into metal plates or mount directly onto PCBs depending on the desired feel and type of keyboard. Since we’re building from scratch I won’t have a premade fancy PCB I’m going to be doing some old fashion hand wiring. The plate I’m using is from Ortholinear Keyboards and is called an Atomic keyboard. It’s a grid like board with a small 2 unit space key. Here is an example of the plate and some switches installed on it and one waiting to go in.

You might notice the the stems of these switch are a plus sign and kind of a translucent white colors. The plus sign is the standard for all Cherry Mx Switches, and the color tells me this is a Clear switch. (trust me on that one) On the switch to the left, it’s two units wide, so we have the keyswitch surrounded by a stabilizer. A stabilizer is used to make sure that a key presses down evenly no matter where on the key you press it. We’ve all had that weird space bar that kinda pushed down awkwardly. (I’m looking at you Microsoft Ergo 4000!) There are several different types of stabilizers; however, the ones pictures are a plate mounted Cherry Mx stabilizer.

A stabilizer is made of two plate mounted slides, two sliders and a small bar that ensures that both sides of the stabilizer move up and down together. Just like the key switches, they snap into the plate. Here is a view of those same switches from the bottom


Keycaps are a subject one could get lost in but from a basic stand point they are typically made from ABS or PBT plastic. PBT keycaps normally have a nicer texture, and are thicker and better made than ABS keycaps. Nonetheless, the simplicity and variety with which ABS keycaps are made make them extremely popular in the show off crowd. I want a board that feels good, but some extreme keyboard modders just wanna look good. There are entire parts of the internet devoted to just this part of keyboarding alone. (sigh so many wasted electrons). Anyway for my build I’m using some high quality simple PBT keycaps from Signature Plastics Here’s a good pic of them from below.

You can see there are stabs built inside the keycap to accept those plus sign posts on the switches, and the longer switches with the stabilizers have multiple stabs. The position of these stabs depends on the layout of your keyboard and how the switches are positioned on your plate or PCB. Since, I’m crazy nothing about this board is standard, but it is very geometric :). So here is our board after putting on the keycaps.

In the next part we’ll talk about how keyboards work and how we’re going to wire this beast up to our microcontroller. Part 3 will be all keyboard layouts and then it’s about that C, and how we build the firmware for our device.

I’ll leave you with some of my previous keyboards during my misspent designer phase.

Building a Keyboard: Part I - Plates, Switches, and Keycaps was originally published by Jason Myers at from n00b import python on April 30, 2015.

Categories: FLOSS Project Planets
Syndicate content