Somehow I stumbled across the inaugural issue of the Hacker School journal: Code Words: A quarterly publication about programming from Hacker School.
I haven't had time to read the entire issue yet, but so far I'm really pleased.
You know that you're really reading a publication targeted at programmers when the first paragraph of the first article is as follows:Floating point is a giant mess. There are known best practices for most issues that come up in everyday use (e.g., using Kahan summation or adding stochastic noise to reduce aggregate numerical error), but there are still corner cases. Different libraries and implementations give different results because there’s no generally accepted standard.
"Numerical error"? "Kahan summation"? "Stochastic noise"?
Oh, yes, yes, YES! Bring it on!
And I also really enjoyed An introduction to functional programming, with its straight-forward, plain-english approach:When people talk about functional programming, they mention a dizzying number of “functional” characteristics. They mention immutable data, first class functions and tail call optimisation. These are language features that aid functional programming. They mention mapping, reducing, pipelining, recursing, currying and the use of higher order functions. These are programming techniques used to write functional code. They mention parallelization, lazy evaluation and determinism. These are advantageous properties of functional programs.
Ignore all that. Functional code is characterised by one thing: the absence of side effects. It doesn’t rely on data outside the current function, and it doesn’t change data that exists outside the current function. Every other “functional” thing can be derived from this property. Use it as a guide rope as you learn.
They seem to be off to a great start, so I'm hoping that Hacker School continue to find success, and continue to publish more issues of their journal.
Meanwhile, reading the journal led me to the Hacker School blog, which seems pretty great too; I particularly like their "Read Along" feature, in which they select a research topic each week, post a pointer to the paper with some introductory thoughts, and encourage the audience to read along with the paper and contribute their own reactions.
If you're looking for some good reading to strengthen and extend your programming skills, give the Hacker School publications a try!
Python links the module namespace directly to the layout of the source locations on the filesystem. And this is mostly fine, certainly for applications. For libraries sometimes one might want to control the toplevel namespace or API more tightly. This also is mostly fine as one can just use private modules inside a package and import the relevant objects into the __init__.py file, optionally even setting __all__. As I said, this is mostly fine, if sometimes a bit ugly.
However sometimes you have a library which may be loading a particular backend or platforms support at runtime. An example of this is the Python zmq package. The apipkg module is also a very nice way of controlling your toplevel namespace more flexibly. Problem is once you start using one of these things Pylint no longer knows which objects your package provides in it's namespace and will issue warnings about using non-existing things.
Turns out it is not too hard to write a plugin for Pylint which takes care of this. One just has to build the right AST nodes in place where they would be appearing at runtime. Luckily the tools to do this easily are provided:def transform(mod): if mod.name == 'zmq': module = importlib.import_module(mod.name) for name, obj in vars(module).copy().items(): if (name in mod.locals or not hasattr(obj, '__module__') or not hasattr(obj, '__name__')): continue if isinstance(obj, types.ModuleType): ast_node = [astroid.MANAGER.ast_from_module(obj)] else: if hasattr(astroid.MANAGER, 'extension_package_whitelist'): astroid.MANAGER.extension_package_whitelist.add( obj.__module__) real_mod = astroid.MANAGER.ast_from_module_name(obj.__module__) ast_node = real_mod.getattr(obj.__name__) for node in ast_node: fix_linenos(node) mod.locals[name] = ast_node
As you can see the hard work of knowing what AST nodes to generate is all done in the astroid.MANAGER.ast_from_module() and astroid.MANAGER.ast_from_module_name() calls. All that is left to do is add these new AST nodes to the module's globals/locals (they are the same thing for a module).
You may also notice the fix_linenos() call. This is a small helper needed when running on Python 3 and importing C modules (like for zmq). The reason is that Pylint tries to sort by line numbers, but for C code they are None and in Python 2 None and an integer can be happily compared but in Python 3 that is no longer the case. So this small helper simply sets all unknown line numbers to 0:def fix_linenos(node): if node.fromlineno is None: node.fromlineno = 0 for child in node.get_children(): fix_linenos(child)
Lastly when writing this into a plugin for Pylint you'll want to register the transformation you just wrote:def register(linter): astroid.MANAGER.register_transform(astroid.Module, transform)
And that's all that's needed to make Pylint work fine with dynamically populated package namespaces. I've tried this on zmq as well as on a package using apipkg and its seems to work fine on both Python 2 and Python 3. Writing Pylint plugins seems not too hard!
After yesterday's late night accomplishments, today I fixed up the UI of joytest a bit. It's still not quite what I think it should look like, but at least it's actually usable with a 27-axis, 19-button "joystick" (read: a PS3 controller). Things may disappear off the edge of the window, but you can scroll towards it. Also, I removed the names of the buttons and axes from the window, and installed them as tooltips instead. Few people will be interested in the factoid that "button 1" is a "BaseBtn4", anyway.
The result now looks like this:
If you plug in a new joystick, or remove one from the system, then as soon as udev finishes up creating the necessary device node, joytest will show the joystick (by name) in the treeview to the left. Clicking on a joystick will show that joystick's data to the right. When one pushes a button, the relevant checkbox will be selected; and when one moves an axis, the numbers will start changing.
I really should have some widget to actually show the axis position, rather than some boring numbers. Not sure how to do that.
Something happened a little while ago which we did not have time to commensurate properly. Our Rocker image for R is now the official R image for Docker itself. So getting R (via Docker) is now as simple as saying docker pull r-base.
This particular container is essentially just the standard r-base Debian package for R (which is one of a few I maintain there) plus a mininal set of extras. This r-base forms the basis of our other containers as e.g. the rather popular r-studio container wrapping the excellent RStudio Server.
Docker itself continues to make great strides, and it has been great fun help to help along. With this post I achieved another goal: blog about Docker with an image not containing shipping containers. Just kidding.
Before I begin, I want to make very clear that most of what I'm about to explain are 'tricks'. They aren't "best practices", and in at least one case, is possibly inadvisable.
Speaking of inadvisable practices, at some point I'll write a 'setup.py traps' blog post, which are things I believe you should never, ever do in a setup.py module.Tricks
These are tricks I have to make package management in python a tiny bit easier. Before you attempt to implement them, I recommend you have at least basic experience with creating new packages. Two ways to learn about python packaging are the New Library Sprint (beginner friendly) and the Python Packaging User Guide (more advanced).'python setup.py publish'
This is where it all started. One day I was looking at some of Tom Christie's code and discovered the python setup.py publish command inside the setup.py module of Django Rest Framework. It goes something like this:# setup.py import os import sys # I'll discuss version tricks in a future blog post. version = "42.0.0" if sys.argv[-1] == 'publish': os.system("python setup.py sdist upload") os.system("python setup.py bdist_wheel upload") print("You probably want to also tag the version now:") print(" git tag -a %s -m 'version %s'" % (version, version)) print(" git push --tags") sys.exit() # Below this point is the rest of the setup() function
What's awesome about this is that using this technique I don't have to look up the somewhat cryptic python setup.py sdist upload command, or the actually cryptic python setup.py bdist_wheel upload. Instead, when it's time to push one of my packages to PyPI, I just type:$ python setup.py publish
Much easier to remember!'python setup.py tag'
The problem with Tom Christie's python setup.py publish command is that it forces me to type out the git tag command. Okay, let's be honest, it forces me to copy/paste the output of my screen. Therefore, all on my very own, I 'invented' the python setup.py tag command:# setup.py if sys.argv[-1] == 'tag': os.system("git tag -a %s -m 'version %s'" % (version, version)) os.system("git push --tags") sys.exit()
Pretty nifty, eh? Now I don't have to remember so many cryptic git commands. And I get to shorten the python setup.py publish command:if sys.argv[-1] == 'publish': os.system("python setup.py sdist upload") os.system("python setup.py bdist_wheel upload") sys.exit()
When I need to do a version release, I commit my code then type:$ python setup.py publish $ python setup.py tag
Why don't I combine the commands? Well, you aren't supposed to put things like 'RC1' or '-alpha' in your PyPI version names. By seperating the commands I have finer grained control over my package releases. I'm encouraged to place alpha, beta, and release candidates in git tags, rather than formal PyPI releases.'python setup.py test'
I'm fairly certain some of my readers are going to have a seriously problem with this trick. In fact, depending on the the response of those who manage Python's packaging infrastructure, it might be moved to my forthcoming 'traps' blog post.
I like py.test. I've blogged about the use of py.test. I try to use it everywhere. Yet, I'm really not a fan of how we're supposed tie it into python setup.py test. The precise moment I get uncomfortable with py.test is when it makes me add special classes into setup.py.
Fortunately, there is another way:if sys.argv[-1] == 'test': test_requirements = [ 'pytest', 'flake8', 'coverage' ] try: modules = map(__import__, test_requirements) except ImportError as e: err_msg = e.message.replace("No module named ", "") msg = "%s is not installed. Install your test requirments." % err_msg raise ImportError(msg) os.system('py.test') sys.exit()
Which means I get to use py.test and python setup.py test with a trivial addition of code:$ python setup.py test
In theory, one could run pip install on the missing requirements, or call them from a requirements file. However, since these are 'tricks', I like to keep things short and sweet. If I get enough positive results for this one I'll update this example to include calling of pip for missing requirements.
note: This doesn't mean I'm not using tox. In fact, I use tox to call my version of python setup.py test.What about subprocess?
There are those who will ask, "Why aren't you using the subprocess library for these shell commands?"
My answer to that question is, "Because if I need a nuclear weapon to kill a rabbit maybe I'm overdoing things." For these simple tricks, the os.system() function is good enough.Why not just use a Makefile?
While I code primarily on Mac OSX and Linux, most of my open source packages are used Windows. Thanks to AppVeyor, I'm testing more and more of them in that environment. In fact, I'll probably be modifying these "tricks" to work better for Windows users.Traps!
Stay tuned for my 'traps' blog post to come out early in 2015.Updates
- 2014/12/21 - Added a note about using tox.
- 2014/12/21 - Added a note about Makefile and Windows
This week we held our last board meeting of the year, and we covered a lot a of ground. Unlike November, where we had a relatively short public meeting, this session took the full two hours to get through. We covered a lot of topics, from DrupalCon Amsterdam to updates from the Working Groups. As always, you can review the minutes, the materials, or the meeting recording to catch up on all the details. Here's a summary for you as well.Operational Update
The month of November was short given the US holiday (Thanksgiving), but we still have a number of initiatives that we managed to push significantly forward. Among them:
- Licensing Working Group: We recently put out a call for volunteers for the Licensing Working Group, whose charter was approved at the November board meeting. If you are interested in licensing issues, we hope that you will consider applying. The Licensing Working Group will play a pivotal role in helping contributors navigate what is and isn't allowed quickly and in keeping our code GPL compliant and safe.
- Social capital and the Driesnote: In Amsterdam, Dries laid out a vision for the future of contribution in our community. We also began sharing a plan for Drupal.org in 2015 at DrupalCon Amsterdam that aligns with that vision. We have been laying the groundwork over the last few months, working on commit messages and profile improvements that will make it possible to illustrate not just code contribution, but the many kinds of contribution that individuals (and soon, organizations!) make in the Drupal community.
- 2015 workplans: Association staff have been very busy preparing workplans for 2015 as well. The DrupalCon team has been rethinking food and fun at DrupalCons based on recent survey feedback. The Drupal.org team has been working on the roadmap. Our revenue team has been planning for solidifying the new revenue programs we launched this year (like Drupal Jobs) and planning for new opportunities as well.
- DrupalCon Latin America: We are all very excited to get to Bogota for DrupalCon Latin America next February. Everything is on track for this event from a logistics standpoint. We have speakers and space and now all we need are more people. We are planning for 400 people to be there and have about 90 registered so far. Normally, we would have a much higher percentage of tickets sold at this point, but with a very minimal price increase between rates, and with the holidays, we suspect we will see more registrations closer to the date of the Con.
We're coming up to a pretty pivotal time for Drupal marketing. As we near a Drupal 8 release, the Marketing and Branding Committee can help lead the community in making this the biggest Drupal release ever. In the meeting, the Board voted to approve the appointment of Gina Montoya of Blink Reaction as the new Chair of that committee. Congratualtions and thank you Gina!DrupalCon Amsterdam Wrap
Over the last few Cons, we have worked hard to collect more data about our attendees and their experience and to analyze that data to understand what's working and what's not. We looked at a LOT of data for DrupalCon Amsterdam, and shared what we learned and what we will be applying to future Cons. In short - the Con was very successful financially, but we continue to struggle to collect session evaluations and, frankly, the food was terrible. We are very sorry about that. Basically, until the last two weeks before the Con, ticket sales looked slow, so we modified the catering order to mitigate the budget loss we were facing. When the upsurge in ticket sales began, it was too late to change our box-lunch order. We will definitely be rethinking food overall. It's one of the single biggest expenses at DrupalCons, and we know it's one of the best ways to keep attendees happy. Check out the complete overview.2015 Budget and Leadership Plan Highlights
The board approved the 2015 Budget and Leadership Plan in executive session at the previous board meeting. We reviewed the highlights this month in the public board meeting. If you're interested in even more details, you can watch the recording of the webcast that we presented on Thursday, 18 December.Governance Updates Board Term Limits
The Board of Directors operate under a set of rules that govern issues like how the board is structured, the length of terms, etc. This set of rules is codified into the organization's Bylaws. Like any good governance document, and like any good governance group, it makes sense to review how the group operates and what rules might need to be changed in order to provide a better framework for governance. The Governance Committee of the board is charged with ensuring that the board is operating at its best, and making recommendations when things could work better.
In the original bylaws of the organization, terms for Class Directors (nominated and approved by the board, not community-elected seats), are set at 3 years, with a limit of 3 terms. That means that any Class Director could serve a total of 9 years on the board. This is not absolutely a problem, but we do know that board operate best when members are energetic and fully committed, and when new ideas and perspectives can be added to the mix. Nine-year terms work against both of those concepts. To solve for this, the board voted to change the bylaws and limit service to two 3-year terms, or 6 years total. A board member does have the option of taking a year off at that point and could be re-appointed after a year of downtime. We are currently updating the bylaws document to reflect this vote and will update the Association site when this work is complete.Community Elected Candidates
One other issue that has been raised by the board is preparing community-elected board members for their service on the board. This class of directors exists to provide a balance of perspective on the board, and everyone understands that many community-elected board members will likely have little board experience prior to their service. The board wants to ensure, however, that these members can jump into their term easily and figure out how to advocate for their agenda quickly. To that end, the boad agrees that it makes sense for candidates to at least have some experience with the mechanics of the Association Board. The Governence Committee recommended that a requirement of board meeting attendance would be a low-threshold to meet, and would expose candidates to how the board operates. The proposal was that, starting in the 2016 elections, candidates will need to attend a minimum of 3 board meetings, which can be tracked by Association staff.
This proposal was voted on and adopted by the board. However, I do want to note that it was not a unanimous vote; we had 2 nay votes. The point was made that currently, all board meetings are held at noon pacific on the third Wednesday of the month. That time slot is during waking hours for the US and Europe. It's early in Australia, but doable. However, anyone in Asia, in particular, can't participate in those awkward hours. The suggestion was made that we shift some of our meeting times to accomodate these other time zones if we are going to make attendance a requirement for running. There was general agreement with this sentiment, but no clear conclusion about how to actually make that happen. The board decided to call the proposal to vote now and work out the logistics of shifting board meeting schedules at a later date.Working Group Updates
Lastly, we got updates from all of the Drupal.org Working Groups: Software, Content, and Infrastructure. In addition to the work they are pursuing related to the Drupal.org roadmap, Working Groups are also reviewing their charters. With more than a year of operations under their belts, and with a full tech team on staff at the Association, it's important to take a look at how things have changed and ensure that charters are still in alignment.Goodbye 2014!
It has been a big year for the Association and the Drupal community. I want to take this opportunity to thank the Drupal community for all your support for the Association. It's a joy to come to this job every day and work together to take on the challenges and opportunities we face. Your generosity, smarts, and sense of humor makes it all that much more rewarding. I can't wait to see what we tackle together in 2015!
Flickr photo: Matt Westgate
yesterday I learned that I can go to FOSDEM in early 2015 because a conflicting event was cancelled. that makes me happy because FOSDEM is great for seeing other debian folks, & especially for meeting friends.
this posting is part of GDAC (gregoa's debian advent calendar), a project to show the bright side of debian & why it's fun for me to contribute.
Drupal 7 provides a file field that allows for uploading files and images to your Nodes but it is limited in functionality. The core file field only allows for uploading one file at a time and does not permit drag and drop functionality. Hower, with a few modules and a little bit of configuration we can easily provide this functionality to your site.
First download and install the following modules:
Packt Publishing recently sent me a copy of the eBook version of Flask Framework Cookbook by Shalabh Aggarwal. I didn’t read it in its entirety as Cookbooks don’t usually make for a very interesting linear read. I just went through it and cherry picked various recipes. But before I get into too much detail, let’s do the quick review!Quick Review
- Why I picked it up: I was asked by the publisher to read the book.
You can get this book in paperback, epub, mobi, or PDF.Book Contents
The book is split up into 12 chapters covering 258 pages with over 80 recipes.Full Review
Packt is always putting out niche Python books. Flask is one of the more popular Python micro-web frameworks, so it probably has a good sized audience. Let’s spend a few moments looking at what the chapters cover. In chapter one, we find out the many ways you can configure Flask. It contains information about using class based settings, static files, blueprints and more. Chapter two changes things up a bit with a group of recipes about Jinja, a templating language. In chapter three, we move into data modeling using SQLAlchemy. There are also recipes for Redis, Alembic and MongoDB. Chapter four is about working with views. It contains information on XHR requests, class-based views, custom 404 handlers and several other recipes.
In chapter five, the author focues on webforms with WTForms. Here we learn about field validation, uploading files and cross-site forgery. For chapter 6, we move into authentication recipes. There are recipes for the Flask-Login extension, OpenID, Facebook, Google and Twitter. Chapter 7 goes into RESTful API building. There are only four recipes in this chapter with two on creating different types of REST interfaces. The last recipe is a complete REST API example though. Chapter eight is all about the admin interface in Flask. Here you will learn about the Flask-Admin extension, custom forms, user roles and more!
Chapter nine takes us into Internationalization and localization. It has the fewest recipes at just 3. You learn how to add a new language, language switching and gettext/ngettext. Moving on to chapter ten, we learn about debugging, error handling and testing. Here we cover everything from emailing errors, to using the pdb debugger to nose, mock and coverage tests. Chapter 11 is about deployment. It covers recipes about apach, Gunicornm Tornado, Fabric, Heroku, AWS Elastic Beanstalk, Application monitoring and a few other items. Chapter 12 rounds out the book with other tips and tricks such as full-text search, working with signals, caching, Celery, etc.
Overall, I found the book fairly well written. There are some spots that are a bit choppy as I don’t believe the author is a native English speaker, but the prose doesn’t suffer very much because of this. Most of the recipes work well as standalone snippets. Sometimes the snippets don’t seem to be completely runnable, but you should be able to download the full code from Packt. I didn’t always find the groupings of recipes to be completely cohesive, but for the most part, they made sense together. I would recommend this book for a beginner in Flask that wants to take his skills to the next level and to those who need a more complete understanding of some of the things you can do with Flask.Flask Framework Cookbook
by Shalabh Aggarwal
This speech by Richard Stallman will be nontechnical, admission is free of charge, and the public is encouraged to attend.
Speech topic to be determined.
Please fill out our contact form, so that we can contact you about future events in and around New Delhi.
Chris Luckhardt is a familiar face in the North American Drupal scene. An active member of the Toronto Drupal community and a frequent attender of camps, meet-ups, conventions, and more, Chris is a Drupal master in many ways, and an expert in others.
“I’m a Drupal specialist and I do photography on the side,” Chris says of himself. “I tend to do a lot of everything, which is why I call myself a specialist — because I specialize in different elements of Drupal. My favorite areas of Drupal are site building, dev-ops, and front-end development, and I do a lot of agile and scrum project management."An Industry Veteran
Chris began his Drupal journey with Drupal 6, back in 2008. He’d already worked in web technology for a while, like with Microsoft's proprietary ASP and C#, dabbled in open source products like PHP, and worked with Linux, Apache, and mySQL.
“I knew that open source values aligned very well with my personal philosophies,” Chris said. “I’d worked on some proprietary software, and by the time I’d finished one particularly bad DotNetNuke project I decided I was going to move on in my career and go totally open source. Around that time, Drupal 6 came out and it coincided with a DrupalCamp Toronto event that was happening. It must have been Toronto's third or fourth DrupalCamp. James Walker, who had a hand in forming the Toronto Drupal group, was there, and I talked to him about what I was looking for in an open source solution to work with. He said, 'yeah this is the right option based on everything you’ve told me,’ and that was how I got started. I’ve considered him a mentor for years.
“I took the time to learn Drupal properly,” Chris continued. “I went to a few Lullabot workshops back in the day, took on some projects, and the rest is history. We all started at one point,” he said of his fellow Drupal users.
“For me, learning things the Drupal way was the biggest challenge, as opposed to coming in and doing some PHP coding. What helped me learn — and what helps me to this day — is the user group meetings. I think by far being involved in the community is the most important thing. It's the gateway — asking questions and seeing presentations is really valuable. Of course, the issue queue is the best way to self-learn, but in my opinion the best learning happens from talking to people, because someone has stumbled across your problem before."Drupal: Powered by People
When it comes to that community, Chris is concerned about how to grow it both locally and globally. “We have a very specific problem here in Toronto, but I think everyone has dealt with it too. We have a batch of old-school Toronto Drupal user group members who date back to 05-08, and we have an influx of new people. This means we have a set of introductory and beginner users — you know, people who come in like, “what’s a Drupal?” — and then we have the advanced users branching into all sorts of wild territory with Drupal.
"It’s hard to cater to both groups in one meet-up and even at our DrupalCamps that we plan every year. We recognize, if we try to cater to the introductory users we’ll turn away the advanced users, and they won’t be interested in coming out, but if we do really advanced sessions and training at our meet-ups, the new people show up and they won’t have any idea what’s going on.
“Between James and myself, we decided to address the problem by doing an introductory presentation and then a more advanced presentation during our meetups. For bigger events, it’s a little different. I created the schedule at the last DrupalCamp, and I engineered it so that there would be enough difference between overlapping session timeframes that it would work to the benefit of both the introductory and the advanced attendees…though unfortunately there's not much middle ground."Linguistic Barriers to Entry
Chris’ other observation about problems with growing Drupal is the language barrier. “I was presenting at a DrupalCamp in Kyoto, and someone raised his hand and said, 'I want to learn Drupal, but I don’t understand Views. How do I learn it?’ So I told him that there are tons of tutorials on YouTube, and he responded, 'But...they are all English.' It occurred to me that those videos show you what to do, but if you don’t understand the spoken information -- why would I click this button, why would I do that -- the vocalization aspect is incredibly important. So there’s a real serious lack of Japanese documentation for people to learn Drupal— and not just Japanese, other languages, too. There’s some work being done by the Japanese community organizers around translating some of the books, like Emma Jane and Angie’s books, so it’s a start.
“So, the biggest challenge I see with Drupal and Drupal.org is how to manage the education… And, actually, sometimes I feel bad about calling myself a Drupal master because the learning curve never stops. It only becomes less dramatic with years of experience."
To see how we plan to address some of the challenges Chris has identified, keep an eye out for conclusion to our Personas series, or look at the results of the user research we’ve performed on Drupal.org.Personal blog tags: drupal.org user researchpersona interviews
Real life has been interesting as of late; as you can see, I didn't post bug stats last week. If you have specific data from last Friday, please let me know and I will update.
The UDD bugs interface currently knows about the following release critical bugs:
- In Total:
179 bugs affecting
- Affecting Jessie:
189 (key packages:
117) That's the number we need to get down to zero
before the release. They can be split in two big categories:
- Affecting Jessie and unstable:
134 (key packages:
90) Those need someone to find a fix, or to finish the
work to upload a fix to unstable:
- 32 bugs are tagged 'patch'. (key packages: 24) Please help by reviewing the patches, and (if you are a DD) by uploading them.
- 13 bugs are marked as done, but still affect unstable. (key packages: 9) This can happen due to missing builds on some architectures, for example. Help investigate!
- 89 bugs are neither tagged patch, nor marked done. (key packages: 57) Help make a first step towards resolution!
- Affecting Jessie only: 55 (key packages: 27) Those are already fixed in unstable, but the fix still needs to migrate to Jessie. You can help by submitting unblock requests for fixed packages, by investigating why packages do not migrate, or by reviewing submitted unblock requests.
- Affecting Jessie and unstable: 134 (key packages: 90) Those need someone to find a fix, or to finish the work to upload a fix to unstable:
- Affecting Jessie: 189 (key packages: 117) That's the number we need to get down to zero before the release. They can be split in two big categories:
How do we compare to the Squeeze release cycle?Week Squeeze Wheezy Jessie 43 284 (213+71) 468 (332+136) 319 (240+79) 44 261 (201+60) 408 (265+143) 274 (224+50) 45 261 (205+56) 425 (291+134) 295 (229+66) 46 271 (200+71) 401 (258+143) 427 (313+114) 47 283 (209+74) 366 (221+145) 342 (260+82) 48 256 (177+79) 378 (230+148) 274 (189+85) 49 256 (180+76) 360 (216+155) 226 (147+79) 50 204 (148+56) 339 (195+144) ??? 51 178 (124+54) 323 (190+133) 189 (134+55) 52 115 (78+37) 289 (190+99) 1 93 (60+33) 287 (171+116) 2 82 (46+36) 271 (162+109) 3 25 (15+10) 249 (165+84) 4 14 (8+6) 244 (176+68) 5 2 (0+2) 224 (132+92) 6 release! 212 (129+83) 7 release+1 194 (128+66) 8 release+2 206 (144+62) 9 release+3 174 (105+69) 10 release+4 120 (72+48) 11 release+5 115 (74+41) 12 release+6 93 (47+46) 13 release+7 50 (24+26) 14 release+8 51 (32+19) 15 release+9 39 (32+7) 16 release+10 20 (12+8) 17 release+11 24 (19+5) 18 release+12 2 (2+0)
Code that performs well should be an assumption, not a requirement. I've never had a client ask me, “will your code make my site run slower?" Clients just assume that I'm going to deliver a codebase that does not hold things up.
Frankly, I don't monitor my retirement accounts anywhere near as closely as I should.
Who has the time?
But I do try to look at them every quarter or so, to think about how they are doing and what I should change, if anything.
So I signed on and looked at my IRA, and was reading through the various positions.
And I noticed information for a company I'd never heard of!
How did a company I'd never heard of appear in my account? Did I fumble-finger some trade, months ago? Was I hacked? The "purchase history" for the new company showed three apparently legitimate trades, dating back to 2011.
Then I did some web surfing, and arrived at: Kimberly-Clark Announces Details for Completion of Kimberly-Clark Health Care Spin-OffKimberly-Clark Corporation (NYSE: KMB) today announced the record date, distribution date and distribution ratio for the previously announced tax-free spin-off of its health care business. The spin-off will form the new publicly traded company, Halyard Health, Inc. Kimberly-Clark also increased its 2014 share repurchase program to take into account expected proceeds as a result of the spin-off.
Kimberly-Clark shareholders will receive one share of Halyard Health common stock for every eight shares of Kimberly-Clark common stock held as of the close of trading on Oct. 23, 2014, the record date for the spin-off.
And, sure enough, the dates of the three trades are precisely the three occasions on which I purchased Kimberly-Clark stock in my IRA.
The computers had quietly taken care of it all.
But, really, I ought to pay more attention to my retirement accounts.
Maybe next year, he says...
Packt Publishing recently contacted me to let me know that they’re having a $5 sale on their website for all their eBooks and Videos. Since they have a LOT of different Python and Python-related books, I thought my readers might want to know about that sale. Here’s their press release:
Following the success of last year’s festive offer, Packt Publishing will be celebrating the holiday season with an even bigger $5 offer. From Thursday 18th December, every eBook and video will be available on the publisher’s website for just $5. Customers are invited to purchase as many as they like before the offer ends on Tuesday January 6th, making it the perfect opportunity to try something new or to take your skills to the next level as 2015 begins. With all $5 products available in a range of formats and DRM-free, customers will find great value content delivered exactly how they want it across Packt’s website this Xmas and New Year.
Find out more at www.packtpub.com/packt5dollar
Even though seeing the word attic reminds me too much of leaking roofs and CVS, I've switched to using the attic backup tool.
I want a simple system which will take incremental backups, perform duplication-elimination (to avoid taking too much space), support encryption, and be fast.
I stopped using backup2l because the .tar.gz files were too annoying, and it was too slow. I started using obnam because I respect Lars and his exceptionally thorough testing-regime, but had to stop using it when things started getting "too slow".
I'll document the usage/installation in the future. For the moment the only annoyance is that it is contained in the Jessie archive, not the Wheezy one. Right now only 2/19 of my hosts are Jessie.
A lot of people are talking about the problems that are being caused by a recent change to taxation in the EU; this TechCrunch article gives a nice overview of the issues. But we thought it would be fun just to tell about our experience - for general interest, and as an example of what one UK startup had to do to implement these changes. Short version: it hasn't been fun.
If you know all about the EU VAT changes and just want to know what we did at PythonAnywhere, click here to skip the intro. Otherwise, read on...The background
"We can fight over what the taxation levels should be, but the tax system should be very, very simple and not distortionary." - Adam Davidson
The tax change is, in its most basic form, pretty simple. But some background will probably help. The following is simplified, but hopefully reasonably clear.
VAT is Value Added Tax, a tax that is charged on pretty much all purchases of anything inside the EU (basic needs like food are normally exempt). It's not dissimilar to sales or consumption tax, but the rate is quite high: in most EU countries it's something like 20%. When you buy (say) a computer, VAT is added on to the price, so a PC that the manufacturer wants EUR1,000 for might cost EUR1,200 including VAT. Prices for consumers are normally quoted with VAT included, when the seller is targetting local customers. (Companies with large numbers of international customers, like PythonAnywhere, tend to quote prices without VAT and show a note to the effect that EU customers have to pay VAT too.)
When you pay for the item, the seller takes the VAT, and they have to pay all the VAT they have collected to their local tax authority periodically. There are various controls in place to make sure this happens, that people don't pay more or less VAT than they've collected, and in general it all works out pretty simply. The net effect is that stuff is more expensive in Europe because of tax, but that's a political choice on the part of European voters (or at least their representatives).
When companies buy stuff from each other (rather than sales from companies to consumers) they pay VAT on those purchases if they're buying from a company in their own country, but they can claim that VAT back from their local tax authorities (or offset it against VAT they've collected), so it's essentially VAT-free. And when they buy from companies in other EU countries or internationally, it's VAT-free. (The actual accounting is a little more complicated, but let's not get into that.)What changed
"Taxation without representation is tyranny." - James Otis
Historically, for "digital services" -- a category that includes hosting services like PythonAnywhere, but also downloaded music, ebooks, and that kind of thing -- the rule was that the rate of VAT that was charged was the rate that prevailed in the country where the company doing the selling was based. This made a lot of sense. Companies would just need to register as VAT-collecting businesses with their local authorities, charge a single VAT rate for EU customers (apart from sales to other VAT-registered businesses in other EU countries), and pay the VAT they collected to their local tax authority. It wasn't trivially simple, but it was doable.
But, at least from the tax authorities' side, there was a problem. Different EU countries have different VAT rates. Luxembourg, for example, charges 15%, while Hungary is 27%. This, of course, meant that Hungarian companies were at a competitive disadvantage to Luxembourgeoise companies.
There's a reasonable point to be made that governments who are unhappy that their local companies are being disadvantaged by their high tax rates might want to consider whether those high tax rates are such a good idea, but (a) we're talking about governments here, so that was obviously a non-starter, and (b) a number of large companies had a strong incentive to officially base themselves in Luxembourg, even if the bulk of their business -- both their customers and their operations -- was in higher-VAT jurisdictions.
So, a decision was made that instead of basing the VAT rate for intra-EU transactions for digital services on the VAT rate for the seller, it should be based on the VAT rate for the buyer. The VAT would then be sent to the customer's country's tax authority -- though to keep things simple, each country's tax authority would set up a process where the company's local tax authority would collect all of the money from the company along with a file saying how much was meant to go to each other country, and they'd handle the distribution. (This latter thing is called, in a wonderful piece of bureaucratese, a "Mini One-Stop Shop" or MOSS, which is why "VATMOSS" has been turned into a term for the whole change.)
As these kind of things go, it wasn't too crazy a decision. But the knock-on effects, both those inherent in the idea of companies having to charge different tax rates for different people, but also those caused by the particular way the details of laws have been worked out, have been huge. What follows is what we had to change for PythonAnywhere. Let's start with the basics.Different country, different VAT rate
"The wisdom of man never yet contrived a system of taxation that would operate with perfect equality." - Andrew Jackson
Previously we had some logic that worked out if a customer was "vattable". A user in the UK, where we're based, is vattable, as is a non-business anywhere else in the EU. EU-based businesses and anyone from outside the EU were non-vattable. If a user was vattable, we charged them 20%, a number that we'd quite sensibly put into a constant called VAT_RATE in our accounting system.
What needed to change? Obviously, we have a country for each paying customer, from their credit card/PayPal details, so a first approximation of the system was simple. We created a new database table, keyed on the country, with VAT rates for each. Now, all the code that previously used the VAT_RATE constant could do a lookup into that table instead.
So now we're billing people the right amount. We also need to store the VAT rate on every invoice we generate so that we can produce the report to send to the MOSS, but there's nothing too tricky about that.
Simple, right? Not quite. Let's put aside that there's no solid source for VAT rates across the EU (the UK tax authorities recommend that people look at a PDF on an EU website that's updated irregularly and has a table of VAT rates using non-ISO country identifiers, with the countries' names in English but sorted by their name in their own language, so Austria is written "Austria" but sorted under "O" for "Österreich").
No, the first problem is in dealing with evidence.Where are you from?
"Extraordinary claims require extraordinary evidence." - Carl Sagan
How do you know which country someone is from? You'd think that for a paying customer, it would be pretty simple. Like we said a moment ago, they've provided a credit card number and an address, or a PayPal billing address, so you have a country for them. But for dealing with the tax authorities, that's just not enough. Perhaps they feared that half the population of the EU would be flocking to Luxembourgeoise banks for credit cards based there to save a few euros on their downloads.
What the official UK tax authority guidelines say regarding determining someone's location is this (and it's worth quoting in all its bureaucratic glory):
1.5 Record keeping
If the presumptions referred above don’t apply, you’ll be expected to obtain and to keep in your records 2 pieces of non-contradictory evidence from the following list to support your taxing decisions. Examples include:
- the billing address of the customer
- the Internet Protocol (IP) address of the device used by the customer
- location of the bank
- the country code of SIM card used by the customer
- the location of the customer’s fixed land line through which the service is supplied to him
- other commercially relevant information (for example, product coding information which electronically links the sale to a particular jurisdiction)
Once you have 2 pieces of non-contradictory evidence that is all you need and you don’t need to collect any further supporting evidence. This is the case even if, for example, you obtain a third piece of evidence which happens to contradict the other 2 pieces of information. You must keep VAT MOSS records for a period of 10 years from 31 December of the year during which the transaction was carried out.
For an online service paid with credit cards, like PythonAnywhere, the only "pieces of evidence" we can reasonably collect are your billing address and your IP address.
So, our nice simple checkout process had to grow another wart. When someone signs up, we have to look at their IP address. We then compare this with a GeoIP database to get a country. When we have both that and their billing address, we have to check them:
- If neither the billing address nor the IP address is in the EU, we're fine -- continue as normal.
- If either the billing address or the IP address is in the EU, then:
- If they match, we're OK.
- If they don't match, we cannot set up the subscription. There is quite literally nothing we can do, because we don't know enough about the customer to work out how much VAT to charge them.
We've set things up so that when someone is blocked due to a location mismatch like that, we show the user an apologetic page, and the PythonAnywhere staff get an email so that we can talk to the customer and try to work something out. But this is a crazy situation:
- If you're a British developer with a UK credit card, currently on a business trip to Germany, then you can't sign up for PythonAnywhere. This does little good for the free movement of goods and services across the EU.
- If you're an American visiting the UK and want to sign up for our service, you're equally out of luck. This doesn't help the EU's trade balance much.
- If you're an Italian developer who's moved to Portugal, you can't sign up for PythonAnywhere until you have a new Portuguese credit card. This makes movement of people within the EU harder.
As far as we can tell (and we've spoken to our [very expensive] accountants about this) there's no other way to implement the rules in a manner consistent with the guidelines.
And it's not all. Now we need to deal with change...Ch-ch-changes
"Time changes everything except something within us which is always surprised by change." - Thomas Hardy
VAT rates change (normally upwards). When you're only dealing with one country's VAT rate, this is pretty rare; maybe once every few years, so as long as you've put it in a constant somewhere and you're willing to update your system when it changes, you're fine. But if you're dealing with 28 countries, it becomes something you need to plan for happening pretty frequently, and it has to be stored in data somewhere.
So, we added an extra valid_from field to our table of VAT rates. Now, when we look up the VAT rate, we plug today's date into the query to make sure that we've got the right VAT rate for this country today. (If you're wondering why we didn't just set things up with a simple country-rate mapping that would be updated at the precise point when the VAT rate changed, read on.)
No big deal, right? Well, perhaps not, but it's all extra work. And of course we now need to check a PDF on display in the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying "Beware of the Leopard" to see when it changes. We're hoping a solid API for this will crop up somewhere -- though there are obviously regulatory issues there, because we're responsible for making sure we use the right rate, and we can't push that responsibility off to someone else. The API would need to be provided by someone we were 100% sure would keep it updated at least as well as we would ourselves -- so, for example, a volunteer-led project would be unlikely to be enough.
So what went wrong next? Let's talk about subscription payments.Reconciling ourselves to PayPal
"Nothing changes like changes, because nothing changes but the changes." - Gary Busey
Like many subscription services, at PythonAnywhere we use external companies to manage our billing. The awesome Stripe handle our credit card payments, and we also support PayPal. This means that we don't need to store credit card details in our own databases (a regulatory nightmare) or do any of the horrible integration work with legacy credit card processing systems.
So, when you sign up for a paid PythonAnywhere account, we set up a subscription on either Stripe or PayPal. The subscription is for a specific amount to be billed each month; on Stripe it's a "gross amount" (that is, including tax) while PayPal split it out into separate "net amount" and a "tax amount". But the common thing between them is that they don't do any tax calculations themselves. As international companies having to deal with billing all over the world, this makes sense, especially given that historically, say, a UK company might have been billing through their Luxembourg subsidiary to get the lower VAT rate, so there's no safe assumptions that the billing companies could make on their customers' behalf.
Now, the per-country date-based VAT rate lookup into the database table that we'd done earlier meant that when a customer signed up, we'd set up a subscription on PayPal/Stripe with the right VAT amount for the time when the subscription was created. But if and when the VAT rate changed in the future, it would be wrong. Our code would make sure that we were sending out billing reminders for the right amount, but it wouldn't fix the subscriptions on PayPal or Stripe.
What all that means is that when a VAT rate is going to change, we need to go through all of our customers in the affected country, identify if they were using PayPal or Stripe, then tell the relevant payment processor that the amount we're charging them needs to change.
This is tricky enough as it is. What makes it even worse is an oddity with PayPal billing. You cannot update the billing amount for a customer in the 72 hours before a payment is due to be made. So, for example, let's imagine that the VAT rate for Malta is going to change from 18% to 20% on 1 May. At least three days before, you need to go through and update the billing amounts on subscriptions for all Maltese customers whose next billing date is after 1 May. And then sometime after you have to go through all of the other Maltese customers and update them too.
Keeping all of this straight is a nightmare, especially when you factor in things like the fact that (again due to PayPal) we actually charge people one day after their billing date, and users can be "delinquent" -- that is, their billing date was several days ago but due to (for example) a credit card problem we've not been able to charge them yet (but we expect to be able to do so soon). And so on.
The solution we came up with was to write a reconciliation script. Instead of having to remember "OK, so Malta's going from 18% to 20% on 1 May so we need to run script A four days before, then update the VAT rate table at midnight on 1 May, then run script B four days after", and then repeat that across all 28 countries forever, we wanted something that would run regularly and would just make everything work, by checking when people are next going to be billed and what the VAT rate is on that date, then checking what PayPal and Stripe plan to bill them, and adjusting them if appropriate.
A code sample is worth a thousand words, so here's the algorithm we use for that. It's run once for every EU user (whose subscription details are in the subscription parameter). The get_next_charge_date_and_gross_amount and update_vat_amount are functions passed in as a dependency injection so that we can use the same algorithm for both PayPal and Stripe.def reconcile_subscription( subscription, get_next_charge_date_and_gross_amount, update_vat_amount ): next_invoice_date = subscription.user.get_profile().next_invoice_date() next_charge_date, next_charge_gross_amount = get_next_charge_date_and_gross_amount(subscription) if next_invoice_date < datetime.now(): # Cancelled subscription return if next_charge_date < next_invoice_date: # We're between invoicing and billing -- don't try to do anything return expected_next_invoice_vat_rate = subscription.user.get_profile().billing_vat_rate_as_of(next_invoice_date) expected_next_charge_vat_amount = subscription.user.get_profile().billing_net_amount * expected_next_invoice_vat_rate if next_charge_gross_amount == expected_next_charge_vat_amount + subscription.user.get_profile().billing_net_amount: # User has correct billing set up on payment processor return # Needs an update update_vat_amount(subscription, expected_next_charge_vat_amount)
We're pretty sure this works. It's passed every test we've thrown at it. And for the next few days we'll be running it in our live environment in "nerfed" mode, where it will just print out what it's going to do rather than actually updating anything on PayPal or Stripe. Then we'll run it manually for the first few weeks, before finally scheduling it as a cron job to run once a day. And then we'll hopefully be in a situation where when we hear about a VAT rate change we just need to update our database with a new row with the appropriate country, valid from date, and rate, and it will All Just Work.
(An aside: this probably all comes off as a bit of a whine against PayPal. And it is. But they do have positive aspects too. Lots of customers prefer to have their PythonAnywhere accounts billed via PayPal for the extra level of control it gives them -- about 50% of new subscriptions we get use it. And the chargeback model for fraudulent use under PayPal is much better -- even Stripe can't isolate you from the crazy-high chargeback fees that banks impose on companies when a cardholder claims that a charge was fraudulent.)In conclusion
"You can't have a rigid view that all new taxes are evil." - Bill Gates
The changes in the EU VAT legislation came from the not-completely-unreasonable attempt by the various governments to stop companies from setting up businesses in low-VAT countries for the sole purpose of offering lower prices to their customers, despite having their operations on higher-VAT countries.
But the administrative load placed on small companies (including but not limited to tech startups) is large, it makes billing systems complex and fragile, and it imposes restrictions on sales that are likely to reduce trade. We've seen various government-sourced estimates of the cost of these regulations on businesses floating around, and they all seem incredibly low.
At PythonAnywhere we have a group of talented coders, and it still took over a week out of our development which we could have spent working on stuff our customers wanted. Other startups will be in the same position; it's an irritation but not the end of the world.
For small businesses without deep tech talent, we dread to think what will happen.
So, Sony caved in (according to Rob Lowe) and demonstrated that America lost its first cyberwar (according to Newt Gingrich). It should not surprise anyone, after the whistle blower Edward Snowden documented that the government of USA and their allies for many years have done their best to make sure the technology used by its citizens is filled with security holes allowing the secret services to spy on its own population. No one in their right minds could believe that the ability to snoop on the people all over the globe could only be used by the personnel authorized to do so by the president of the United States of America. If the capabilities are there, they will be used by friend and foe alike, and now they are being used to bring Sony on its knees.
I doubt it will a lesson learned, and expect USA to lose its next cyber war too, given how eager the western intelligence communities (and probably the non-western too, but it is less in the news) seem to be to continue its current dragnet surveillance practice.
There is a reason why China and others are trying to move away from Windows to Linux and other alternatives, and it is not to avoid sending its hard earned dollars to Cayman Islands (or whatever tax haven Microsoft is using these days to collect the majority of its income. :)