FLOSS Project Planets

So you're launching a new website or replacing an old one and want to make sure everything goes smoothly? This guide will give you a run-down of everything you can check to avoid common pitfalls!

Site status

You should always start by checking the status report (http://example.com/admin/reports/status). This page shows you all of the basic requirements for your Drupal site to run correctly.

Any issues will be highlighted in red and typically have a link to a configuration page or the documentation to help you resolve the problem.

Scheduled tasks

Drupal 7 will run scheduled tasks (known as cron jobs) out of the box, but only when users are visiting your site.

This is great for small sites which don't need much housekeeping, but if your site is a bit bigger or if you don't have visitors 24/7, you should set up a cron job to run periodically.

You can also look at a module such as Ultimate cron which gives you fine grained control over when each scheduled task will run.

Web services

Many web services such as Mollom or Google analytics need a domain name specific API key to use.

If you use any of these services on your website, you should ensure that you've registered your real domain name with the service and you've updated Drupal with your new API key.

Broken links

It can be easy when copying and pasting to accidentally link directly to a file or image on a development site.

These links can often stop working or perhaps worse, may direct users away from your live site and onto the development site instead.

You can use a module such as Link checker to ensure this doesn't happen, and it is good practice to password protect your development website, so that users (or more likely Google!) cannot stumble accross it.

Site optimisation

The site performance page (http://example.com/admin/config/development/performance) will allow you to configure a number of options to help optimise your Drupal site. This includes page caching and optimising CSS and JavaScript files.

See our series on High Performance in Drupal for some expert tips - High performance in Drupal Part 1: Give your site a boost and High performance in Drupal Part 2: Lightning fast code.

Development modules

Development modules such as devel can often reduce your website's performance, so it's worth turning them off on your live site. You can still keep them running in your development environment if needed.

User accounts

Many of us are guilty of using a common or simple password to make life easier when building a website.

Once the site is live, it's worth taking the time to update any administrative accounts with secure passwords.

It is also worth removing any unnecessary development accounts and content. Just in case.

Error messages

Being able to see debug messages and errors are handy when creating a site, but may scare off users once the site is live.

Ensure that errors and warnings are hidden by visiting the Logging and errors config page.

Site information

The site information config page holds all of the most common site information, such as the website name and email address.

It's worth double checking that all of this information is correct. It could be quite embarrassing if your first newsletter arrives from dev@example.com.

 

Some of these pitfalls can be avoided from the get go, if you follow a few simple principles. Check out Rick Donohoe's blog article Drupal site building 101 for some handy hints and tips on this!

Moving Drupal website clients to cloud hosting has been great as they're able to get high performance, scalable capacity at a pretty reasonable rate. However, we have discovered when clients offer an email sign-up, the emails that are generated from the cloud-hosted Drupal website are often rejected as spam. For those clients who have chosen the Rackspace cloud, here is a step-by-step solution to the problem.... Read more

He shows us three kinds of (more or less) open source projects.

Type 1: public source

Once upon a time there was an "open source project" called the facebook SDK. Basically it just stopped working one day and nobody could help, despite offers for help on the issue tracker. Hacker news got wind of it and it was on the front page for a while. Facebook's reaction? Disabling the issue tracker... (Later on they fixed it).

That's not open source, that's public source. Often it is abandoned due to loack of interest, change of focus or so. The motivation for having it as open source simply is not clear.

Type 2: shared investment

A different example: gittip. They aim to be the world's first open company. There's a github issue for everything, even the company name. Major decisions are voted for on github. The code is open source, of course. All interviews with journalists are filmed and live-streamed. And all otherwise-often-backdoor-cooperation-agreements are fully open.

Projects like gittip are shared investment projects. Shared ownership, extreme transparency. There is very little questioning of motivations. The motivation is clear and public. There's a documented process for new contributers. The advantage? It is low risk. There's a high bus factor.

Type 3: dictatorship project

Kenneth is the author of requests. An open source project, very succesful. But all the decisions are made by Kenneth.

That's really more of an dictatorship project. A totalitarian BDFL that owns everything. The dictator is responsible for all decisions. Requests' values lie in its extreme opinions. If he'd involve more people, the value would be dilluted. There are drawbacks. A low bus factor. High risk of burnout: Kenneth is the single point of failure.

Lessons learned

• Be cordial or be on your way. As a user, you need to keep all your interactions with the maintainer as respectful as possible. The maintainer put a lot of work in it and they don't owe you any of their time.

  As a maintainer, you also must be cordial. Be thankful to all contributions. Feedback is the liveblood of your project, even the negative. You'll need to ignore non-constructive comments. Be careful with the words you choose, sometimes contributors take what you say VERY personally. You might have to educate your users. And: a bit of kindness goes a long way.

• Sustainability is almost the biggest challenge. Don't burn out. Try to get others to help.

  He quotes Wes Beary: "open source provides a unique opportunity for the trifecta of purpose, mastery and autonomy". Pay equal attention to all of these three. Learn to do less, focus more on your purpose, for instance.

• Learn to say no. People ask for crazy features. Or they submit quite sane pull requests that, if you allow them all in, makes your project slow and unfocused. Kenneth wants as few lines of codes in his project. Negative diffs are the best diffs!

• Open source makes the world a better place. Don't make it complicated!

Metaclasses are a handy feature of Python and Django makes good use of them.

When you create certain kinds of classes in Django, a metaclass will do something to the class before it is created. For forms, the various attributes of the class are converted into a base_fields dictionary on the class.

Similarly, a subclass of Model also fires up a metaclass that does some registering. A foreignkey to another model adds a relation back on that other model, for instance.

As a recap, a class is something that can be instantiated into an object. It can have an __init__() method that does something upon instantiation. type(your_instance) will return the class.

Did you know that you can create classes dynamically? See for yourself:

>>> name = 'ExampleClass' >>> bases = (object,) >>> attrs = {'__init__': lambda self: print('Hello from __init__')} >>> ExampleClass = type(name, bases, attrs) >>> example = ExampleClass() Hello From __init__ >>> type(example) <class '__console__.ExampleClass'>

So... we can actually control how classes are created! You could create a create_class() method that calls type but that modifies, for instance, the name. Or we could take all the attributes and add them to a base_fields dictionary on the instance. Hey, that's what we saw in the first Django form example!

Now, what is type exactly? It is a class that creates classes.

This also means we can subclass it! The most useful thing to override in our subclass is the __new__() method. The __init__() method creates instances from the class, the __new__() creates classes. So again we can modify the name and/or the attributes.

How do you use it in practice? Normally you'd set a __metaclass__ attribute on a class. This tells python to use that metaclass for creating the class. The same for subclasses. This is how our Django form classes use the metaclass specified in Django's base Form class.

Django uses metaclasses in five places: admin media, models, forms, formfields, form widgets. Grep for metaclass in your local django source code once to get a better feel for how Django uses it.

Note on python 3: it uses a slightly different syntax for specifying metaclasses. So Django 1.5 uses six to support both ways in a single codebase.

Warning: don't overuse metaclasses. They can make code difficult to debug and follow. Use Django as a good example of how to use metaclasses. Django saves you a lot of work by using metaclasses in a few locations.

See https://github.com/inglesp/Metaclasses

Nice way of giving a presentation, btw. Some sort of semi-interactive python prompt. The software is online at https://github.com/inglesp/prescons

This post discusses the differences between Drupal and WordPress, and also takes a quick look at a couple of other web frameworks. What are the benefits with each platform, and how do you know which one to choose?

Amjith Ramanujam recently wrote a thread profiler in Python and it was rediculously simple. He works for New Relic, which is all about performance and expecially performance measuring.

Python comes with batteries included, so it also includes a C profiler that works pretty well. But it doesn't work nice for django because the output is so huge. If you use the GUI RunSnakeRun, it is more managable that way.

Additional problem with cProfile: it has about 100% overhead, so you can't run it in production (which they need).

You can do more targeted profiling. For instance in Django. The way a web framework processes requests is normally always in the same way. You can use that during profiling.

There are two important stages: interrupt and inquire.

Interrupt

A statistical profiler looks how often a function is called and by who. For this it needs to interrupt the regular process. You could set an OS-level signal to call your profiler every x miliseconds so that it can do something. It only works in linux, btw. Another way is to create a python background thread that wakes up every x miliseconds. It is cross-platform and mod_wsgi compatible. It is less accurate for CPU tasks and it cannot interrupt C extensions.

Inquire

You can use sys._current_frames() to get yourself the frames ("stack trace") from the current thread. Here you can extract the filename, line number and so of the most interesting frames. He's building a "call tree" structure, mapping how often a function is called, preserving the parent/child relation between functions. The tricky part was visualizing it :-) In the end they did it with d3.js.

There's some 3% overhead in the thread approach, so that's real good. They can switch it on for a type of request and it'll profile the next 100 requests of that type.

Brandon Rhodes mentions Nicolai Kopernik, a famous Polish scientist. He "lifted Earth into Heaven" by his book where he put the sun in the center of our universe instead of Earth. We're no longer at the bottom.

The near-earth environment was pretty well mapped out. 300BC, the size of the spherical earth was already known. Around 100BC the distance to the moon was known.

But what about those planets? They were harder. Stars did move around the sky more or less linearly. But those planets. They seemed to move back and forth a bit. Did they need to have a different model? The sun-centric model was already known, but it didn't catch on. The reason? Medieval science was too emperic: the earth cannot be moving, it seems to stay in place. Throw a ball and it falls down again towards the earth, it doesn't career off into space. The church wasn't totally idiotic by later convicting Galileo: there was just no solid emperical evidence :-)

One of the pieces of evidence that was missing was that there was no observable stellar parralax; visible movement between stars because of earth movement. It was only in 1838 that the instruments were good enough to actually observe parralax. Only then did we figure out how far away the stars were. So it took 200-300 year to get real evidence.

So why did Kopernik trust his theory despite the lack of evidence? Because of beauty. Brandon showed Kopernik's theory as Python code; an algorithm for planet movements. And showed his theory as a code refactoring. The end result was quite simple and nice once you factored out Earth's motion. The code looks nicely ordered.

There's truth in beauty.

Brandon proposes a new term: Kopernican refactoring: making a system simpler by moving something new into the center.

It is so easy to behave according to a rule that's not even really there. You look at a problem in a certain way and you're stuck. Once you put something else in the center, you look at the problem in a different way.

For instance, you can use argparse to generate a help message when you pass --help. You can also do it the other way around: using docopt which generates a parser from your help message! You get a much nicer help message.

So if you're stuck, think of Copernicus and his refactoring.

His goal: teach programmers to be more creative.

He's got a love/hate relationship with creativity. The first part of his talk was impossible to summarize. You'll have to watch the video later on :-)

• Artists tell him he's not artistic because he works on developing technical skills.
• Guitarists tell him he's not a real guitarist as he doesn't play in a band. And 'cause he builds his own guitars he's a programmer, not a Real Guitarist.
• Writers tell him he's not a writer because he writes technical books.
• Programmers tell him he's not a programmer because he doesn't work on their project. And by the way, he's a (technical) writer now, so he's not a programmer.

He's not creative enough. Or so the others say. Or he's not acceptable. How to deal with creativity? In a way, you can re-phrase creativity. Programmers are always making something from nothing, right? Isn't that the pinnacle of creativity?

Here are four hypothetical persons:

• Technique, no imagination: a stereotypical programmer.
• Imagination, no technique: stereotypical biz dude.
• No imagination, no technique. Probably doesn't exist.
• Both imagination and technique. Zed's goal.

Zed's imaginative programmer process. Everyone has a process (if they're good), here's the one he proposes to help you be more creative:

• You start with an idea.
• You establish a concept that helps form the idea. It gives your idea clothes.
• Research techniques or tools. Do some research or you'll pay for it later on.
• Refine the concept through composition. So put a box around the vast world of available possibilities. Just mark which features are inside and outside the concept.
• Explore through prototypes. Throw away code or use paper prototypes for instance. This saves you so much time later.
• You make it real.

We are programmers, so we should iterate this process.

He tried this process with http://projectzorn.com/, going through all the stages. And he's probably going to work on it

He started with a little history lesson. The sea battle of mobile bay. The admiral (Faragut) ordered the ships straight through the minefield (called "torpedoes" at the time). "Damn the torpedoes, full speed ahead". And it worked.

What does this to have to do with Django? Well, "damn the torpedoes, full speed ahead" feels a bit like how rapid prototyping feels afterwards. He's often involved with hackathons. Lot of quick coding in limited time with a lot of people. He learned a lot about his tools that way (and he often used Django).

There's a time to make a distinction between production and prototype. Sometimes it is better to just try something with a prototype. Throw-away code.

Aaargh! Throw-away code?!? We never throw code away. But it is something we must learn. It is good to let go once in a while. Let your code go. It isn't yourself, it is just some code.

The danger is that prototype code is put into action as production code. With some work, this danger can be prevented.

What about Django? Django is the best for prototyping. For rapid prototyping, Django is better than micro-frameworks like Flask that might seem better at first glance. Here are some reasons:

• Django was build for rapid prototyping. It originated at a newspaper! 24 hours to build something.

• It is flexible. It was build to bend. He can prepare something for the other people programmign with him and get them going and still keep the code in reasonable shape.

• Us. The strength of Django is the community that supports it. Stack overflow questions and answers. The django websites. Books like two scoops of Django (see my review). That's not something you have with many other frameworks!

  Tip: read especially chapter 2 and 3 of the two scoops book.

  One thing he'd add to the book is stuff like fabfiles and makefiles. Handy for rapid prototyping.

Use stuff that's available. For instance Django's staticfiles app for grabbing together all the css/js/png. Whether it is in one directory or split out over multiple apps. It also helps with production.

Also look at brunch for setting up your javascript app's structure. It works well with Django.

Deployment: you need to show your prototype. Heroku is very quick for prototypes. (He mentioned that they have a data center in Europe now in case you need it).

Deployment: use chef. Lots of recipes. You could also use Salt if you're more into Python. Also lots of stuff available. Both take a while to learn, but it is a very good investment.

Configuration management is an extremely useful skill. Do it well.

Tastypie is the quickest way to get a REST api out of your Django. It is the best for rapid prototyping. Another good one is django-rest-framework. It will take a little bit longer to set up, but once done you're working with actual Django views. And django-rest-framework's browseable API is very helpful when you're working with a couple of others

Social auth connectors: everyone makes one and there are way too many half-working ones. He's got two that he can recommend. django-social-auth is very complete. The other is django-allauth for when speed is important for you.

If you don't want to play fair to others during a hackathon: use celery. It is very unfair to use celery, python and Django. The combination with Django is pretty OK to set up. You can do a lot what others cannot do easily. So use it for rapid prototyping. (Regarding setting it up: there are good chef recipes for it).

TEST. Yes, even during a hackathon. He doesn't advocate full test driven development. It is a balance. But the errors that kill you during a hackathon are the errors you make twice. So, for instance, test that all your views simply return a 200 Ok. This already helps prevent a lot of problems.

Look at AngularJS. Even if you don't use the framework itself. Why? It has a great javascript test runner. Good for testing while rapid prototyping.

I am excited to announce that my book, Write Your First Program is finally available for public consumption. In this day and age, when there are easily accessible programming books galore (in the context of both, price and availability), why write a new book? Through this post, I hope to reach out to the prospective readers and […]

We are pleased to announce that Commerce Kickstart has won the Walkthrough documentation prize. The prize, which was determined by votes from Walkthrough.it backers, will use the Commerce Kickstart Drupal distribution to showcase the capabilities of Walkthrough.it.

Commerce Kickstart is the quickest way to get up and running with Drupal Commerce. The distribution provides everything to create a fully-featured demo store out of the box, complete with theme, catalog, and custom back office interface.

The Linux init systems are a bit in flux at the moment. That is, they're in flux in Debian; outside Debian, most other distributions have stepped away from sysvinit and towards something else (systemd, openrc, or upstart). I've not been a proponent of any switch, though I understand the reasoning, and it probably makes sense for us to switch at some point. But yesterday, the fact that this customer's system was running sysvinit and not systemd or upstart saved me quite a bit.

There's a server. It has one quadcore processor. For reasons that I won't go into here, the customer wants an extra quadcore processor to be added to the system.

After having done so, I power on the system... only to see it power itself off at some point during boot. I did notice some kernel messages fly by just moments before the system would power itself off, but it was impossible for me to read them. So what did I do?

• Boot the system with init=/bin/bash,
• After having booted the system, go to /etc/rcS.d and manually run each and every one of the scripts there in turn. When the system powers off, I know what the problem is.
• Disable the init script that causes the problem, and boot the system normally.

That last bit is, obviously, a bit of an ugly workaround; the better way to fix this issue would have been to debug what the actual issue was, and implement a proper fix. However, I didn't have time for that (the fact that there was need for a second quadcore chip explains how much this system is in use), and the workaround was acceptable for the customer. It is not the first time that this ability to single-step the init system has saved me. The fact that sysvinit is so simplistic is what makes this possible, and I consider that one of its most important features.

Recently, I came into contact with a distribution that uses systemd as its init system (in casu, Arch Linux). I had made a mistake in configuration; I had installed and enabled a graphical login system, but had no xterm or similar available, and had done something else wrong through which I couldn't get a regular shell on the console anymore, either. To fix this, I tried doing something like the above (running with init=/bin/bash and single-stepping the init system), but found that doing so with systemd is nigh impossible. In the end, I knew what exactly the problem was and could disable automatically starting the login manager through removing a symlink, but it brought home the issue that debugging a similar issue when running systemd rather than sysvinit might be a lot harder to do.

We'll see what the future brings.

Last year at DebConf12 and the Paris mini-DebConf I mentioned some of the sophisticated techniques that the likes of Microsoft and Facebook are using to monitor their customers.

So when Skype was busted spying on the content of chat messages, it was no surprise for many people in the Debian community.

People are already rushing to find alternatives like XMPP and Jitsi. Debian 7 has been released just in time, with powerful features like TURN support that finally allow users to make free calls and chats with seamless NAT traversal. Sadly, Debian's built-in VoIP/RTC client, Empathy, only uses Google's TURN servers and not native Debian servers, but hopefully a solution will come soon, but it is easy enough to install Jitsi instead and configure it to use any of the free TURN server software on Debian.

It should be emphasized that Skype does not just spy on URLs in chat - it has simply been possible to detect this form of spying by detecting when the URL is accessed. Microsoft has taken out various patents for secretive monitoring of Internet phone calls and the analysis of speech patterns to detect both the content and emotions during a conversation. This allows them to get a very thorough analysis of the state of mind of every user at almost every moment and fine-tune the type of advertising and branding that is delivered to that person through conventional means and also through biased `news' reporting and other means.

Python offers many different template engines for web application development to turn your view logic to HTML code on the server and then send the resulting HTML code to a web browser. When you are dealing with complex page templates, especially when using third party libraries, it often gets confusing what template context variables are available and what they have eaten.  In this case, the best method of debugging is to insert a breakpoint into your page template and inspect the variables in runtime context. This very useful “advanced” debugging method is not known to many people, especially if they come from the background where command-line debugging tools have not been the norm.

Table Of Content

1. Python debugging

2. Setting a breakpoint in Django templates

3. Other templating engines

1. Python debugging

Python offers pdb command-line debugger out of the box. I recommend to use more advanced version ipdb, which comes with proper command history, colored tracebacks and tab completion / arrow key support (though there seems to be an issue using ipdb with multithreaded / autoreloading programs). There also exist more GUIful, but still terminal based, pudb. Eclipse + PyDev offer remote debugging support with full GUI.

Read some command-line debugging tutorials if you are not familiar with the concept. Especially how to inspect local variables with commands of locals(), dir(object), for i in dict.items(): print i come to hand.

2. Setting a breakpoint in Django templates

Here is an example how to create a breakpoint tag for Django templates and then go around and debug your templates. In the example, I debug the field rendering look of django-crispy-forms.

First we need to create a custom Django template tag invoking pdb, because you cannot run arbitrary Python snippets directly from Django templates. The code here is based on the example on djangosnippets.org.

templatetags/debug.py

# -*- coding: utf-8 -*- # # http://djangosnippets.org/snippets/1550/ # import pdb as pdb_module from django.template import Library, Node register = Library() class PdbNode(Node):     def render(self, context):         pdb_module.set_trace()         return '' @register.tag def pdb(parser, token):     return PdbNode()

Then we use this tag our template:

{% load crispy_forms_field %} {% load debug %} {% if field.is_hidden %}     {{ field }} {% else %}     <div>         <div>             {{ field.long_help }}             {% pdb %}         </div>     </div> {% endif %}

We run Django normally with manage.py runserver and encounter this breakpoint when rendering a template.

Now we can go around and see what variables are avaialble in the template and what they have eaten.

(Pdb) locals().keys() ['self', 'context'] (Pdb) context.__class__ <class 'django.template.context.RequestContext'>

Ok, so we have available template variables in a local variable called context (makes sense, we are now in our templatetag.py code).

(Pdb) for i in dir(context): print i ... _reset_dicts autoescape current_app dicts get has_key new pop push render_context update use_l10n use_tz

There seems to be a way to access all template variables:

(Pdb) for d in context.dicts: print d.keys() [] ['csrf_token'] ['request'] ['perms', 'user'] ['debug', 'sql_queries'] ['LANGUAGES', 'LANGUAGE_BIDI', 'LANGUAGE_CODE'] ['MEDIA_URL'] ['STATIC_URL'] ['TIME_ZONE'] ['messages'] ['downtime'] ['block'] ['flat_attrs', 'inputs', 'csrf_token', 'form_id', 'form_error_title', 'form_action', 'error_text_inline', 'html5_required', 'help_text_inline', 'formset_error_title', 'form_style', 'form_method', 'form_show_errors', 'is_formset', 'form_tag', 'attrs', 'form_attrs', 'form_class']

Now we can see what come through to our template as the field and widget in the field rendering loop.

(Pdb) context["field"] <django.forms.forms.BoundField object at 0x1038ae590> (Pdb) for i in context["field"].__dict__.items(): print i ('html_initial_name', u'initial-ad-xxx_type') ('form', <xxx.Form object at 0x103064c10>) ('html_name', 'ad-xxx_type') ('html_initial_id', u'initial-ad-id_ad-xxx_type') ('label', u'Foobar') ('field', <django.forms.fields.TypedChoiceField object at 0x103062e50>) ('help_text', '') ('name', 'xxx_type')

And after you see this, you can figure out if your data is coming through the templating stack and is it coming through correctly and what you need to do in order to bend it to your will.

3. Other templating engines

See also an example for Chameleon templates. Unfortunately Plone, out of the box, cannot support this debugging method due to the security.

 

 

 

 Subscribe to this blog in a reader Follow me on Twitter Follow me on Facebook Follow me Google+

Field as Block is a lightweight module that allows you to display a field as a block. The same results can be achieved by using Panels, Display Suite or custom code but this module offers a lightweight alternative.

CCK Blocks offers similar functionality, however on the project page they recommend that you use Field as Block for new projects. It looks like CCK Blocks will be deprecated in favour of Field as Block. For more details read issue #1920636 (comment #4).

Today Google announced PHP support for Google App Engine! I have been one of the lucky folks who had early access and so of course I worked on getting Drupal up and running on GAE. There are a few things that still need to be worked out which I will continue to discuss with the app engine team, but I have a working Drupal setup which I will detail below. Note that much of this may also apply to other PHP frameworks.

Getting up and running

I will cover the steps specific to getting Drupal 7 (notes for Drupal 6 along with branches in repository) up and running on App Engine and not how to use the SDK and development flow which is detailed in the documentation. For an example (minimal profile from core) of Drupal running on Google App Engine see boombatower-drupal.appspot.com.

Sign up to be whitelisted for PHP runtime

Currently, the PHP runtime requires you to sign up specifically for access. Assuming you have access you should be able to follow along with the steps below. Otherwise, the following steps will give you a feel for what it takes to get Drupal running on GAE.

Create an app

Create app by visiting appengine.google.com and clicking Create Application, see the documentation for more details.

Create a Cloud SQL Instance

Follow the documentation for setting up a Cloud SQL Instance. Be sure to give your application access to the instance.

Once the instance has been created select the SQL Prompt tab and create a database for your Drupal site as follows.

CREATE DATABASE drupal;

Download Drupal

There are a few tweaks that need to be made to get Drupal to run properly on GAE which are explained below, but for the purposes of this walk-through one can simply download my branch containing all the changes from github.

git clone --branch 7.x-appengine https://github.com/boombatower/drupal-appengine.git   # or for Drupal 6 git clone --branch 6.x-appengine https://github.com/boombatower/drupal-appengine.git

or download as a zip or for Drupal 6 download as a zip.

Configure Drupal database settings

Since GAE does not allow the filesystem to be writeable one must configure the database settings ahead of time.

Copy default.settings.php as settings.php and add the following bellow <?php $databases = array(); ?> around line 213.

<?php
$databases = array();
$databases['default']['default'] = array(
  'driver' => 'mysql',
  'database' => 'drupal', // The database created above (example used 'drupal').
  'username' => 'root',
  'password' => '',
  // Setting the 'host' key will use a TCP connection which is not supported by GAE.
  // The name of the instance created above (ex. boombatower-drupal:drupal).
  'unix_socket' => '/cloudsql/[INSTANCE]',
//  'unix_socket' => '/cloudsql/boombatower-drupal:drupal',
  'prefix' => '',
);
?>

For Drupal 6 around line 91.

<?php
$db_url = 'mysql://root:@cloudsql__boombatower-drupal___drupal/drupal';
?> Push to App Engine

Update the application name in the app.yaml file to the one you created above and upload by following the documentation.

# See https://developers.google.com/appengine/docs/php/config/appconfig.   application: drupal # <-- change this to your application version: 1 runtime: php api_version: 1 threadsafe: true   handlers: # Default handler for requests (wrapper which will forward to index.php). - url: / script: wrapper.php   # Handle static requests. - url: /(.*\.(ico$|jpg$|png$|gif$|htm$|html$|css$|js$)) # Location from which to serve static files. static_files: \1 # Upload static files for static serving. upload: (.*\.(ico$|jpg$|png$|gif$|htm$|html$|css$|js$)) # Ensures that a copy of the static files is left for Drupal during runtime. application_readable: true   # Catch all unhandled requests and pass to wrapper.php which will simulate # mod_rewrite by forwarding the requests to index.php?q=... - url: /(.+) script: wrapper.php appcfg.py update drupal/ Install

Visit your-app.appspot.com/install.php and follow the installation steps just as you would normally except that the database information will already be filled in. Go ahead and ignore the mbstring warning and note that the GAE team is looking into supporting mbstring.

Explanation of changes

If you are interested in what changes/additions were made and the reasons for them continue reading, otherwise you should have a working Drupal install ready to explore! There are a few basic things that do not work perfectly out of the box on GAE. The changes can be seen by diffing the 7.x-appengine branch against the 7.x branch in my repository.

File directory during installation

The Drupal installer requires that the files directory be writeable, but GAE does not allow for local write access thus the requirement must be bypassed in order for the installation to complete.

Author: boombatower <boombatower@google.com> Date: Wed May 15 15:49:03 2013 -0700   Hack to trick Drupal into ignoring that file directory is not writable.   diff --git a/modules/system/system.install b/modules/system/system.install index 1b037b8..9931aad 100644 --- a/modules/system/system.install +++ b/modules/system/system.install @@ -333,6 +333,8 @@ function system_requirements($phase) { } $is_writable = is_writable($directory); $is_directory = is_dir($directory); + // Force Drupal to think the directories are writable during installation. + $is_writable = $is_directory = TRUE; if (!$is_writable || !$is_directory) { $description = ''; $requirements['file system']['value'] = $t('Not writable'); Clean URLs

In order to take advantage of clean urls, of which most sites take advantage, mod_rewrite is required for Apache environments. Since GAE does not use Apache it does not support mod_rewrite and thus another solution is needed. The app.yaml can configure handlers which allow for wildcard matching which means multiple paths can easily be routed to a single script. Taking that one step further we can alter the <?php $_GET['q']?> variable just as mod_rewrite would so that Drupal functions properly. Rather than modify core this can be done via a wrapper script as show below (this should work well for other PHP applications).

<?php
/**
 * @file
 * Provide mod_rewrite like functionality and correct $_SERVER['SCRIPT_NAME'].
 *
 * Pass through requests for root php files and forward all other requests to
 * index.php with $_GET['q'] equal to path. In terms of how the requests will
 * seem please see the following examples.
 *
 * - /install.php: install.php
 * - /update.php?op=info: update.php?op=info
 * - /foo/bar: index.php?q=/foo/bar
 * - /: index.php?q=/
 */

$path = parse_url($_SERVER['REQUEST_URI'], PHP_URL_PATH);

// Provide mod_rewrite like functionality. If a php file in the root directory
// is explicitely requested then load the file, otherwise load index.php and
// set get variable 'q' to $_SERVER['REQUEST_URI'].
if (dirname($path) == '/' && pathinfo($path, PATHINFO_EXTENSION) == 'php') {
  $file = pathinfo($path, PATHINFO_BASENAME);
}
else {
  $file = 'index.php';

  // Provide mod_rewrite like functionality by using the path which excludes
  // any other part of the request query (ie. ignores ?foo=bar).
  $_GET['q'] = $path;
}

// Override the script name to simulate the behavior without wrapper.php.
// Ensure that $_SERVER['SCRIPT_NAME'] always begins with a / to be consistent
// with HTTP request and the value that is normally provided (not what GAE
// currently provides).
$_SERVER['SCRIPT_NAME'] = '/' . $file;
require $file;
?> PHP $_SERVER['SCRIPT_NAME'] variable

The <?php $_SERVER['SCRIPT_NAME'] ?> implementation differs from Apache mod_php implementation which can cause issues with a variety of PHP applications. The variable matches the HTTP spec and not the filesystem when called through Apache.

For example a script named foo.php contains the following.

<?php
var_dump($_SERVER['SCRIPT_NAME']);
?>

When executed from command line here are the results.

$ php foo.php string(7) "foo.php"   $ php ./foo.php string(9) "./foo.php"

When invoked through Apache like http://example.com/foo.php.

string(8) "/foo.php"

The documentation does not talk about this behavior (although many comments demonstrated the expected Apache behavior), but it is definitely depended on.

The difference causes Drupal to format invalid URLs.

example.com.foo.css (instead of ...com/foo.css) example.comsubdir/foo.css (instead of ...com/subdir/foo.css)

Drupal derives the URL from <?php dirname() ?> of <?php $_SERVER['SCRIPT_NAME'] ?> which will return . if no slashes or just / for something like /index.php.

The wrapper script above solves this by ensuring that the SCRIPT_NAME variable alway starts with a leading slash.

HTTP requests

GAE does not yet support support outbound sockets for PHP (although supported for Python and Java) and if/when it does the preferred way will continue to be streams due to automatic caching of outbound requests using urlfetch. I have included a small change to provide basic HTTP requests through drupal_http_request(). A proper solution would be to override the drupal_http_request_function variable and provide a fully functional alternative using streams. Drupal 8 has converted drupal_http_request() to use Guzzle which supports streams. Making a similar conversion for Drupal 7 seems like the cleanest way forward rather than reinventing the change.

php.ini

GAE disables a number of functions for security reasons, but only softly disables some functions which may then be enabled. Drupal provides access to phpinfo() from admin/reports/status and uses output buffering, both of which are disabled by default. The included php.ini enables both functions in addition to getmypid which is used by drupal_random_bytes().

# See https://developers.google.com/appengine/docs/php/config/php_ini.   # Required for ob_*() calls which you can find by grepping. # grep -nR '\sob_.*()' . output_buffering = "1"   # See https://developers.google.com/appengine/docs/php/runtime#Functions-That-Must-Be-Manually-Enabled # phpinfo: Provided on admin/reports/status under PHP -> "more information". # getmypid: Used by drupal_random_bytes(), but not required. google_app_engine.enable_functions = "getmypid, phpinfo" Future

I plan to continue working with the GAE team to ensure that support for Drupal can be provided in a clean and simple manner. Once current discussions have been resolved I hope to provide more formal documentation and support for Drupal.

File handling

I worked on file support, but there were a number of upcoming changes that would make things much cleaner so I decided to wait. GAE provides a stream wrapper for Google Cloud Storage which makes using the service very simple. Assuming you have completed the prerequisites files on GCS may be accessed using standard PHP file handling functions as shown in the documentation.

<?php
$file = 'gs://my_bucket/hello.txt';
file_put_contents($file, 'hello world');

$contents = file_get_contents($file);
var_dump($contents); // prints: hello world
?>

Unfortunately, the wrapper does not currently support directories nor does file_exists() work properly. Keep in mind that the filesystem is flat so a file may be written to any path without explicitly creating the directory. Meaning one can write to gs://bucket/foo/bar.txt without creating the directory foo. With that being the case it is possible to get some hacky support by simply disabling all the directory code in Drupal, but not really usable. It should be possible to hack support in through the stream wrapper since directories are simply specially name files, but the app engine team has indicated they will look into the matter so hopefully this will be solved cleanly.

Assuming the stream wrappers are fixed up then support can be added in much the same way as that Amazon S3 support is added except that no additional library will be needed.

Additionally, the documentation also notes the following.

Direct file uploads to your POST handler, without using the App Engine upload agent, are not supported and will fail.

In order to support file uploads the form must be submitted to the url provided by CloudStorageTools::createUploadUrl() and the forwarded result handled by Drupal. A benefit of proxying requests through uploader service is that uploaded files may be up to 100TB in size.

Other

There are a number of additional services provided as part of GAE of which Drupal could take advantage.

• Task Queue API
• Google account authentication
• Memcache seems to work out of the box with the memcache module
• <?php $SERVER['HTTP_X_APPENGINE*'] ?> variables (like geo information) Large number of services in API Console

Closing

Hopefully this will be useful in getting folks up and running quickly on GAE with Drupal and understanding the caveats of the environment. Obviously there is a lot more to cover and I look forward to seeing what others publish on the matter.

Tags:

• drupal
• drupal 6
• drupal 7
• google
• google app engine

Review: Asimov's Science Fiction, July 2011

Editor: Sheila Williams Issue: Volume 35, No. 7 ISSN: 1065-2698 Pages: 112

Williams's editorial is a mildly interesting piece about story titles. Silverberg's column is a more interesting (and rather convincing) rebuttal of the joke that fiction authors are "professional liars," combined with an examination of a fake and fantastic 14th travelogue that (at least in Silverberg's telling) was widely believed at the time. The precis of Silverberg's argument is that lying requires an intent to deceive, which is a property of deceptive memoir writers but not of fiction authors.

Di Filippo's review column, as usual, is devoted almost entirely to esoterica, although I was moderately interested to hear of Stableford's continued work on translating early French SF. None of it seems compelling enough to go buy, but good translations of early works seem like a good thing to have in the world.

"Day 29" by Chris Beckett: The conceit of this novelette is an interstellar travel system akin to a transporter that allows near-instantaneous travel between worlds. The drawback is that all memories from somewhere between 40 and 29 days before transit up until transit are wiped. The progatonist is a data analyst who is about to travel, and therefore by agency rule is required to stop doing work on day 40 before transmission since he can't be held legally liable for anything he has no recollection of doing. (I would like to say that I find this implausible, since one could always keep records, but it's exactly the sort of ass-covering regulation that a human resources department would come up with.)

The premise is quite interesting: what do you do during that period that you're going to forget? Beckett wisely mixes Stephen's current waiting period on the colony world with his diary of his original waiting period on Earth the first time he went through the transmission process, and the latter adds greatly to the reader's appreciation of the weirdness of the forgotten interval.

Unfortunately, this is a story more about psychological exploration than about plot, and Stephen just isn't very interesting. The telepathic but possibly nonsentient aliens add weirdness but not much else, and the ending of the story provided little sense of closure or conclusion for me. A good idea, but not the execution I wanted. (5)

"Pug" by Theodora Goss: Since I grew up with a pug, I have a soft spot for a story featuring one; sadly, though, this story has insufficient pug in it. This is a quiet fantasy (Asimov's calls it SF, presumably on the basis of parallel worlds and a hypothesized scientific explanation, but it reads like fantasy to me) featuring Victorian girls, including one with a bad heart. They discover a hidden door to other versions of their world and do some minor exploration. There's little or nothing in the way of plot; the story is more of an attempt to capture a mood. It's mildly diverting, but I wish it had gone somewhere more substantial. (5)

"Dunyon" by Kristine Kathryn Rusch: A Rusch story is often the highlight of an issue, and this is no exception. The protagonist is the owner of a bar in a space station that's become a combination of a refugee camp and a slum. War and chaos have created desperate people, most of whom are attempting to find some way to resources and get out of the bottom of society. The story is about a rumor: a mythical system named Dunyon that's safe and far away. And it's about how people react to that rumor. There's nothing particularly surprising about the direction the story goes (it's fairly short), but Rusch is always a good storyteller. (7)

"The Music of the Sphere" by Norman Spinrad: I've had mixed feelings about Spinrad's fiction (and some of his essays), but I liked this story, despite its implausibility. It's set in the near future, featuring an expert in cetaceans and dolphin perception and a composer obsessed with both loud music and classical musical style. Just from that description, you can probably predict much of the story, but I thought it had some neat ideas about dolphins, whales, and alternate perception and aesthetics. (Note: neat, not necessarily biologically plausible.) Enjoyable. (6)

"Bring on the Rain" by Josh Roseman: In a change of pace from the rest of the issue, this is a post-apocalyptic story of caravans of wheeled ships traversing a scorched and ruined landscape in search of weather systems and rain. The feel is of an inverted Waterworld, but with more emphasis on military tactics and cooperating fleets. The transposition of fleet maneuvers to huge ground vehicles adds some extra fun. The plot has little to do with the background and is a fairly stock military adventure scenario, but it's reasonably well-told. The story feels like an excerpt from a larger military-SF-inspired adventure, but the length keeps the quantity of tactics and maneuvering below the threshold where I would get bored. (6)

"Twelvers" by Leah Cypess: This is a sharp and occasionally mean story of adolescent cruelty and alienation. Darla is a "twelver," a child who was carried an extra three months in the womb using newly-invented medical technology because of a belief in the advantages this would bring in later life. Unfortunately for all those who used this technique, what it also brought was a preternatural calm and an unusual reaction to emotions. Darla finds it almost impossible to get upset at anything, and that, of course, prompts the cruelty and abuse of other children. Most of the story is a description of that abuse, leading up to Darla stumbling into a nasty solution to her immediate problem. It's all very believable (well, apart from the motivating biology), but I didn't enjoy reading about it, and I'm certainly not convinced that the ending will lead to anything good. (5)

"The Messenger" by Bruce McAllister: This is a very short time travel story, where time travel is used to try to unwind old family pain. This world follows the unalterable history model: no changes to the past are possible, and anything you do in the past has already happened. The mechanics are mostly avoided. Instead, McAllister concentrates on his mother, his father, and their complex relationship. I would have needed a bit more background on the characters to care enough about them for the story to be fully effective, but while the heartstring-pulling is kind of obvious, it's still a solid story. (6)

"The Copenhagen Interpretation" by Paul Cornell: This is the most ingenious of the stories in this issue. It's set in a future world that extends what seemed to me to be pre-World-War-I great power politics, although there may be a hint of the Cold War. Great nations have reached a careful balance of power, and spies and secret services work to sustain that balance. The progatonist is one of those agents, making use of advanced technology like space folds in the service of a cause that he doesn't entirely believe in. Cornell mixes in mental conditioning, artificial people, space travel, and even aliens (maybe) in a taut thriller plot that, for me, gained a great deal from the unexplained strangeness of its background. If you like diving into the deep end and following a fast-moving plot against a background of strangeness, this is the sort of SF you'll enjoy. (7)

Rating: 6 out of 10

The Python standard library comes with a module called wave.py. It allows you to read and write WAV format audio files.

Here is an example of reading WAV file info with Python:

import wave

wavf = wave.open('horse.wav', 'r')
print wavf.getnchannels()
print wavf.getsampwidth()
print wavf.getframerate()
print wavf.getnframes()
print wavf.getcompname()

The above code prints the number of channels (mono or stereo), the sampling width, the frame rate, the number of frames, and the name of the compression type (if any) of the WAV file horse.wav.

You can also write WAV files, but you need to know what data to write.

Wikipedia entry for WAV:

http://en.m.wikipedia.org/wiki/WAV

- Vasudev Ram
www.dancingbison.com

Vasudev Ram

This is summary of the Debugging chapter in The Practice of Programming by Brian Kernighan and Rob Pike. It was an absolute pleasure for me to read this paragraph, I have spent tons of time in debugging issues both in CPython and at work. The master programmers provide a great insight into the process of Debugging.

With the right attitude debugging can be fun, like solving a puzzle, whether we enjoy it or not, debugging is a art that you need to practise regularly. Still, it would be nice, if bugs didn't happen, so we try to avoid them by writing code well in the first place. Well written code has fewer bugs to begin with and those that remain are easier to find. Once a bug has been seen, the first thing to do is to think hard about the clues it presents. How could it have come about? Is it something familiar? Was something just changed in the program? Is there is something special about the input data that provoked it? A few well chosen test cases and few print statements in the code may be enough. If there aren't any good clues, hard thinking is still the best first step, to be followed by systematic attempts to narrow down the location of the problem. One step is cutting down the input data to make a small input that that fails; another is cutting out code to eliminate regions that can't be related. It's possible to insert checking code that gets turned on only after the program is executed some number of steps, again to try to localize the problem. All of these are instances of general strategy, divide and conquer, which as effective in debugging as it is politics and war. Use other aids as well. Explaining your code to someone else (even a teddy bear) is wonderfully effective. Use a debugger to get a stack trace. Use some of the commercial tools that check for memory leaks, array bound violations, suspect code and the like. Step through your program when it has become clear that you have the wrong mental picture of how the code works. Know yourself, and kinds of errors you make. Once you have found and fixed a bug, make sure you eliminate other bugs that might be similar. Think about what happened so you can avoid making that kind of mistake again.

And finally there was statement in the chapter which is very sums up what a good bug writing technique is.

Send the kind of bug report you'd like to receive yourself.