FLOSS Project Planets

Tennessee Leeuwenburg: Don't Click Me Results: Even More Linkbait

Planet Python - Tue, 2015-03-24 06:25
Okay, so I should have seen this coming. My last post, "Don't click me if you're human", generated over double my ordinary rate of traffic. My sincere thanks go out to those 22 souls who left comments so that I could discount them from the accounting.

I find it a bit hard to get the data I want at a glance from the blogger interface. I categorise my blog's traffic into the following categories:

  1. Number of visits while dormant (not really posting much)
  2. Number of visits on days I post
  3. Number of visits the day after that
  4. Number of visits during non-posting days, but not dormant

What I'd really like to get at is: how many humans do I actually provide some value to as a proportion of my visits? I'm not really an analytics exports...

Feedback comments are relatively rare. That's fine, but I do still hope that people are enjoying the posts.

The number of visits when dormant seems to be about 20-30 per day. I'm assuming that's robots. However, it's logical to assume that, like with humans, there will be more visits on posting days than on non-posting days.

My first two posts on this blog attracted something like 800 page views. Yay, great! I'm guessing that's maybe 400 actual people, and maybe 50 of those read the whole post, and maybe 35 of them really got something out of it. That's pretty good, it's a rare day that I manage to actually help 35 people in the rest of my life.

My "don't click me" post got 2100 page views. *facepalm*.

Now, presumably all (2100 - 800) of those are actual people.

I don't really know what it all means, except that the observer effect is very real, and linkbait works like a charm.

So, stay tuned for my next amazing post: "I'm going to waste all your time and you'll hate me for it".

Until next time, happy coding!
-Tennessee
Categories: FLOSS Project Planets

OpenSource.com: Why OpenStack is different from other open source projects

Planet Apache - Tue, 2015-03-24 05:00

How does OpenStack differ from other large, popular open source projects and how do these differences affect the way the project is growing and maturing?

Categories: FLOSS Project Planets

S. Lott: Configuration Files, Environment Variables, and Command-Line Options

Planet Python - Tue, 2015-03-24 04:00
We have three major tiers of configuration for applications. Within each tier, we have sub-tiers, larding on yet more complexity. The organization of the layers is a bit fungible, too. Making good choices can be rather complex because there are so many variations on the theme of "configuration". The desktop GUI app with a preferences file has very different requirements from larger, more complex applications.

The most dynamic configuration options are the command-line arguments. Within this tier of configuration, we have two sub-tiers of default values and user-provided overrides to those defaults. Where do the defaults come from? They might be wired in, but more often they come from environment variables or parameter files or both.

There's some difference of opinion on which tier is next in the tiers of dynamism. The two choices are configuration files and environment variables. We can consider environment variables as easier to edit than configuration files. In some cases, though, configuration files are easier to change than environment variables. Environment variables are typically bound to the process just once (like command-line arguments), where configuration files can be read and re-read as needed.

The environment variables have three sub-tiers. System-level environment variables tend to be fixed. The variables set by a .profile or .bashrc tend to be specific to a logged-in user, and are somewhat more flexible that system variables. The current set of environment variables associated with the logged-in session can be modified on the command line, and are as flexible as command-line arguments.

Note that we can do this in Linux:

PYTHONPATH=/path/to/project python3 -m some_app -opts

This will set an environment variable as part of running a command.

The configuration files may also have tiers. We might have a global configuration file in /etc/our-app. We might look for a ~/.our-app-rc as a user's generic configuration. We can also look for our-app.config in the current working directory as the final set of overrides to be used for the current invocation.

Some applications can be restarted, leading to re-reading the configuration files. We can change the configuration more easily than we can bind in new command-line arguments or environment variables.

Representation Issues

When we think about configuration files, we also have to consider the syntax we want to use to represent configurable parameters. We have five common choices.

Some folks are hopelessly in love with Windows-style .ini files. The configparser module will parse these. I call it hopelessly in love because the syntax is rather quite limited. Look at the logging.config module to see how complex the .ini file format is for non-trivial cases.

Some folks like Java-style properties files. These have the benefit of being really easy to parse in Python. Indeed, scanning a properties file is great exercise in functional-style Python programming.
I'm not completely sold on these, either, because they don't really handle the non-trivial cases well.

Using JSON or YAML for properties has some real advantages. There's a lot of sophistication available in these two notations. While JSON has first-class support, YAML requires an add-on module.

We can also use Python as the language for configuration. For good examples of this, look at the Django project settings file. Using Python has numerous advantages. The only possible disadvantage is the time wasted arguing with folks who call it a "security vulnerability."

Using Python as the configuration language is only considered a vulnerability by people who fail to realize that the Python source itself can be hacked. Why waste time injecting a bug into a configuration file? Why not just hack the source?

My Current Fave 

My current favorite way to handle configuration is by defining some kind of configuration class and using the class object throughout the application. Because of Python's import processing, a single instance of the class definition is easy to guarantee.

We might have a module that defines a hierarchy of configuration classes, each of which layers in additional details.

class Defaults:
mongo_uri = "mongodb://localhost:27017"
some_param = "xyz"

class Dev(Defaults):
mongo_uri = "mongodb://sandbox:27017"

class QA(Defaults):
mongo_uri = "mongodb://username:password@qa02:27017/?authMechanism=PLAIN&authSource=$external"

Yes. The password is visible. If we want to mess around with higher levels of secrecy in the configuration files, we can use PyCrypto and a key generator to use an encrypted password that's injected into the URI. That's a subject for another post. The folks to can edit the configuration files often know the passwords. Who are we trying to hide things from?

How do we choose the active configuration to use from among the available choices in this file? We have several ways.
  • Add a line to the configuration module. For example, Config=QA will name the selected environment. We have to change the configuration file as our code marches through environments from development to production. We can use from configuration import Config to get the proper configuration in all other modules of the application.
  • Rely on the environment variable to specify which configuration use. In enterprise contexts, an environment variable is often available.We can import os, and use Config=globals()[os.environ['OURAPP_ENVIRONMENT']] to pick a configuration based on an environment variable. 
  • In some places, we can rely on the host name itself to pick a configuration. We can use os.uname()[1] to get the name of the server. We can add a mapping from server name to configuration, and use this: Config=host_map(os.uname()[1],Defaults).
  • Use a command-line options like "--env=QA". This can a little more complex than the above techniques, but it seems to work out nicely in the long run.
Command-line args to select a specific configuration

To select a configuration using command-line arguments, we must decompose configuration into two parts. The configuration alternatives shown above are placed in a config_params.py module. The config.py module that's used directly by the application will import the config_params.py module, parse the command-line options, and finally pick a configuration. This module can create the required module global, Config. Since it will only execute once, we can import it freely.
The config module will use argparse to create an object named options with the command-line options. We can then do this little dance:

import argparse
import sys
import config_params

parser= argparse.ArgumentParser()
parser.add_argument("--env", default="DEV")
options= parser.parse_args()

Config = getattr(config_params, options.env)
Config.options= options

This seems to work out reasonably well. We can tweak the config_params.py flexibly. We can pick the configuration with a simple command-line option.

If we want to elegantly dump the configuration, we have a bit of a struggle. Each class in the hierarchy introduces names: it's a bit of work to walk down the __class__.__mro__ lattice to discover all of the available names and values that are inherited and overridden from the parents.

We could do something like this to flatten out the resulting values:
Base= getattr(config_params, options.env)
class Config(Base):
def __repr__(self):
names= {}
for cls in reversed(self.__class__.__mro__):
cls_names= dict((nm, (cls.__name__, val))
for nm,val in cls.__dict__.items()
if nm[0] != "_")
names.update( cls_names )
return ", ".join( "{0}.{1}={2}".format(class_val[0], nm, class_val[1])
for nm,class_val in names.items() )

It's not clear this is required. But it's kind of cool for debugging.
Categories: FLOSS Project Planets

eGenix.com: eGenix pyOpenSSL Distribution 0.13.8 GA

Planet Python - Tue, 2015-03-24 04:00
Introduction

The eGenix.com pyOpenSSL Distribution includes everything you need to get started with SSL in Python. It comes with an easy to use installer that includes the most recent OpenSSL library versions in pre-compiled
form, making your application independent of OS provided OpenSSL libraries:

>>>   eGenix pyOpenSSL Distribution Page

pyOpenSSL is an open-source Python add-on that allows writing SSL-aware networking applications as as certificate managment tools. It uses the OpenSSL library as performant and robust SSL engine.

OpenSSL is an open-source implementation of the SSL/TLS protocol.

News

This new release of the eGenix.com pyOpenSSL Distribution includes the following updates:

New in eGenix pyOpenSSL
  • Added FreeBSD as supported platform.
  • Updated the Mozilla CA root bundle to version 2015-02-19
New in OpenSSL
  • Updated included OpenSSL libraries from OpenSSL 1.0.1k to 1.0.1m. We had skipped OpenSSL 1.0.1l, since the 1.0.1l release only included a patch for Windows we had already included in our 0.13.7 release. See ​https://www.openssl.org/news/secadv_20150319.txt for a complete list of changes. The following fixes are relevant for pyOpenSSL applications:
    • CVE-2015-0286: Segmentation fault in ASN1_TYPE_cmp.
    • CVE-2015-0287: ASN.1 structure reuse memory corruption.
    • CVE-2015-0289: PKCS#7 NULL pointer dereference.
    • CVE-2015-0292: A vulnerability existed in previous versions of OpenSSL related to the processing of base64 encoded data. Any code path that reads base64 data from an untrusted source could be affected (such as the PEM processing routines). Already fixed in OpenSSL 1.0.1h, but wasn't listed, so repeated here for completeness.
    • CVE-2015-0293: Denial-of-Service (DoS) via reachable assert in SSLv2 servers.
    • CVE-2015-0209: Use After Free following d2i_ECPrivatekey error. A malformed EC private key file consumed via the d2i_ECPrivateKey function could cause a use after free condition.
  • The FREAK Attack (CVE-2015-0204) patch was already available in our last release with OpenSSL 1.0.1k.
Please see the product changelog for the full set of changes.
pyOpenSSL / OpenSSL Binaries Included

In addition to providing sources, we make binaries available that include both pyOpenSSL and the necessary OpenSSL libraries for all supported platforms: Windows, Linux, Mac OS X and now FreeBSD, for x86 and x64.

To simplify installation, we have uploaded a web installer to PyPI which will automatically choose the right binary for your platform, so a simple

pip install egenix-pyopenssl

will get you the package with OpenSSL libraries installed. Please see our installation instructions for details.

We have also added .egg-file distribution versions of our eGenix.com pyOpenSSL Distribution for Windows, Linux and Mac OS X to the available download options. These make setups using e.g. zc.buildout and other egg-file based installers a lot easier.

Downloads

Please visit the eGenix pyOpenSSL Distribution page for downloads, instructions on installation and documentation of the package.

Upgrading

Before installing this version of pyOpenSSL, please make sure that you uninstall any previously installed pyOpenSSL version. Otherwise, you could end up not using the included OpenSSL libs.

More Information

For more information on the eGenix pyOpenSSL Distribution, licensing and download instructions, please write to sales@egenix.com.

Enjoy !

Marc-Andre Lemburg, eGenix.com

Categories: FLOSS Project Planets

Triquanta Web Solutions: What the fq? A short summary of Solr query fields.

Planet Drupal - Tue, 2015-03-24 03:24
#And how to use them in Drupal

A popular search engine for Drupal is Apache Solr. Although installation and configuration of Solr can be done almost completely via the Drupal Admin UI, in some cases it can be very instructive to see and understand what data is sent to and from Solr when a search is done from Drupal.

First place to look when using Solr inside Tomcat is the log file of Tomcat, usually /var/log/tomcat6/catalina.out. If this file is not present in this directory on your system use

locate catalina.out

or a similar command to find it.

If Solr is used in its own Jetty-container and is run as a seperate service (which is the only option for Solr 5.x), log4j is used to implement logging and configuration is done in the log4j.properties-file.

By default the logs are written to 'logs/solr' under the Solr root, this can be set with in log4j.properties with the 'solr.log'-option, for example:

solr.log=/var/solr/logs

For more information about log4j, see Apache Log4j 2.

In the log, each line like the following represents one search query:

INFO: [solrdev] webapp=/solr path=/select params={spellcheck=true& spellcheck.q=diam& hl=true& facet=true& f.bundle.facet.mincount=1& f.im_field_tag.facet.limit=50& f.im_field_tag.facet.mincount=1& fl=id,entity_id, entity_type, bundle, bundle_name, label, ss_language, is_comment_count, ds_created, ds_changed, score, path, url, is_uid, tos_name& f.bundle.facet.limit=50& f.content.hl.alternateField=teaser& hl.mergeContigious=true& facet.field=im_field_tag& facet.field=bundle& fq=(access_node_tfslk0_all:0+OR+access__all:0)& mm=1& facet.mincount=1& qf=content^40& qf=label^5.0& qf=tags_h2_h3^3.0& qf=taxonomy_names^2.0& qf=tos_content_extra^0.1& qf=tos_name^3.0& hl.fl=content& f.content.hl.maxAlternateFieldLength=256& json.nl=map& wt=json& rows=10& pf=content^2.0& hl.snippets=3& start=0& facet.sort=count& q=diam& ps=15} hits=10 status=0 QTime=12

NB: one way to get a clearer look at the log-lines, is by copying one of them into a text editor and replace '&' with '&\n' and ',' with ',\n' to get a more readable text.

Here '[solrdev]' indicates the core the query was submitted to and 'path=/select' the path.

Everything between the {}-brackets is what is added to the query as parameter. If your Solr host is localhost, Solr is running on port 8080 and the name of your core is solrdev then you can make this same query in any browser by starting with:

http://localhost:8080/solr/soldev/select?

followed by all the text between the {}-brackets.

This looks like no simple query and in fact a lot is going on here: not only is the Solr/Lucene index searched for a specific term, we also tell Solr which fields to return, to give us spellcheck suggestions, to higlight the search term in the return snippet, to return facets etcetera.

For better understanding of the Solr query we will break it down and discuss each (well, maybe not all) of the query parameters from the above log line.

Query breakdown q: Search term

The most basic Solr query would only contain a q-field, e.q.

http://localhost:8080/solr/solrdev/select?q=diam

This would return all fields present in Solr for all matching documents. This will either be fields directly defined in the schema.xml (in this examples we use the schema's based on the schema included in the Search API Solr module) like bundle_name:

<field name="bundle_name" type="string" indexed="true" stored="true"/>

or dynamic fields, which are created according to the field definition in the schema.xml, eg:

<dynamicField name="ss_*" type="string" indexed="true" stored="true" multiValued="false"/>

The above query run on my local development environment would return a number of documents like this one:

<doc> <bool name="bs_promote">true</bool> <bool name="bs_status">true</bool> <bool name="bs_sticky">false</bool> <bool name="bs_translate">false</bool> <str name="bundle">point_of_interest</str> <str name="bundle_name">Point of interest</str> <str name="content">Decet Secundum Wisi Cogo commodo elit eros meus nisl turpis.(...) </str> <date name="ds_changed">2015-03-04T08:45:18Z</date> <date name="ds_created">2015-02-19T18:55:44Z</date> <date name="ds_last_comment_or_change">2015-03-04T08:45:18Z</date> <long name="entity_id">10</long> <str name="entity_type">node</str> <str name="hash">tfslk0</str> <str name="id">tfslk0/node/10</str> <long name="is_tnid">0</long> <long name="is_uid">1</long> <str name="label">Decet Secundum Wisi</str> <str name="path">node/10</str> <str name="path_alias">content/decet-secundum-wisi</str> <str name="site">http://louis.atlantik.dev/nl</str> <arr name="sm_field_subtitle"> <str>Subtitle Decet Secundum Wisi</str> </arr> <arr name="spell"> <str>Decet Secundum Wisi</str> <str>Decet Secundum Wisi Cogo commodo elit eros meus nisl turpis. (...) </str> </arr> <str name="ss_language">nl</str> <str name="ss_name">admin</str> <str name="ss_name_formatted">admin</str> <str name="teaser"> Decet Secundum Wisi Cogo commodo elit eros meus nisl turpis. Abluo appellatio exerci exputo feugiat jumentum luptatum paulatim quibus quidem. Decet nutus pecus roto valde. Adipiscing camur erat haero immitto nimis obruo pneum valetudo volutpat. Accumsan brevitas consectetuer fere illum interdico </str> <date name="timestamp">2015-03-05T08:30:11.7Z</date> <str name="tos_name">admin</str> <str name="tos_name_formatted">admin</str> <str name="url"> http://louis.atlantik.dev/nl/content/decet-secundum-wisi </str> </doc>

(In this document I have left out most of the content of the fields content and spell).

Obviously, when searching we are not interested in all fields, for example the afore mentioned field content contains a complete view (with HTML tags stripped) of the node in Drupal, which most of the times is not relevant when showing search results.

fl: Fields

So the first thing we want to do is limit the number of fields Solr is returning by adding the fl parameter, in which we name the fields we want returned from Solr:

http://localhost:8080/solr/solrdev/select?q=diam&fl=id,entity_id,entity_type,bundle,bundle_name,label,ss_language,score,path,url

This would return documents like:

<doc> <float name="score">0.1895202</float> <str name="bundle">point_of_interest</str> <str name="bundle_name">Point of interest</str> <long name="entity_id">10</long> <str name="entity_type">node</str> <str name="label">Decet Secundum Wisi</str> <str name="path">node/10</str> <str name="ss_language">nl</str> <str name="url"> http://localhost/nl/content/decet-secundum-wisi </str> </doc>

Here we not only use fields which are direclty present in the index (like bundle) but also a generated field score which indicates the relevance of the found item. This field can be used to sort the results by relevance.

By the way, from Solr 4.0 on, the fl can be added multiple times to the query with in each parameter one field. However the "old" way of a comma-seperated field list is still supported (also in Solr 5).

So in Solr 4 the query could (or should I say: should?) be written as:

http://localhost:8080/solr/solrdev/select?q=diam&fl=id&fl=entity_id&fl=entity_type&fl=bundle&fl=bundle_name&fl=label&fl=ss_language&fl=score&fl=path&fl=url

NB: in Solr 3 this would give a wrong result, with only the first fl field returned.

Please note that you can not query all dynamic fields at once with a fl-parameter like 'fl=ss_*': you must specify the actual field which are created while indexing: fl=ss_language,ss_name,ss_name_formatted... etc.

fq: Filter queries

One thing we do not want is users finding unpublished content which they are not allowed to see. When using the Drupal scheme, this can be accomplished by filtering on the dynamic fields created from

<dynamicField name="access_*" type="integer" indexed="true" stored="false" multiValued="true"/>

To filter, we add a fq-field like this:

http://localhost:8080/solr/solrdev/select?q=diam&fl=id,entity_id,entity_type,bundle,bundle_name,label,ss_language,score,path,url&fq=(access_node_tfslk0_all:0+OR+access__all:0)

The queries in fq are cached independent from the other Solr queries and so can speed up complex queries. The query can also contain range-queries, e.g. to limit the returned documents by a popularity-field present in the Drupal-node (and of course, indexed) between 50 and 100, one could use

fq=+popularity:[50 TO 100]

For more info see the Solr wiki, CommonQueryParameters

To add filter queries programmatically in Drupal when using the Apache Solr Search-module, implement

hook_apachesolr_query_alter()

and use

$query->addFilter

to add filters to the query.

For example, to filter the query on the current language, use:

function mymodule_apachesolr_query_alter(DrupalSolrQueryInterface $query) { global $language; $query->addFilter("ss_language", $language->language); }

If using the Solr environment with Search API Solr Search, implement

hook_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query)

In this hook you can alter or add items to the

$call_args['params']['fq']

to filter the query.

 

In both cases the name of the field to use can be found via the Solr admin. In this case it is a string fieks created from the dynamic ss_ *-field in 'schema.xml'.

rows and start: The rows returnd

The 'row'-parameter determines how many documents Solr will return, while 'start' detemines how many documents to skip. This basically implements pager-functionality.

wt: Type of the returned data

The 'wt'-parameter determines how the data from Solr is returned. Most used are:

  • xml (default)
  • json

See https://cwiki.apache.org/confluence/display/solr/Response+Writers for a complete list of available formats and their options.

qf: Boosting

The DisMax and EdisMax plugins have the abilty to boost the score of documents. In the default Drupal requestHandler ("pinkpony") the Edismax-parser is set as query plugin:

<requestHandler name="pinkPony" class="solr.SearchHandler" default="true"> <lst name="defaults"> <str name="defType">edismax</str>

Boosting can be done by adding one or more qf-parameters which define the fields ("query field") and their boost value in the following syntax:

[fieldName]^[boostValue]

E.g:

qf=content^40& qf=label^5.0& qf=tags_h2_h3^3.0& qf=taxonomy_names^2.0&

In Drupal this is done by implementing the relevant (Search API or Apache Solr) hook and adding the boost.

For example, let's say you want to boost the result with a boost value of "$boost" if a given taxonomy term with id "$tid" is present int the field "$solr_field"

For Apache Solr module we should use:

function mymodule_apachesolr_query_alter(DrupalSolrQueryInterface $query) { $boost_q = array(); $boost_q[] = $solr_field . ':' . $tid . '^' . $boost; $query->addParam('bq', $boost_q); }

For Search API Apache Solr:

function mymodule_search_api_solr_query_alter(array &$call_args, SearchApiQueryInterface $query) { $call_args['params']['bq'][] = $solr_field . ':' . $tid . '^' . $boost; }

NB: both examples we ignore any boost-values set by other modules for the same field. In real life you should merge the exisitng boost-array with our new one.

Negative boosting

Negative boosting is not possible in Solr. It can however be simulated by boosting all documents who do not have a specific value. In the above example of boosting on a specific taxonomy term, we can use:

$boost = abs($boost); $boost_q[] = '-' . $solr_field . ':' . $tid . '^' . $boost;

or, for Search API

$boost = abs($boost); $call_args['params']['bq'][] = '-' . $solr_field . ':' . $tid . '^' . $boost;

where the '-' for the field name is used to indicated that we want to boost the items that do not have this specific value.

mm: minimum should match

The Edismax parser also supports querying phrases (as opposed to single words). With the 'mm' parameters the minimum amount of words that must be present in the Solr-document is given. For example: if the query is "little red corvet" and 'mm' is set to '2', only documents which contain at least:

  • "little" and "red"
  • "little" and "corvette"
  • "red" and "corvette"

are returned, and documents which contain only of the words are not.

q.alt: for empty queries

If specified, this query will be used when the main query string is not specified or blank. This parameter is also specific for the EdisMax query handler.

And even more

Of course the above mentioned fields are not all fields used in a Solr query. Much more can be done with them and there are a lot of other parameters to influence the output of Solr.

To mention just a few:

Facets

Facets are one of the strong assets of Solr. A complete description of all possibilities and settings for facets would take to far in this scope, but a number of usefull parameters are discussed her.

The Drupal Facet API module and its range of plugins have a plethora of settings and possibilites, like the Facet Pretty Paths-module or, for an example of the numerous possibilities of the Facet API, the Facet API Taxonomy Sort module.

The most important Solr parameters in relation to facets are:

facet

Turn on the facets by using "facet=true" in the query.

facet.field

The field to return facets for, can be added more than once.

facet.mincount

This indicates the minimum amount of results for which to show a facet.

If this is set to '1', all facets items which would return documents are returned, if set to '0' a facet item for each value in the field will be returned, even if clicking on the item would return zero documents.

facet.sort

How to sort the facet items. Possible values are:

facet.sort=count

Sort by number of returned documents

facet.sort=index

Amounts to sorting by alphabet, or, in the Solr-wiki style: return the constraints sorted in their index order (lexicographic by indexed term). For terms in the ascii range, this will be alphabetically sorted.

facet.limit

The maximum amount of facet items returned, defaults to 100.

facet.offset

The number of facet items skipped. Defaults to '0'.

'facet.limit' and 'facet.offset' can be combined to implement paging of facet items.

Highlighting hl

Highlighting is turned on with 'hl=true' and enables highlighting the keywords in the search snippet Solr returns. Like facetting and spellchecking, highlighting is an extensive subject, see http://wiki.apache.org/solr/HighlightingParameters for more info.

hl.formatter

According to the Solr wiki: currently the only legal value is "simple". So just use 'simple'.

hl.simple.pre/hl.simple.post

Set the tags used to surround the highlight, defaults to <em> / </em> To wrap the highlight in for example bold tags, use:

hl.simple.pre=<b>&hl.simple.post=<b> Spellchecking spellcheck

Spellchecking is enabled with 'spellcheck=true'.

Because spellchecking is a complicated and language dependend process, it is not discussed here in full, see http://wiki.apache.org/solr/SpellCheckComponent for more information about the spellcheck component.

If the query for the spellchecker is given in a seperate 'spellcheck.q'-parameter like this:

spellcheck.q=<word>

this word is used for spell checking. If the 'spellcheck.q'-parameter is not set, the default 'q'-parameters is used as input for the spellchecker. Of course the word in the 'spellcheck.q' should bare a strong relation to the word in the 'q'-parameter, otherwise ununderstandable spelling suggestions wpould be given.

One can also make seperate requests to the spellchecker:

http://localhost:8080/solr/solrdev/spell?q=<word>&spellcheck=true&spellcheck.collate=true&spellcheck.build=true

Where <word> in the 'q-parameter is the word to use as input for the spellchecker.

One important subject releated to spellchecking is the way content is analyzed before it is written into the index or read from it. See SpellCheckingAnalysis for the default settings for the spellcheck-field.

In Drupal there is a spellcheck module for Search API Search API Spellcheck which can be used for multiple search backends.

Conclusion

Although most of the parameters mentioned above are more or less hidden by the Drupal admin interface, a basic understanding of them, can help to understand why your Solr search does (or more usually: does not) returns the results you expected.

As said in the introduction: looking at the queries in the Tocmat log can help a lot when debugging Solr.

Categories: FLOSS Project Planets

Kushal Das: Tunir, a simple CI with less pain

Planet Python - Tue, 2015-03-24 02:46

One of my job requirement is to keep testing the latest Fedora cloud images. We have a list of tests from Fedora QA team. But the biggest problem is that I don’t like doing these manually. I was looking for a way to run these automatically. We can do this by the normal CI systems, but there are two problems in that.

  • Most CI systems cannot handle cloud images, unless there is a real cloud running somewhere.
  • Maintaining the CI system & the cloud is a pain in my standard.

Tunir came out as a solution to these problems. It is a simple system, which can run predefined set of commands in a fresh cloud instance, or in a remote system. Btw, did I mention that you don’t need a cloud to run these cloud instances in your local system? This is possible thanks to the code from Mike Ruckman.

Each job in Tunir requires two files, jobname.json and jobname.txt. The json file contains the details of the Cloud image (if any), or the remote system details, ram required for the vm etc. The .txt file contains the shell commands to run in the system. For now it has two unique commands for Tunir. You can write @@ in front of any command to mark that this command will return non zero exit code. We also have a SLEEP NUMBER_OF_SECONDS option, we use it when we reboot the system, and want Tunir to wait before executing the next command.

Tunir has a stateless mode, I use that all the time :) In stateless mode, it will not save the results in any database. It will directly print the result in the terminal.

$ tunir --job fedora --stateless

Tunir uses redis to store some configuration information, like available ports. Remember to execute createports.py to fill the configuration with available ports.

You can install Tunir using pip, a review request is also up for Fedora. If you are on Fedora 21, you can just test with my package.

I am currently using unittest for the Cloud testcases, they are available at my github. You can use fedora.json and fedora.txt from the same repo to execute the tests. Example of tests running inside Tunir is below (I am using this in the Fedora Cloud tests).

curl -O https://kushal.fedorapeople.org/tunirtests.tar.gz tar -xzvf tunirtests.tar.gz python -m unittest tunirtests.cloudtests sudo systemctl stop crond.service @@ sudo systemctl disable crond.service @@ sudo reboot SLEEP 30 sudo python -m unittest tunirtests.cloudservice.TestServiceManipulation @@ sudo reboot SLEEP 30 sudo python -m unittest tunirtests.cloudservice.TestServiceAfter

UPDATE: Adding the output from Tunir for test mentioned above.

sudo ./tunir --job fedora --stateless [sudo] password for kdas: Got port: 2229 cleaning and creating dirs... Creating meta-data... downloading new image... Local downloads will be stored in /tmp/tmpZrnJsA. Downloading file:///home/Fedora-Cloud-Base-20141203-21.x86_64.qcow2 (158443520 bytes) Succeeded at downloading Fedora-Cloud-Base-20141203-21.x86_64.qcow2 download: /boot/vmlinuz-3.17.4-301.fc21.x86_64 -> ./vmlinuz-3.17.4-301.fc21.x86_64 download: /boot/initramfs-3.17.4-301.fc21.x86_64.img -> ./initramfs-3.17.4-301.fc21.x86_64.img /usr/bin/qemu-kvm -m 2048 -drive file=/tmp/tmpZrnJsA/Fedora-Cloud-Base-20141203-21.x86_64.qcow2,if=virtio -drive file=/tmp/tmpZrnJsA/seed.img,if=virtio -redir tcp:2229::22 -kernel /tmp/tmpZrnJsA/vmlinuz-3.17.4-301.fc21.x86_64 -initrd /tmp/tmpZrnJsA/initramfs-3.17.4-301.fc21.x86_64.img -append root=/dev/vda1 ro ds=nocloud-net -nographic Successfully booted your local cloud image! PID: 11880 Starting a stateless job. Executing command: curl -O https://kushal.fedorapeople.org/tunirtests.tar.gz Executing command: tar -xzvf tunirtests.tar.gz Executing command: python -m unittest tunirtests.cloudtests Executing command: sudo systemctl stop crond.service Executing command: @@ sudo systemctl disable crond.service Executing command: @@ sudo reboot Sleeping for 30. Executing command: sudo python -m unittest tunirtests.cloudservice.TestServiceManipulation Executing command: @@ sudo reboot Sleeping for 30. Executing command: sudo python -m unittest tunirtests.cloudservice.TestServiceAfter Job status: True command: curl -O https://kushal.fedorapeople.org/tunirtests.tar.gz status: True % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 8019 100 8019 0 0 4222 0 0:00:01 0:00:01 --:--:-- 4224 command: tar -xzvf tunirtests.tar.gz status: True tunirtests/ tunirtests/cloudservice.py tunirtests/LICENSE tunirtests/testutils.py tunirtests/__init__.py tunirtests/cloudtests.py command: python -m unittest tunirtests.cloudtests status: True .suu ---------------------------------------------------------------------- Ran 4 tests in 0.036s OK (skipped=1, unexpected successes=2) command: sudo systemctl stop crond.service status: True command: @@ sudo systemctl disable crond.service status: True Removed symlink /etc/systemd/system/multi-user.target.wants/crond.service. command: @@ sudo reboot status: True command: sudo python -m unittest tunirtests.cloudservice.TestServiceManipulation status: True . ---------------------------------------------------------------------- Ran 1 test in 0.282s OK command: sudo python -m unittest tunirtests.cloudservice.TestServiceAfter status: True . ---------------------------------------------------------------------- Ran 1 test in 0.070s OK
Categories: FLOSS Project Planets

Drupal.org frontpage posts for the Drupal planet: Announcing The Aaron Winborn Award to honor amazing community members

Planet Drupal - Tue, 2015-03-24 00:17

In honor of long-time Drupal contributor Aaron Winborn (see his recent Community Spotlight), whose battle with Amyotrophic lateral sclerosis (ALS) (also referred to as Lou Gehrig's Disease) is coming to an end later today, the Community Working Group, with the support of the Drupal Association, would like to announce the establishment of the Aaron Winborn Award.

This will be an annual award recognizing an individual who demonstrates personal integrity, kindness, and above-and-beyond commitment to the Drupal community. It will include a scholarship and stipend to attend DrupalCon and recognition in a plenary session at the event. Part of the award will also be donated to the special needs trust to support Aaron's family on an annual basis.

Thanks to Hans Riemenschneider for the suggestion, and the Drupal Association executive board for approving this idea and budget so quickly. We feel this award is a fitting honor to someone who gave so much to Drupal both on a technical and personal level.

Thank you so much to Aaron for sharing your personal journey with all of us. It’s been a long journey, and a difficult one. You and your family are all in our thoughts.

Front page news: Planet Drupal
Categories: FLOSS Project Planets

Russ Allbery: Review: Fukushima

Planet Debian - Mon, 2015-03-23 23:59

Review: Fukushima, by David Lochbaum, et al.

Author: David Lochbaum Author: Edwin Lyman Author: Susan Q. Stranahan Author: Union of Concerned Scientists Publisher: The New Press Copyright: 2014 ISBN: 1-59558-927-9 Format: Kindle Pages: 320

This is a very interesting book, and I can recommend it, but there are two things you should be aware of up-front. The packaging does not necessarily make clear what expectations you should have of it going in.

First, the subtitle (The Story of a Nuclear Disaster) should have appended to it And Its Implications for US Nuclear Power Policy. This book is very concerned with the impact of the Fukushima disaster on US policy and nuclear regulation, to the point where I think more than half of the book is about US agencies, nuclear regulatory history, and US reaction. There's nothing wrong with that, of course: the US should take a hard look at its own nuclear energy policy given the events at Fukushima, and it's a worthy topic for a book. But if you go into this book expecting a broader perspective, you will be disappointed. For example, I think the fact that France has a lot of nuclear power was mentioned maybe twice in the whole book, and French reaction was never discussed at all. There is a very detailed examination of exactly what happened at Fukushima (more on that in a moment), but most of the policy implications are examined purely from a US perspective. Even Japanese nuclear policy gets somewhat short shrift.

Second, note that the fourth listed co-author is the Union of Concerned Scientists. For those not familiar with US environmental groups, the UCS has a reputation as an anti-nuclear advocacy organization. I don't think that's entirely fair; I think the UCS's position on nuclear power is better summarized as holding that it is theoretically possible to run a nuclear power plant safely, but the actual US nuclear power industry is not very close to that standard, and it would require much tighter regulation and more investment in safety systems to reach that standard. But be aware that the authors of this book have a clear position on the adequacy of current nuclear power safety standards, namely that they aren't. And they don't try to conceal that position in this book. Personally, I prefer authors to be open about their perspective in books like this, but your mileage may vary.

There, disclaimers out of the way. I bought this book for a specific reason: I had followed some of the news coverage at the time of the earthquake and tsunami, and then (like many people, I suspect) lost track of the final outcome as the story fell out of the news and I started ignoring people who didn't understand how large the Pacific Ocean is. Now that we've had the benefit of several years of analysis and thoughtful reconstruction of events, I wanted to know what had actually happened. I'm happy to say that this book delivers quite well on that front. Roughly the first half of the book is a detailed blow-by-blow description of exactly what happened at Fukushima, at least as well as we've been able to reconstruct, told as an engrossing and dramatic narrative. There may be a little too much interleaving of reactions within the US government, which I suspect will particularly annoy non-US readers, but the level of factual detail is excellent, clear, and well-explained.

What I wasn't expecting, but was pleasantly surprised by, is that it's also a great story. There's tension, conflict, heroism, hard choices, and moral quandries, and the authors do a great job conveying factual information while still giving the reader the sense of being in the middle of the unfolding drama. They resist the urge to disclose all the results of later analysis in the middle of the story, which may provide a slightly less clear view of the disaster, but which makes the telling far more compelling. I usually read non-fiction more slowly than fiction, but Fukushima dragged me in. I found myself grabbing moments to read just another few pages.

Unfortunately, this is only about half the book. The other half is a mix of other things that won't have as broad of appeal: an analysis of the challenges of US nuclear regulation, a history of the US nuclear power industry, and a presentation of the authors' opinions about the best path forward for regulation of nuclear power in the US. Since I'm a US citizen and resident with an interest in both nuclear power and regulation of nuclear power in my country, I found this interesting, if not as engrossing as the rest of the book. But it felt a bit oddly tacked on, and I think it's a stretch to say that it's part of the story of Fukushima.

The authors try to draw that link by presenting the Japanese nuclear power industry as heavily influenced by their US counterparts, and their regulatory problems as similar to the problems in the US, but there is nowhere near enough detail about Japanese regulatory practices here to support that conclusion. I think the largest weakness, and the most obvious gap, in this book is the lack of detailed analysis of the history and players in the Japanese nuclear regulatory environment. This is an odd miss. If one is concerned about regulatory inadequacy, Japanese government policy is far more obviously part of the story of Fukushima than US policy. I can only speculate that the authors had inside sources for the US policy discussions but not for the Japanese policy discussions (and, sadly, fall back on painting with a rather broad brush and making unsupported generalizations about Japanese regulatory approaches in a few spots). The result feels like two partly-unrelated books stacked and partly shuffled together.

So, there are parts of Fukushima that are rather disappointing, particularly for non-US readers. But I still recommend it as a great detailed history of the actual incident and a summary of what we now think happened. That summary is unfortunately sketchy and still very unclear, but I don't think that's the fault of the authors. The inside of a nuclear power plant during a meltdown is a very difficult environment to measure or analyze, and there's a lot of data that we will probably never have. Some details may never be known. But what we do know, and how that knowledge unfolded, is told very well.

This is the only book-length treatment on Fukushima I've read, so I can't compare it against other books on the same topic. But it satisfied my curiousity nicely. If you have a similar curiosity, I recommend this book to your attention, although be aware of its approach and its US-centric analysis going in so that you're not surprised by a mismatch of expectations.

Rating: 8 out of 10

Categories: FLOSS Project Planets

James Mills: A Docker-based mini-PaaS

Planet Python - Mon, 2015-03-23 23:05
The Why

So by now everyone has heard of Docker right? (If not, you have some catching up to do!)

Why have I created this mini-PaaS based around Docker? What's wrong with the many myriad of platforms and services out there:

Well. Nothing! The various platforms, services and stacks that exist to service, deploy, monitor applications using Docker all have their use-cases and pros and cons.

If you call the post Flynn vs. Deis: The Tale of Two Docker Micro-PaaS Technologies i said the following about ~10months ago.

I've stuck by this and 10 months later here it is.

docker-compose.yml:

autodock: image: prologic/autodock ports: - "1338:1338/udp" - "1338:1338/tcp" volumes: - /var/run/docker.sock:/var/run/docker.sock autodocklogger: image: prologic/autodock-logger links: - autodock autodockhipache: image: prologic/autodock-hipache links: - autodock - hipache:redis hipache: image: hipache ports: - 80:80 - 443:443

Gist here: https://gist.github.com/prologic/72ca4076a63d5dd1687d

This uses the following tools and software (all which I wrote as well):

Now. Here's the thing. Nothing here is particularly fancy.

  • There's no DNS management
  • There's no fancy services to speak of
  • There's no web interface at all.
  • There's no API or even a CLI tool

So what is there?

Basically this works in a very simple way.

  1. Setup a wildcard A record on a domain pointing it at your Docker host.
  2. Spin up containers with the -e VIRTUALHOST environment variable.

That's it!

The How

How this works:

  • autodock is a daemon that listens for Docker events via the Docker Remote API
  • autodock is pluggable and provides a UDP-based distributed interface to other plugins.
  • When autodock sees a Docker event it broadcasts it to all nodes.
  • When autodock-hipache sees container start/stop/died/killed/paused/unpaused events it: - Checks for a VIRTUALHOST environment variable. - Checks for a PORT environment variable (optional, default: ``80``). - Checks the configuration of the container for exposed ports. - If PORT is a valid exposed port; reconfigure hipache with the provided virtualhost from the VIRTUALHOST environment variable routing web requests to the container's ip address and port given by PORT.
Usage

Using this is quite simple. Copy the above docker-compose.yml and run:

docker-compose up -d

Then start a container:

docker run -d -e VIRTUALHOST=hello.local prologic/hello

And visit: http://hello.local

Assuming (of course) hello.local points to your Docker host in /etc/hosts.

Of course real deployments of this will use real domains and a real DNS server.

Two example deployments of this can be seen here:

Enjoy! :)

Categories: FLOSS Project Planets

lakshminp.com: The Drupal 8 plugin system - part 4

Planet Drupal - Mon, 2015-03-23 21:53

We defined what a plugin is, discussed some plugins used in core and wrote our own custom plugin previously. We shall tune it up a bit in this post.

Real world plugins have a lot more properties than the label property mentioned in our breakfast plugin. To make our plugin more "real world", we introduce 2 properties, image and ingredients. It makes more sense now to have a custom annotation for breakfast instead of using the default Plugin annotation.

How different are custom annotations from the usual Plugin annotation?

1) They convey more information about a plugin than what an @Plugin does. Here's a custom annotation for a views display plugin from search_api, taken from here.

/** * Defines a display for viewing a search's facets. * * @ingroup views_display_plugins * * @ViewsDisplay( * id = "search_api_facets_block", * title = @Translation("Facets block"), * help = @Translation("Display the search result's facets as a block."), * theme = "views_view", * register_theme = FALSE, * uses_hook_block = TRUE, * contextual_links_locations = {"block"}, * admin = @Translation("Facets block") * ) */

2) Custom annotations are a provision to document the metadata/properties used for a custom plugin.
Check out this snippet from FieldFormatter annotation code for instance:

/** * A short description of the formatter type. * * @ingroup plugin_translatable * * @var \Drupal\Core\Annotation\Translation */ public $description; /** * The name of the field formatter class. * * This is not provided manually, it will be added by the discovery mechanism. * * @var string */ public $class; /** * An array of field types the formatter supports. * * @var array */ public $field_types = array(); /** * An integer to determine the weight of this formatter relative to other * formatter in the Field UI when selecting a formatter for a given field * instance. * * @var int optional */ public $weight = NULL;

That gave us a lot of information about the plugin and its properties.
Now that you are convinced of the merits of custom annotations, let's create one.

Checkout the module code if you haven't already.

$ git clone git@github.com:badri/breakfast.git

Switch to the custom-annotation-with-properties tag.

$ git checkout -f custom-annotation-with-properties

and enable the module. The directory structure should look like this:

The new Annotation directory is of interest here. Custom annotations are defined here.

/** * A glimspe of how your breakfast looks. * * This is not provided manually, it will be added by the discovery mechanism. * * @var string */ public $image; /** * An array of igredients used to make it. * * @var array */ public $ingredients = array();

Now, the plugin definition is changed accordingly. Here's the new annotation of the idli plugin.

/** * Idly! can't imagine a south Indian breakfast without it. * * * @Breakfast( * id = "idly", * label = @Translation("Idly"), * image = "https://upload.wikimedia.org/wikipedia/commons/1/11/Idli_Sambar.JPG", * ingredients = { * "Rice Batter", * "Black lentils" * } * ) */

The other change is to inform the Plugin Manager of the new annotation we are using.
This is done in BreakfastPluginManager.php.

// The name of the annotation class that contains the plugin definition. $plugin_definition_annotation_name = 'Drupal\breakfast\Annotation\Breakfast'; parent::__construct($subdir, $namespaces, $module_handler, 'Drupal\breakfast\BreakfastInterface', $plugin_definition_annotation_name);

Let's tidy the plugin by wrapping it around an interface. Though this is purely optional, the interface tells the users of the plugin what properties it exposes.
This also allows us to define a custom function called servedWith() whose implementation is plugin specific.

With the plugin class hierarchy looking like this now:

The servedWith() is implemented differently for different plugin instances.

// Idly.php public function servedWith() { return array("Sambar", "Coconut Chutney", "Onion Chutney", "Idli podi"); } // MasalaDosa.php public function servedWith() { return array("Sambar", "Coriander Chutney"); }

We make use of the interface functions we wrote in the formatter while displaying details about the user's favorite breakfast item.

// BreakfastFormatter.php foreach ($items as $delta => $item) { $breakfast_item = \Drupal::service('plugin.manager.breakfast')->createInstance($item->value); $markup = '<h1>'. $breakfast_item->getName() . '</h1>'; $markup .= '<img src="'. $breakfast_item->getImage() .'"/>'; $markup .= '<h2>Goes well with:</h2>'. implode(", ", $breakfast_item->servedWith()); $elements[$delta] = array( '#markup' => $markup, ); }

And the user profile page now looks like this.

Derivative plugins

We have Idly plugin instance mapped to the Idly class and Uppuma instance mapped to the Uppuma class. Had all the plugin instances been mapped to a single class, we would have got derivative plugins. Derivative plugins are plugin instances derived from the same class.
They are employed under these scenarios:
1. If one plugin class per instance is an overkill.
There are times where you don't want to define a class for each plugin instance. You just say that it is an instance of a particular class that is already defined.
2. You need to dynamically define plugin instances.
The Flag module defines a different Flag Type plugin instance for different entities. The entities in a Drupal site are not known beforehand and hence we cannot define one instance each for every entity. This calls for a plugin derivative implementation.

Lets add derivative plugins to our breakfast metaphor.

$ git checkout -f plugin-derivatives

Here's a new user story for the breakfast requirement. We are going to add desserts to our menu now. All desserts are of type Sweet. So, we define a derivative plugin called Sweet which is based on breakfast.

This calls for 3 changes as shown in the new directory structure outlined below:

1) Define the Sweet plugin instance class on which all our desserts are going to be based on.

/** * A dessert or two whould be great! * * * @Breakfast( * id = "sweet", * deriver = "Drupal\breakfast\Plugin\Derivative\Sweets" * ) */ class Sweet extends BreakfastBase { public function servedWith() { return array(); } }

Note the "deriver" property in the annotation.

2) Next, we define the deriver in the Derivative directory.

/** * Sweets are dynamic plugin definitions. */ class Sweets extends DeriverBase { /** * {@inheritdoc} */ public function getDerivativeDefinitions($base_plugin_definition) { $sweets_list = drupal_get_path('module', 'breakfast') . '/sweets.yml'; $sweets = Yaml::decode(file_get_contents($sweets_list)); foreach ($sweets as $key => $sweet) { $this->derivatives[$key] = $base_plugin_definition; $this->derivatives[$key] += array( 'label' => $sweet['label'], 'image' => $sweet['image'], 'ingredients' => $sweet['ingredients'], ); } return $this->derivatives; } }

3) The derivative gets sweets info from the sweets.yml present in the module root directory. This could even be an XML/JSON or any file format which holds metadata. I've used a YAML file for consistency's sake.

mysore_pak: label: 'Mysore Pak' image: 'https://upload.wikimedia.org/wikipedia/commons/f/ff/Mysore_pak.jpg' ingredients: - Ghee - Sugar - Gram Flour jhangri: label: 'Jhangri' image: 'https://upload.wikimedia.org/wikipedia/commons/f/f8/Jaangiri.JPG' ingredients: - Urad Flour - Saffron - Ghee - Sugar

The class hierarchy for derivative plugins looks a little big bigger now.

Clear your cache and you must be able to see the sweets defined in the yaml file in the breakfast choice dropdown of the user profile.

Enjoy your dessert!

That concludes our tour of Drupal 8 plugins. Hope you liked it and learnt something. Stay tuned for the next series about services.

Categories: FLOSS Project Planets

Chris Moffitt: Generating fake data with barnum

Planet Python - Mon, 2015-03-23 21:28
Introduction

Several years ago, I developed a very simple program called barnum to generate fake data that could be used to test applications. Over the years, I had forgotten about it. With the recent closing of Google code, I decided to take the opportunity to move the code to github and see if it might be useful to people.

Motivation

I am moving the code and re-announcing it for a couple of reasons:

  1. I will admit that I hated the idea of it totally dying.
  2. I was curious to see if it is useful to anyone else.
  3. This would give me the opportunity to get more familiar with git (I am not afraid to admit that mercurial is my favorite but I think I need to expand my knowledge)
  4. I also chose this as an opportunity to update the code and make it run with python 3.
Basic Usage

Barnum will allow you to create the following types of data which can be used to test your own applications :

  • First name and/or last name in either gender
  • Job title
  • Phone number
  • Street number and name
  • Zip code plus city & state
  • Company name
  • Credit card number and type (with valid checksum)
  • Dates
  • Email addresses
  • Sample password
  • Words (latin)
  • Sentences and/or paragraphs of random latin words

Here is the basic usage and some of the unique aspects of barnum.

import gen_data

Create some fake names

gen_data.create_name() ('Arnoldo', 'Ulmer') gen_data.create_name() ('Louisa', 'Foy') gen_data.create_name(full_name=False) 'Gayla'

You can also specify the gender

gen_data.create_name(gender='female') ('Mandy', 'Pena')

We can also create job titles based on US Census data

gen_data.create_job_title() 'Security Coordinator Computer'

One of barnum’s unique capabilities is that there is some intelligence to the data creation. We can pass in a US Zip code and barnum will use a valid area code in the fake data.

gen_data.create_phone(zip_code="55082") '(612)242-2894' gen_data.create_phone() '(863)265-6706'

We can use zip codes for address data.

gen_data.create_city_state_zip() ('12136', 'Old Chatham', 'NY') gen_data.create_city_state_zip(zip_code="55112") ('55112', 'Saint Paul', 'MN')

Barnum can create fake sentences and paragraphs as well as nouns for all your data population needs.

gen_data.create_sentence() 'Aliquip vulputate consequat suscipit amet adipiscing molestie dignissim nulla molestie hendrerit.' gen_data.create_paragraphs() 'Illum eros et eu ad ipsum vulputate. Delenit commodoconsequat delenitaugue molestie iustoodio nonummy ut erat duis feugait. Doloremagna Utwisi aliquip molestie erat suscipit. Nonummy exerci eufeugiat illum vel nislut nisl at dolor at. nn' gen_data.create_nouns(max=4) 'rifle giraffe nerve kettle'

There is some date creation capability as well.

gen_data.create_date() datetime.datetime(2025, 1, 12, 19, 36, 25, 639016) gen_data.create_date(past=True) datetime.datetime(2014, 2, 23, 19, 37, 29, 323165) gen_data.create_date(max_years_future=2) datetime.datetime(2016, 11, 17, 19, 37, 52, 674284) gen_data.create_birthday(min_age=2, max_age=75) datetime.date(2007, 3, 25)

Create email and company name info.

gen_data.create_email() 'Valorie.Wise@nullafeugait.com' gen_data.create_email(tld="net") 'Trina@uttation.net' gen_data.create_email(name=("Fred","Jones")) 'F.Jones@wisiillum.edu' gen_data.create_company_name() 'Application Telecom Inc' gen_data.create_company_name(biz_type="LawFirm") 'Marion, Navarro & Quintero LLP' gen_data.create_company_name(biz_type="Generic") 'Application Data Direct Limited'

Finally, you can also create credit card numbers and simple passwords.

gen_data.cc_number() ('visa', ['4716823823471406']) gen_data.cc_number(card_type="mastercard") ('mastercard', ['5531134290292667']) gen_data.create_pw() 'W7jWw4kn' gen_data.create_pw(length=10) '4KvqFS8Znu' Customization

One final component available to you is that you can customize the data used to create your "random" results. Within the source-data directory, there are several text files. If you choose to customize the files, you can update them and use the convert_data.py file to regenerate the random data file.

Next Steps

I know there are a lot of areas where this code can be improved. I’m hopeful that placing it on github will breathe some new life into it. However, if there’s no real interest in it, I’ll be able to rest easier knowing that it is being hosted somewhere where others can use it.

Enjoy!

Categories: FLOSS Project Planets

Colan Schwartz: Drupal-specific hosting: Choose a provider from those offering comprehensive platforms

Planet Drupal - Mon, 2015-03-23 21:12
Topics: 

There are many Web hosting companies claiming their ability to host Drupal sites in the enterprise space. However, most of the time, these providers simply provide the hardware or a virtual machine (VM/VPS) with an operating system (OS) capable of hosting Drupal if you build the application stack, configure it and manage it all yourself. They may even claim that they can set all of this up for you, but they'll charge extra for the labour. They don't have a comprehensive platform where instances can be automatically deployed as needed.

What do I mean by a "platform"?

I'm considering a somewhat complex requirement: a solution that includes a fully-managed Drupal application stack with several environments where it is trivial to move code, databases and files between them.

While it seems like it would be difficult to find such a service, there are now several available. While these options may seem expensive relative to generic Web-site hosting, they'll save you immensely on labour costs, as you won't have to pay to do any of the following yourself:

  • Installation of the application stack on each environment (Dev, Staging and Prod, see below.)
  • Managing all OS software updates and security patches.
  • Managing the configuration between environments (a.k.a. Configuration Management)
  • Self-hosting and managing a deployment tool such as Aegir or creating and managing your own deployment scripts.
  • Developing, maintaining and documenting processes for your development workflow.

The environments in question are Dev (or Development, the integration server where code from all developers is combined and tested), Staging (or QA, where quality assurance testing is done with the content and configuration from Production/Live and the code from Dev) and Prod (Production/Live, your user-facing site). Typically, these three (3) server environments are set up for you, and the Drupal hosting provider provides a simple-to-use command-line (CLI) and/or Web interface to deploy code to Dev, to Staging and finally to Prod while allowing you to easily move site data (content and configuration) from Prod to Staging to Dev. Your developers then work in their own sandboxes on their laptops pushing code up through the environments, and pulling site data from them.

This allows you to significantly reduce or eliminate the resources you'll need for system administration and DevOps. You can focus your project resources on application development, which is to get your content management system up and running as quickly as possible, while leaving infrastructure issues to the experts. Even if these items are outsourced, they add additional costs to your project.

However, you may have specific requirements which preclude you from going with such a managed solution. You may have a need to keep the environments behind your corporate firewall (say to communicate securely with other internal back-end systems such as CRMs, authentication servers or other data stores) or your country/jurisdiction has strict privacy legislation preventing you from physically locating user data in regions where electronic communication is not guaranteed to be kept private. For example, in Canada, personal information can only be used for purposes that users have consented to (see PIPEDA). As a good number of the described hosting services rely on data centres in the United States, the PATRIOT Act there (which gives the US government full access to any electronic communication) could make it difficult to enforce such privacy protections.

At the time of this writing, the following Drupal hosting companies are available to provide services as described above. They're listed in no particular order.

I haven't had a chance to try all of these yet as I have clients on only some of them. It would be great if someone who's evaluated all of them were to publish a comparison.

This article, Drupal-specific hosting: Choose a provider from those offering comprehensive platforms, appeared first on the Colan Schwartz Consulting Services blog.

Categories: FLOSS Project Planets

Justin Mason: Links for 2015-03-23

Planet Apache - Mon, 2015-03-23 19:58
Categories: FLOSS Project Planets

Carl Chenet: Unverified backups are useless. Automatize the controls!

Planet Debian - Mon, 2015-03-23 19:00

Follow me on Identi.ca  or Twitter  or Diaspora*

Unverified backups are useless, every sysadmins know that. But manually verifying a backup means wasting time and resources. Moreover it’s boring. You should automatize it!

Charlie Chaplin Modern Times

Backup Checker is a command line software developed in Python 3.4 on GitHub (stars appreciated :) ) allowing users to verify the integrity of archives (tar, gz, bz2, lzma, zip, tree of files) and the state of the files inside an archive in order to find corruptions or intentional of accidental changes of states or removal of files inside an archive.

Backup Checker on github

The new feature of the latest version 1.4 is the control of outdated archives with the new outdated parameter. Lots of data are outdated quite fast, because they are dependent of other data, or because they are only useful in a specific context.

Hey, this database dump is 6 months old, it’s useless today!

Backup Checker now controls the expiration duration and triggers a warning if the given duration starting from the last modification of the archive (mtime) is expired. Short examples of the warning:

WARNING:root:/backups/backups-12022015.tar.gz is outdated. Was good until 01/03/15 00:00:00 – now 22/03/15 21:38:20

You won’t be surprized any more by outdated useless data in your backups.

Backup Checker also offers lots of other controls. Check the features list!

Installing Backup Checker

Backup Checker is available from PyPI using the following command:

# pip3.4 install backupchecker

It’s also available for your Debian Squeeze or Debian Wheezy. Check how to get it for your specific distributions.

What about you? How and what for do you use Backup Checker? We would be happy to get your feedbacks. The project cares about our users and the outdated feature was a awesome idea in a feature request by one of the Backup Checker user, thanks Laurent!

 


Categories: FLOSS Project Planets

Cutelyst 0.7.0 is out and now has a tutorial!

Planet KDE - Mon, 2015-03-23 18:15

Cutelyst the Qt/C++ web framework just got a new release, 0.7.0 brings many improvements, bugfixes and features.

Most notably of the changes is a shinny new Chained dispatcher that finally got implemented, with it Cutelyst Tutorial got finished, and a new command line tool to bootstrap new projects.

* Request::bodyData() can return a QJsonDocument is content-type is application/json
* Chained dispatcher
* Fixes on QStringListeral and QLatin1String usage
* More debug information
* Some API changes to match Catalyst’s
* Some rework on the API handlying Plugins (not the best thing yet)
* Moved to Gitlab (due to the gitorious being acquired by it)
* “cutelyst” command line tool

For the next release I’ll try to make the QML integration a reality, and try to improve the Plugins API, which is right now the part that needs more love would love to hear some advice.

Download it here.

Have fun!


Categories: FLOSS Project Planets

Mike Driscoll: The Python 101 Screencast Kickstarter is Launching Soon

Planet Python - Mon, 2015-03-23 18:15

I am currently working on another Kickstarter campaign to turn my book, Python 101 into a video series. I am planning on launching this project on Wednesday, March 25th. There are 44 chapters in my book, so I’ll be creating a minimum of 44 screencast videos. Here is what is currently covered in the book and what will also be covered in the video series:

Part I: Learning the Basics

• Chapter 1 – IDLE
• Chapter 2 – Strings
• Chapter 3 – Lists, Tuples and Dictionaries
• Chapter 4 – Conditional Statements
• Chapter 5 – Loops
• Chapter 6 – Python Comprehensions
• Chapter 7 – Exception Handling
• Chapter 8 – Working with Files
• Chapter 9 – Importing
• Chapter 10 – Functions
• Chapter 11 – Classes

Part II: Learning from the Library

• Chapter 12 – Introspection
• Chapter 13 – The csv Module
• Chapter 14 – The ConfigParser Module
• Chapter 15 – The logging Module
• Chapter 16 – The os Module
• Chapter 17 – The email and smtplib Modules
• Chapter 18 – The sqlite Module
• Chapter 19 – The subprocess Module
• Chapter 20 – The sys Module
• Chapter 21 – The threading Module
• Chapter 22 – The datetime and time Modules
• Chapter 23 – The xml Modules

Part III – Intermediate Odds and Ends

• Chapter 24 – The Python Debugger  • Chapter 25 – Decorators
• Chapter 26 – The lambda
• Chapter 27 – Code Profiling
• Chapter 28 – An Intro to Testing

Part IV – Tips, Tricks and Tutorials

• Chapter 29 – Installing Modules
• Chapter 30 – ConfigObj
• Chapter 31 – Parsing XML with lxml
• Chapter 32 – Python Code Analysis
• Chapter 33 – The requests package
• Chapter 34 – SQLAlchemy
• Chapter 35 – virtualenv

Part V – Packaging and Distribution

• Chapter 36 – Creating Modules and Packages
• Chapter 37 – How to Add Your Code to PyPI
• Chapter 38 – The Python egg
• Chapter 39 – Python wheels
• Chapter 40 – py2exe
• Chapter 41 – bbfreeze
• Chapter 42 – cx_Freeze
• Chapter 43 – PyInstaller
• Chapter 44 – Creating an Installer

 

I will probably be tossing in other videos on related topics as I go along. I look forward to hearing from my readers and backers. It’s going to be a blast to figure out how to get all this information into your hands!

Categories: FLOSS Project Planets

KRecipes 2.1

Planet KDE - Mon, 2015-03-23 18:02

The KDE Gardening team has at last finished the Love project KRecipes with the 2.1 release which can be found here: http://download.kde.org/stable/krecipes/2.1.0/src/krecipes-2.1.0.tar.xz.mirrorlist

Enjoy.

Categories: FLOSS Project Planets

Drupal Association News: Viva DrupalCon Latin America

Planet Drupal - Mon, 2015-03-23 17:51

2015 started out strong with our first DrupalCon of the year, which took place from 10-12 February in Bogota, Columbia. Nothing feels better than to bring the power of DrupalCon to a new region where attendees can revel in their love for Drupal, the community, and enjoy time together. As people listened to the Driesnote, attended sessions and sprints, and celebrated with some Tejo, we heard a lot of “this is a real Con” and “it feels so good to experience this in my own backyard”.

Sharing the gift of DrupalCon with the Latin American community was a joy for Drupal Association staff and community organizers. It wouldn’t have happened without help from Aldibier Morales, Carlos Ospina, Ivan Chaquea, Nick Vidal, and Jairo Pinzon, who helped organize the event. Conversely, it better connected the Drupal Association with this region, helping us better understand the high level of contribution as well as new ways to support this region.

263 people attended DrupalCon Latin America from 23 countries including 12 Latin American countries. 63% of attendees said that this was their first DrupalCon, which underscores why it’s so important to bring DrupalCon to new parts of the world. Attendees were primarily developers from Drupal Shops, but there was more diversity than expected. The event also attracted a higher level of beginners than expected and 14% of attendees were women, which falls between DrupalCon Europe (10% women) and DrupalCon North America (22%). Below are some demographic tables that compare DrupalCon Latin America with DrupalCon Austin.

As you can imagine, the most attended sessions were focused on Drupal 8. DrupalCon Latin America was the first event to offer translated sessions and all sessions were recorded and posted to the DrupalCon YouTube Channel. Thanks to Lingotek, 20 additional session recordings were translated, too, and can be found on Youtube.

One of the big takeaways for Drupal Association staff was finding out how many attendees contribute to Drupal. When Megan Sanicki, COO, asked in her keynote introduction presentation how many people contributed, a large number of hands went up. It explains why DrupalCon Latin America had the largest percentage of attendees attend the sprint compared to any other DrupalCon --  38.4% of attendees showed up to make a difference. Thanks to the sprint leads, YesCT, alimac, DevelCuy, jackbravo and the other 19 sprint mentors, 101 people were able to participate in the sprints.

We’re also happy that financially the event achieved its budget goals. When planning DrupalCon Latin America, we knew that hosting the event in a new region would create budget challenges. We accepted that and were willing to operate this DrupalCon at a loss. We see this as an investment in a region that will grow because DrupalCon was hosted here. Below is the high level budget and further below is a more detailed view into the expenses.

DrupalCon Latin America Budget Budget Actual Income $150,150 $104,513.74 Expenses $250,750 $188,034.40 Net Profit -$99,920 -$83,520.66

Overall, DrupalCon Latin America was a success! Session evaluations came back strong and the event received a high Net Promoter Score of 80. Also, attendees felt that they received more value than expected (see chart below).

While we hoped for larger numbers, it’s important to point out that DrupalCon Amsterdam in 2005 had about 100 attendees. When the event returned in 2014, it hosted 2,300 people. All regions have to start somewhere and DrupalCons have the power to infuse community members with a burst of energy and passion that helps the community grow. We saw this immediately after DrupalCon Latin America with the growth of Global Training Days. Last year, the region hosted 7 trainings total, but right after DrupalCon Latin America, the region hosted 10 - not even ¼ of the way into the year. Additionally, three Latin American community members nominated themselves in the Drupal Association At-Large Board Elections.

We are thrilled that we were able to bring DrupalCon to new regions of the world. Be sure and attend the closing session of DrupalCon Los Angeles to find out where we are bringing DrupalCon next.


DRUPALCON STATISTICS DEMOGRAPHIC COMPARISONS

Business (sales, marketing) Front end (design, themer) C-Level Site Builder Other (PM, Trainer, etc) Site Administrator

Job Function Austin Latin America Developer 40% 48% Business (sales, marketing) 11% 12% Front end (design, themer) 13% 10% C-Level 9% 9% Site Builder 11% 8% Other (PM, Trainer, etc) 9% 12% Site Administrator 7% 3%

 

How I use Drupal Austin Latin America Drupal Shop 47% 61% Site Owner 30% 12% Freelance 5% 9% Evaluating 6% 4% Hobbyist 2% 2%

 

Skill Level Austin Latin America Advanced 37% 40% Intermediate 39% 38% Beginner 23% 22%

 

SESSION STATISTICS DrupalCon Latin America: Highest Attended Sessions Count #d8rules - Web-automation with Rules in Drupal 8 87 An Overview of the Drupal 8 Plugin System 70 Drupal 8 CMI on Managed Workflow 67 Getting Content to a Phone in less than 1000ms 58 AngularJS + Drupal = Un Dúo Superheróico! 52 DevOps, por donde comenzar? 49 The Future of Commerce on Drupal 8 (and beyond) 43 I Want it All and I Want it Now: Configuration Management and CI 38 SEO for Drupal 37

 

DrupalCon Latin America: Youtube views (as of 3/11/2015) # of views DrupalCon Latin America 2015: Keynote by Dries Buytaert 1053 DrupalCon Latin America 2015: Keynote by Larry Garfield 546 DrupalCon Latin America 2015: The Future of Commerce on Drupal 8 (and beyond) 407 DrupalCon Latin America 2015: Drupal 8 CMI on Managed Workflow 241 DrupalCon Latin America 2015: AngularJS + Drupal = Un Dúo Superheróico! 238
DRUPALCON FINANCIALS Expenses

 

  Staff Wages, Benefits, Overhead $106,621.54 Catering $11,784.76 Staff Travel & Accommodations $11,552.25 Event Planning $9,244.45 Registration Materials, Conference Supplies, Tees $8,180.90 Taxes, Fees, VAT $7,009.54 Speaker Fees, Travel Awards, Etc $6,973.16 Translation $6,772.00 IT, Wifi, Electrical $6,705.89 Archiving $5,500.00 Design $4,500.00 Conference Facility $3,013.78 Shipping $176.13 Total Expenses $188,034.4

 

Categories: FLOSS Project Planets

FSF Blogs: LibrePlanet 2015: Highlights and what comes next

GNU Planet! - Mon, 2015-03-23 17:40

With approximately 350 people in attendance, we kept pace with last year—with a few improvements. In particular, our tech team did a fantastic job improving our video feeds, with peak usage at around 300 simultaneous views. We also worked hard to streamline the registration process, reducing time spent waiting in the registration line before Richard Stallman's Saturday morning keynote, with few hiccups.

Morning registration at LibrePlanet

Let's take a look at some more highlights.

  • Read our post about day one of the conference and the election of our newest board member, Kat Walsh.

  • FSF board member Benjamin Mako Hill kicked off Sunday morning with an examination of how the free software community has progressed with two major goals: access and empowerment. He argued that empowerment has a long way to go, and offered some suggestions for next steps.

  • This year, we had over 40 speakers, and like last year, about 40% were women.

  • Karen Sandler, in her new role as executive director of the Software Freedom Conservancy, gave her third LibrePlanet keynote to close the conference, expanding on the recent suit brought by Christoph Hellwig, contributor to the kernel Linux, against VMWare for violating the GPL by not releasing the source code for the version of the operating system kernel they distribute with their ESXi software. Read our statement of support here.

  • Sunday's talks explored the possibilities for free software: Jennie Rose Halperin of Safari looked at free software's potential role in cultural heritage organizations, Brett Smith discussed Arvados, a free software platform for big data science, Free Software Award-winner Sébastien Jodogne gave one of several talks on free software's role in medicine, and other talks looked at free software possibilities for universities and journalism, as well as examining threats to free software, gathering the community for strategy sessions, and offering practical training.

  • Our sustained partnership with MIT's Student Information Processing Board once again helped make LibrePlanet great, thanks to the energy and enthusiasm of SIPB's membership.

We on the FSF staff owe many thanks to our tireless volunteers, without whom LibrePlanet couldn't possibly be as large or well-run as it is. Thank GNU to Francis Rowe of Gluglug and Libreboot, rsiddharth, Janith Perera, Sharon Ramage, Anke Nowottne, Mo Moulton, Diane Williams, Elyes Ighilaza, Brendan Kidwell, Rimma Shakhbatyan, Andy Zimolzak, Herm Pena, Taylor Gunnoe, Tanya Balyan, Devin Ulibarri, Ryan Desfosses, Geoffrey Jenkins, Martin Yim, Daniel Felix, Evan Mulvaney, Veronika Alexander, Patrick Engelman, Dan Fitzmartin, Ian Denhardt, John Saylor, Ben Cook, Sunil Kumar, Tim Jordan, Alice Alisme, Sunil Kumar, Julian Daich, Lizz van Allen, Kendra Moyer, Loren Chen, Helen Jiang, Max Dunitz, Mariah Villareal, Shayna Cummings, Matthew Coleman, Weston Cooke, and George Chriss. If you're interested in volunteering next year (you'll receive gratis admission and a t-shirt), please contact resources@fsf.org.

Libreboot installation

If you attended LibrePlanet and haven't filled out our feedback survey yet, please take a few minutes to do so. LibrePlanet is for you, and your ideas, compliments, and criticisms will help us make the next LibrePlanet even more productive, welcoming, and fun. If you missed a session, or want to share them with friends, keep an eye on the LibrePlanet video archive (powered by GNU MediaGoblin). We're working hard to get videos posted within a week or two, and we'll inform you when they're available.

We had a great group of sponsors this year and we are looking to build on that for next year. If your company would like to support LibrePlanet, please contact us any time at donate@fsf.org.

With the renewed energy that comes from spending a great weekend with the free software community, we're excited about the year ahead. Mark your calendars for October 3, when we'll celebrate our 30th anniversary, with events here in the Boston area and around the world. If you'd like us to let you know when registration opens for LibrePlanet 2016, join the announcements list (anchored at the top of the 2015 conference page) today. Remember that becoming an FSF member gets you gratis admission to the conference, and bragging rights for supporting the FSF's work year-round.

Categories: FLOSS Project Planets

LibrePlanet 2015: Highlights and what comes next

FSF Blogs - Mon, 2015-03-23 17:40

With approximately 350 people in attendance, we kept pace with last year—with a few improvements. In particular, our tech team did a fantastic job improving our video feeds, with peak usage at around 300 simultaneous views. We also worked hard to streamline the registration process, reducing time spent waiting in the registration line before Richard Stallman's Saturday morning keynote, with few hiccups.

Morning registration at LibrePlanet

Let's take a look at some more highlights.

  • Read our post about day one of the conference and the election of our newest board member, Kat Walsh.

  • FSF board member Benjamin Mako Hill kicked off Sunday morning with an examination of how the free software community has progressed with two major goals: access and empowerment. He argued that empowerment has a long way to go, and offered some suggestions for next steps.

  • This year, we had over 40 speakers, and like last year, about 40% were women.

  • Karen Sandler, in her new role as executive director of the Software Freedom Conservancy, gave her third LibrePlanet keynote to close the conference, expanding on the recent suit brought by Christoph Hellwig, contributor to the kernel Linux, against VMWare for violating the GPL by not releasing the source code for the version of the operating system kernel they distribute with their ESXi software. Read our statement of support here.

  • Sunday's talks explored the possibilities for free software: Jennie Rose Halperin of Safari looked at free software's potential role in cultural heritage organizations, Brett Smith discussed Arvados, a free software platform for big data science, Free Software Award-winner Sébastien Jodogne gave one of several talks on free software's role in medicine, and other talks looked at free software possibilities for universities and journalism, as well as examining threats to free software, gathering the community for strategy sessions, and offering practical training.

  • Our sustained partnership with MIT's Student Information Processing Board once again helped make LibrePlanet great, thanks to the energy and enthusiasm of SIPB's membership.

We on the FSF staff owe many thanks to our tireless volunteers, without whom LibrePlanet couldn't possibly be as large or well-run as it is. Thank GNU to Francis Rowe of Gluglug and Libreboot, rsiddharth, Janith Perera, Sharon Ramage, Anke Nowottne, Mo Moulton, Diane Williams, Elyes Ighilaza, Brendan Kidwell, Rimma Shakhbatyan, Andy Zimolzak, Herm Pena, Taylor Gunnoe, Tanya Balyan, Devin Ulibarri, Ryan Desfosses, Geoffrey Jenkins, Martin Yim, Daniel Felix, Evan Mulvaney, Veronika Alexander, Patrick Engelman, Dan Fitzmartin, Ian Denhardt, John Saylor, Ben Cook, Sunil Kumar, Tim Jordan, Alice Alisme, Sunil Kumar, Julian Daich, Lizz van Allen, Kendra Moyer, Loren Chen, Helen Jiang, Max Dunitz, Mariah Villareal, Shayna Cummings, Matthew Coleman, Weston Cooke, and George Chriss. If you're interested in volunteering next year (you'll receive gratis admission and a t-shirt), please contact resources@fsf.org.

Libreboot installation

If you attended LibrePlanet and haven't filled out our feedback survey yet, please take a few minutes to do so. LibrePlanet is for you, and your ideas, compliments, and criticisms will help us make the next LibrePlanet even more productive, welcoming, and fun. If you missed a session, or want to share them with friends, keep an eye on the LibrePlanet video archive (powered by GNU MediaGoblin). We're working hard to get videos posted within a week or two, and we'll inform you when they're available.

We had a great group of sponsors this year and we are looking to build on that for next year. If your company would like to support LibrePlanet, please contact us any time at donate@fsf.org.

With the renewed energy that comes from spending a great weekend with the free software community, we're excited about the year ahead. Mark your calendars for October 3, when we'll celebrate our 30th anniversary, with events here in the Boston area and around the world. If you'd like us to let you know when registration opens for LibrePlanet 2016, join the announcements list (anchored at the top of the 2015 conference page) today. Remember that becoming an FSF member gets you gratis admission to the conference, and bragging rights for supporting the FSF's work year-round.

Categories: FLOSS Project Planets
Syndicate content